March 2023
This month’s reports summarises four articles that in/directly relate to the impact of universities. The first represents a systematising approach to the evaluation of research impact (Williams, 2020). The second study examines research impact evaluation in Australia and that specific political context (Gunn & Mintrom, 2016). The third paper studies the use of altmeretics to evidence research impact claims on why university rankings persist, despite their ongoing criticism (Bornmann et al. 2016). The piece of research represents a positioning paper for the a sociology on valuation (Lamont, 2012).
[1]) The evaluation of research and its impact determines what types of knowledge are valued, incentivized and rewarded. However, the measurement of impact has been highly contested and dominated by pragmatic research policy imperatives. This article provides a new theoretical framework for understanding research impact and its assessment from a sociological perspective, which argues that research impact occurs at the intersection of various fields of power. The approach permits research actors and evaluators to consider the weighting and relative importance of each parent field, to gain legitimacy and accumulate capital across fields. It can guide analysis of the effects of assessing impact in terms of research agenda, concern with measures, and investment.
[2]) Australia's national Research Engagement and Impact Assessment presents an opportunity to review attempts to improve the non-academic impact of academic research. The impact agenda represents a new phase in academic research evaluation and funding, with a heightened need to demonstrate a return on public investments in research. This paper reviews the policy journey of research impact in Australia from the proposed Research Quality Framework (RQF) to the National Innovation and Science Agenda (NISA). The meandering political process highlights the contentious nature of higher education policy making, and the difficulty of designing a valid audit mechanism that encourages researchers to enhance the impact of their work beyond the campus.
[3]) This study examines the use of a new altmetrics source, policy documents, to measure the societal impact of research. The authors use a comprehensive dataset of publications on climate change to study the usefulness of this new data source. They find that only a small percentage of publications have at least one policy mention, but that papers published in Nature and Science and in the areas of earth and related environmental sciences and social and economic geography are more relevant in the policy context. The authors suggest that more empirical studies are necessary to fully evaluate the use of policy documents as an altmetrics source.
[4]) This paper discusses the sociology of valuation and evaluation (SVE) and aims to bring various bodies of work into conversation with each other to stimulate more cumulative theory building. The paper focuses on sub-processes such as categorization and legitimation, conditions that sustain heterarchies, and valuation and evaluative practices. The paper reviews these literatures and provides directions for a future research agenda, including driving questions that are currently attracting interest, such as how shared tastes are formed through networks and the impact of information technology on evaluation. The paper suggests that more systematically cumulative work is needed to continue developing the SVE literature.
[1] Williams, K. (2020) ‘Playing the Fields: Theorizing Research Impact and Its Assessment’, Research Evaluation, 29: 191–202.
[2] Gunn, A., & Mintrom, M. (2018). Measuring research impact in Australia. Australian Universities Review, 60(1), 9-15
[3] Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?. Scientometrics, 109(3), 1477-1495.
[4] Lamont, M. (2012) ‘Toward a Comparative Sociology of Valuation and Evaluation’, Annual Review of Sociology, 38: 201–21
