We Need New Ways to Measure Impact

Categories: Evaluation, Impact, Research, Tools and Resources

Tags: , , , , ,

During October SAGE MethodSpace focused on evaluation and other types of applied research. This post, originally published on LSE Impact, discusses the challenges associated with evaluating research in complex, controversial fields. Posts from the series and relevant posts from the archives can be found using this link

Sustainability researchers are experiencing increasing pressure to produce practical, as well as politically and socially acceptable, solutions for addressing complex societal challenges such as climate change and the rapid depletion of natural resources. To do so, researchers must work with a wide variety of scientific and non-scientific actors to produce research that is relevant, accessible, timely, and context-dependent. These shifting expectations are reflected in the growing number of public and private funding streams that favor “policy-relevant” sustainability research and require funding recipients to demonstrate “value for money”, defined in terms of cost-benefit and calculated using one or more quantitative measurements.

At the same time, an increasingly protracted debate is currently underway concerning how different science, arts, and humanities subjects contribute in diverse and sometimes contradictory ways to societal development. There is a growing body of evidence that conventional forms of impact assessment, which broadly insist on quantifying research outputs, cannot adequately capture this diversity. For example, the use of citation scores and journal impact factors cannot be used to measure the societal impact of research. In fact, these measurements may not even represent scientific impact very well, because they ignore less tangible aspects of academics’ success, such as being well connected to others in the fieldCalls thus abound to radically reinvent the impact agenda and make current ways of assessing impact more wide-ranging, diverse and inclusive.

The post can originally be on LSE Impact Blog under the title “From invisibility to impact: radically different measures are needed to capture the true impact of research” by Henrike Rau, Gary Goggins, Frances Fahy.

Alternative forms of impact assessment that capture the complexity of the short-, medium- and long-term effects of research, and that work across different areas and disciplines, clearly demand new ways of thinking and measuring. Here, the concept of “science usability” has emerged as a credible alternative to conventional research quality assessment. This innovative approach promotes an improved understanding of the opportunities and constraints concerning the application of knowledge in society. The identification of different types of users and divergent forms of use add further nuance to the science usability debate. Some attempts have also been made to develop indicators that go beyond conventional bibliometric analyses. Proposals also exist to include in research quality assessment many of the less tangible influences on the trajectory and outcomes of research, such as intuition and passion.

Image credit: Wilderness by Ravi Roshan, via Unsplash (licensed under a CC0 1.0 license).

Drawing on a large-scale interdisciplinary research project undertaken in the Republic of Ireland between 2009 and 2015, our study demonstrates the range and diversity of outputs and outcomes that fall outside of most traditional research impact assessments. Funded by Ireland’s Environmental Protection Agency, CONSENSUS (consumption, environment and sustainability) examined key challenges relating to the domestic use of water, energy, food, and mobility options. Linking qualitative and quantitative methods, it brought together a broad range of experiential and scientific expertise to produce robust scientific insights into social and environmental aspects of domestic consumption. The project engaged over 100,000 members of the public and more than 150 representatives from business, policy, and civil society organisations to identify myriad opportunities and constraints for lowering unsustainable resource use in households.

In carrying out a quasi-longitudinal qualitative impact assessment of the project, we were able to demonstrate the significant impact of CONSENSUS on the local and national sustainability research landscape in Ireland, its multi-faceted influence on international research agendas and activities, and its contribution to shaping sustainability policy and practice. For example, the production of a CONSENSUS Briefing Note on the subject of mobility biographies in Ireland presented complex content on the relationship between mobility practice and life events in an easily digestible way that was intended to appeal to time-poor political and civil society actors. Similarly, the CONSENSUS Transition Frameworks proposed concrete measures for policymakers, education and community workers, research and business innovators alike, to achieve greater sustainability in home-heating, washing and eating over the short-, medium- and long-term. Finally, key findings from CONSENSUS fed directly into the National Economic and Social Council’s 2012 report on social and behavioural aspects of climate change, which subsequently informed Ireland’s climate change policy. Many of these impacts would not, and could not, be captured by traditional quantitative assessment procedures.

Based on these findings, we argue that new and innovative forms of impact assessment not only need to be more inclusive but also time-sensitive. Scientific and societal impacts of research may only become visible after an extended period of time; usually one that exceeds the time span of most conventional impact assessment procedures. Moreover, short-term thinking often dominates the domain of research management and governance, creating tensions between those who conduct research and those who manage and assess it. Instead, continuous long-term monitoring of research endeavors such as CONSENSUS promises to provide a much more accurate picture of their actual impact. Importantly, the recent push for novel ways of conceptualizing and measuring impact has provided fresh opportunities for both academics and non-academics to debate the societal relevance of research.

Relevant MethodSpace Posts

Leave a Reply