For October, SAGE MethodSpace will focus on evaluation and other types of applied research. Posts in the series and relevant posts from the archives will be found using this link.
This post, while discussed in the context of the UK’s 2014 Research Excellence Framework, is relevant to anyone looking for ways to use testimonials to build credibility. It was originally posted on the LSE Impact blog.
The Research Excellence Framework (REF) impact assessment requires impact case studies to include evidence of impact. This evidence can take many forms, quantitative or qualitative. Statements from research users, stakeholders, and beneficiaries can be a powerful form of qualitative evidence, as seen in REF 2014 (see here for an attempt to analyse links between evidence types and impact case study scores). For the next REF we have the opportunity to collect these statements as we go along and this guide should facilitate such statement-gathering.
What to Include?
In general, statements should:
- Be written on the external organisation’s headed paper (or a suitably professional-looking email).
- Be signed by someone at an appropriate level. This will vary by case study but considerations should include seniority vs. connection to the research (e.g. should the statement come from your direct contact, the person with responsibility for the area, or the head of the organisation?), maturity of relationship, reputation, and conflicts of interest.
- Name the researcher and refer to the research (could be in descriptive terms, citation of a research output, name of research programme – whatever comes across as most fitting).
- Describe how the organisation “found” the research/researcher.
- Describe how it fits with the organisation’s activities, strategy, needs, challenges, opportunities, and other drivers.
- Describe how the research/knowledge/skills were put into action or used – e.g. did the organisation work with the academic (maybe through commissioned research, consultancy, knowledge transfer grant, advisory work, other joint activities); did they use the research in their decision-making; did they train their staff according to the research?
- Describe what happened as a result of using the research/knowledge/skills or working with the researcher – e.g. did they produce guidelines for practice; were they better informed in making strategic decisions; was their service provision directly improved?
- Describe the resulting impact of this work – what was the benefit of using the research/knowledge or working with the researcher? Include quantitative or qualitative indicators to show the impact – i.e. how they know it was beneficial. They could also say where they’d be if they hadn’t used the research. This is the most important part of the statement as it’s where the impact is really articulated (and any quantitative/qualitative evidence the organisation provides can be quoted in the case study and woven into the narrative).
- Say something about the future – what’s next in this line of work? Do they foresee continued and growing benefits? Will they work with the researcher again? Will they be more open to using academic research in the future? Maybe they’ll change the way they operate as a result of the impactful piece of work.
How to gather testimonials?
The first (and by far the best) option is to put the above list into your own words (so it doesn’t sound so much like a checklist) and use this to prompt a statement from your partner. You could either put it in writing and let them respond accordingly or you could use this as the basis of a conversation/interview. The beauty of the latter approach is that you can explore and clarify and it may uncover other relevant information.
Secondly, you could use the above list as a checklist/questionnaire. I wouldn’t recommend this approach. It may save time but you are unlikely to get the richness or authenticity of a more personal/tailored interaction. It won’t strengthen your relationship with the organisation and it may even damage it.
Finally, in some cases it may be necessary to essentially write the statement yourself and hand it over to the partner to sign. This is not recommended as you lose the authentic voice and you may miss some aspect of what made the work so valuable (including possibly some extra information the partners would have included if they’d had to write it themselves). On top of this there is a very real risk that if the academic writes a number of such letters for different partners to sign, they could all end up looking fundamentally the same which undermines the credibility of the messages. Consider how you’d view this as an assessor.
- Make sure the person providing the statement knows what it is for and has the authority to give it.
- Observe the relevant data management and ethics policies as you gather, hold, and use this information.
- In REF2014, some organisations were overwhelmed with requests for testimonials to the point where relationships were affected and in some cases they simply refused to provide testimonials. This is where strong relationships really count so focus on lasting, rather than superficial, interactions with stakeholders and partners.
- Some organisations will be concerned about confidentiality. Although we don’t currently know how this will work for impact case studies and evidence in REF2021, there were provisions for this in REF2014 so we can expect similar in REF2021.
- Don’t wait to get testimonials. The details, nature, and value of impacts may become dull with time so jump on them while they’re fresh. Plus you never know where people will be in a couple of years.
These statements are not only a source to be referenced – you can use choice quotes to tell the story and illustrate the outcomes. Here are some examples from REF2014 showing the kinds of things said in testimonials and how they can be weaved into the narrative. They all happen to be from UoA 22 (Social Work and Social Policy) just because they were handy. You will find similar examples across the UoAs.
This is evidenced by a comment from a Regional HIV and AIDS Advisor at Save the Children: “By sharing rigorous evidence on how children affected by HIV faced increased vulnerabilities over time, Dr Cluver has engaged major policy makers, donors and program implementers in critically ‘rethinking’ and redesigning programs with an emphasis on effectively measuring results. Based on Dr Cluver’s research, UNICEF, PEPFAR have launched new technical guidance (UNICEF: From Evidence to Impact; PEPFAR: OVC Guidance: 2012) for program implementers. Her work is a testimony of how rigorous research is the foundation for effective programming…a clear example how a true partnership between researchers, policy makers and implementers can result in programs that actually make a difference in children’s lives”.
A senior advisor for PEPFAR also stated: “I have quoted Dr Cluver’s data more than any other research to support our work and also to justify and set policy for our global portfolio. All of the data presented from these studies has been influential, however of particular influence has been the data discussed in Cluver [et al] (2011) and 2012.”
As explained by the then Deputy Director of the Social Exclusion Task Force at the Cabinet Office, who played a major part in driving New Labour policies on early parenting interventions in the late 2000s: “Support from the Start” and Gardner’s UK trials were highly influential in creating momentum towards these major policy developments, including the very substantial rollout led by the National Academy of Parenting Practitioners (NAPP) from 2008.
The founder and research director of NAPP adds to this: “Gardner’s research was pivotal in helping persuade the Cabinet Office to set up NAPP. Her trial was the first in the world to show that parenting programmes could be effective in reducing severe antisocial behaviour, outside the narrow confines of child mental health services, showing that the voluntary sector could do just as good a job with difficult cases. Influenced by these findings, the government awarded £35 million to NAPP to disseminate evidence-based parenting programmes across the voluntary sector; this is estimated to have benefited over 150,000 children; Gardner was an important member of its steering group to ensure this research was implemented.”
Gardner’s team’s research is frequently cited in influential systematic reviews, guidelines, and policy documents. For example, four of the trials have been cited in the NICE (National Institute of Health and Clinical Excellence) Guidelines; the Chair confirms their policy influence: “As Chair of the NICE Guideline on antisocial behaviour and conduct disorders, I oversaw the marshalling of the evidence which has to be relevant to British practice, and Gardner’s work was important in shaping the recommendations, since it showed that the interventions work in Britain — not all psychosocial interventions developed in the USA do this. Their work also added to the plausibility of the recommendations since it demonstrated that the mediating mechanism was an increase in positive parenting”.
3. From “Situational crime prevention policy and practice”:
A letter from the Home Office states: “The work of the Department and its advocacy for, and expertise in, situational crime prevention strategies are well known by officials responsible for Crime policy within the Home Office and to the experts within my own area, Home Office Science, whose task it is to ensure that we provide the best evidence on which to base policy developments”. As an example it is noted that “at a recent Forum ‘away-day’ he [Wortley] led a session on preventing theft from the person, which is one of the only crime types to have increased recently, and in particular focused on the need to convert good ideas for prevention into concrete policies and practices.”
The New Zealand Police explicitly credits SCS with a significant role in shaping their crime strategies and helping to reduce crime: “The Jill Dando institute (JDI) has made a significant contribution to the development of crime science and intelligence led policing in New Zealand. We are grateful to JDI staff and associates who have increased our knowledge base, contributed to the development of our key products and fundamentally helped evolve the mindset within the New Zealand Police over the last 10 years…Over the past 4 years we have achieved sizeable reductions in crime against a backdrop of very high (and improving) levels of public trust and confidence in Police.”
The practical value of the research is acknowledged by the Agency: “Of particular value in recent years has been the work done by the Department around the topic of Internal Child Sex Trafficking (ICST)… In 2010, SOCA actively supported two research projects carried out by Professor Gloria Laycock with two masters students, Ella Cockbain and Helen Brayley. One project examined the social networks of offenders and victims and the other deconstructed the offence into a ‘crime script’. The research has been influential in helping to shape SOCA’s response to ICST”.
I hope this is useful. If you have any experiences or tips of your own then please share! Use the comment box below.