For October, SAGE MethodSpace will focus on evaluation and other types of applied research. Posts in the series and relevant posts from the archives will be found using this link.
Many of us are turning to posting audio podcasts or videos to reach more people with our research insights, others have the opportunity to write or speak for the popular press. This post, originally on LSE Impact, discusses ways to evaluate the success and reach of media broadcasts.
When the media calls to say “we’re interested in your research,” it’s certainly a rather flattering moment. There may have been a bit of run-up and brokering for this to happen, and it may be just the first step on a long road, with quite a while passing before the research is broadcast. But at some point beyond this, the REF-aware researcher will begin to think about what impact this might produce and how exactly to measure that. The media has a pervasive role in the communication of research; whether a short announcement or a feature-length documentary, the content can be broadcast to millions of people in a very short space of time. So reach, one aspect of REF impact assessment, may be high – but what about significance? That short space of time might not be long enough to leave a lasting impression.
We have been collaborating on an evaluation of this type of significance. In January we published an article in Research for All exploring the evaluation of research presented on television. By using a pragmatic mixed-methods approach, we attempted to measure the impact of research broadcast in two one-hour factual health-related programs on prime time television. The reach of engaging with TV programming did not disappoint, with an average of 2.5 million viewers (11 per cent of the total evening’s viewers) across the two programs, the majority of whom watched on the night of the broadcast (data obtained from the Broadcasters’ Audience Research Board) in the UK. To determine the significance of these broadcasts, our approach required participants for questionnaires (to be completed before and after the broadcasts) and focus groups, and Twitter data mining, as a specific hashtag was associated with the programs.
There were many hurdles along the way, such as recruitment and retention of participants, and the effect of participants being aware they were taking part in research. These are well known and can be countered by using professional recruitment companies and through reimbursement of time and expenses.
For our focus groups we employed a professional recruitment agency to target particular demographics. The targeting is important to ensure the evaluation is addressing the views of that program’s likely audience in terms of age, gender, socioeconomic group, etc. If participants do not mirror the audience the evaluation may underestimate the impact. For instance our self-selecting participants were recruited through our own networks and had a slightly higher educational level compared to those of the focus groups and the likely viewers.
We were particularly interested in discovering if the broadcasts increased the knowledge of the participants and, in the case of the focus groups, if this translated into behavior changes that could alter their health. Indeed we saw quite a leap in knowledge in some cases, as demonstrated by some participants correctly answering certain questions after the viewing compared to before. However we also found that in response to some questions the audience certainly initially knew more than we were expecting. The focus groups in particular allowed for new topics to come out that could be useful for expanding or targeting research in the future. In addition, focus group participants were contacted nine weeks later and asked if they had changed any habits to the benefit of their health: 30 per cent revealed that they had. This indicates that the impact did indeed have significance in addition to its documented reach.
Mining Twitter elicited nearly 1,300 tweets but only one percent were relevant to our research themes and it was impossible to say if there was lasting change in knowledge or habits, as the majority were made during the hours of broadcast and responded directly to what was shown on screen. So, fascinating as it was to see the response to the programs without any kind of intervention, this route did not necessarily provide information about impact but rather on different audiences and topics to engage with in the future.
Overall we found an indication that broadcast programming did indeed have a significant impact, coupled with a wide reach. The evaluation went beyond what might be the norm for the assessment of a media broadcast – particularly in the eyes of the broadcasters, where this is largely just a numbers game. The evaluation required investment of both time and money but we feel this kind of engaged investigation is important not only to find out about the value of the initial broadcast, but also to help connect with the audience to find new ways forward, potentially in partnership in the future.