Posts tagged: REF2021

Guide to Research Impact Evidence Collection

There is a growing body of opinion that holds that researchers have a responsibility to articulate the impact of their research to non-academic audiences. Indeed, the way in which research funding is allocated now increasingly reflects researchers’ ability to generate, demonstrate and evidence their impact.

A renewed emphasis on the importance of both planning and evidencing research impact requires researchers to develop ever-more robust and integrated ‘pathways to impact’ (as illustrated by Research Councils UK) within research funding bids in order to identify and reward the contribution that high-quality research will make to the economy or to society. 

Although there are as yet no specific guidelines on what constitutes impact evidence in terms of the next REF2021 exercise, the following pointers provide an indication of what researchers should be thinking about as they start to collect and collate evidence to support their impact.

 

General advice on collecting impact evidence

  • If you are looking to generate impact from your research, ensure that you engage from the start with the University Impact, Engagement and Environment Coordinator (Emma Sutton) and your School Impact Coordinator
  • Consider what indicators of impact are going to be used at the onset of the project – how will success be measured throughout and what will need to be captured?
  • Complete a stakeholder analysis for your potential impact
  • Be able to clearly demonstrate the pathway to impact: what were the steps taken, what is the embedded research etc.
  • Be able to articulate the significance of the potential impact (reach, audience, policy change, technology development) – the “Why should we care?” question
  • Use existing and well-understood baselines and gold standards to measure impact
  • Remember to store all impact evidence on an ongoing basis in the Figshare repository (see below for further details)

 

Here are examples of what evidence could look like:

    • Quotations from high profile figures
    • Testimonials, interviews (always including who, when, where and job title and with consent to reproduce)
    • Specific examples e.g. increased value of a company or number of lives saved by a new technology
    • Published reports as a result of research conducted (especially reports commissioned by independent bodies or those external to the immediate project)
    • Delegate lists to key meetings/conferences/exhibitions/events
    • Letters of support from external bodies

**Look to use both qualitative and quantitative data where possible!**

 

Points to remember:

    • Make sure that the evidence will be available in time to meet REF2021 deadlines
    • Ensure that information is robust and credible
    • Ensure that information is independently verifiable
    • Link evidence to clear targets and indicate whether these were met or exceeded
    • Provide evidence of research being widely disseminated, e.g. through tweets, blogs, access to websites, press coverage, broadcastings, downloads, sales
    • Find ways of communicating the research as it progresses to develop wider impact along the way (not just at the end)
    • Conduct exit interviews with the business if ending relationship/researcher if leaving institution – evidence of impact must be captured before departure
    • Be able to demonstrate that without the research, the impact would not have occurred: how has the research made the difference?

 

The University now uses the Figshare data repository alongside USIR in order for researchers to store evidence relating to their research.

It is therefore good practice for all researchers to begin storing all their impact evidence in Figshare from now onwards so that an institutional repository of impact case study evidence can be built upon.

Currently, Figshare accounts have been created for all those researchers within each School who have been identified as potentials for submitting an impact case study to REF2021 and beyond.

Figshare can be accessed at the following link: https://salford.figshare.com/

Figshare also run monthly webinars to help researchers understand the basics of the system and it is highly recommended that you register for one of these at the following address:

https://zoom.us/webinar/register/WN_5kx95ST4RC-XKMYlguumgw

 

 

Remember: the earlier you begin collecting and collating your impact evidence, the easier it will make the final impact case study submission!

 

 


How to Write a 4* Journal Article

Professor Mark Reed, Professor of Socio-Technical Innovation at Newcastle University

In December, Prof Mark Reed, Professor of Socio-Technical Innovation at Newcastle University and the man behind Fast Track Impact, tweeted some thoughts on how to write a 4* paper for the REF and wrote a blog about it. This post is published here with the author’s permission.

How do you write a 4* paper for the Research Excellence Framework (REF)? It is a question I’ve asked myself with some urgency since the Stern Review shredded my REF submission by not allowing me to bring my papers with me this year to my new position at Newcastle University.

Obviously the answer is going to differ depending on your discipline, but I think there are a few simple things that everyone can do to maximize their chances of getting a top graded research output.

I’m going to start with the assumption that you’ve actually done original, significant and rigorous work – if you haven’t then there is no point in reading any further. However, as I am increasingly asked to pre-review papers for colleagues across a range of disciplines, I am seeing examples of people who write up work as a 2* or 3* paper that has the potential to get a better score. I should point out that I believe that there is an important role for 1* and 2* papers, and that I regularly write these on purpose to address a problem of national significance and frame it for the specific, narrow audience that is likely to be able to benefit most from my work. However, whether I like it or not, as a Professor in a research-intensive University, there is an expectation that I will be submitted as a 4* researcher, which means I need a few 4* papers as well.

You can see some more detailed thoughts on what I think makes 4* for different types of paper in this Tweet:

https://twitter.com/profmarkreed/status/801348612345253888/photo/1

As you’ll see from the discussion under that tweet though, my more detailed thoughts probably only apply to Units of Assessment across panels A-C, and probably isn’t relevant to the arts and humanities.

Having said this, I think there are a number of things we can all do to maximize the chances of our work being viewed favourably by REF panelists.

  1. Write to the criteria:when I was learning to drive, my instructor told me that in the test I should make sure I moved my head when I was looking in the rear view mirror, to make sure the examiner noticed I was using my mirrors. We’re all used to writing to the criteria of funding calls, and in fact we are all perfectly used to writing papers to the criteria of our target journals. In the last REF, research outputs were judged against three criteria: originality, significance and rigour. Whatever the interpretation of these criteria in your discipline, have you made it explicit to REF panelists reading your work exactly what is original, and why it is so original? Have you explained and effectively justified the significance of your work? And have you included evidence that your methods, analysis and interpretation is rigorous, even if you have to use supplementary material to include extra detail about your methods and data to get around journal word limits?
  2. Get REF feedback before you submit your work for publication:find out who is going to be reviewing research outputs for REF internally within your Unit of Assessment at your institution and ask them to review your work before you submit it. They may be able to make recommendations about how you might improve the paper in light of the REF criteria. Sometimes a little bit of extra work on the framing of your research in relation to wider contexts and issues can help articulate the significance of your work, and with additional reading and thinking, you may be able to position your work more effectively in relation to previous work to demonstrate its originality more clearly. Adding a few extra details to your methods and results may re-assure readers and reviewers that your approach is indeed rigorous. This is not just about doing world-leading research; it is about demonstrating to the world that your work is indeed world-leading. For me, these criteria are nothing new and are worth paying attention to, whether or not we are interested in REF. Meeting these three criteria will increase the chances that you get through peer-review and will increase the likelihood that your work gets cited.
  3. Analyse and discuss good practice in your own area: the only way to really “get your eye in” for REF is to actually look at examples of good and poor practice in your own area. Below, I’ve described how you can design an exercise to do this with your colleagues. You can do it yourself and learn a lot, but from my own experience, you learn a lot more by doing this as a discussion exercise with colleagues who work in your area. If you can, take notes from your discussion and try and distill some of the key lessons, so you can learn collectively as a group and more effectively review and support each other’s work.

How to organize a discussion to work out what makes a 4* paper in your area:

  • Identify top scoring institutions for your Unit of Assessment (UOA): download the REF2014 results, filter for your UOA (columns E or F), then filter so it only shows you the outputs (column J), and then filter for 4* (column L), showing only the institutions from your UOA that had the highest percentage of 4* outputs. Now for those institutions, look across the table (columns L-P) to see which has the highest proportion of outputs at either 3* or 4*. For example, an institution may have 80% of its outputs graded at 4* and 15% graded at 3*, meaning that 95% of its outputs were graded at 3-4*
  • Download a selection of papers from the top scoring institutions: go to your UOA on the REF website, find and click on the institutions you’ve identified in step 1, under “view submission data”, click on “research outputs”, copy and paste output titles into Google Scholar (or your search engine of choice) and download the articles. You may want to select outputs randomly, or you may want to go through more selectively, identifying outputs that are close to the areas your group specialize in
  • Repeat for low scoring institutions so you can compare and contrast high and low scoring outputs
  • Discuss examples: print copies of the high and low scoring outputs, labeled clearly, and in your next UOA meeting, let everyone choose a high and a low-scoring example. Given them 10-15 minutes to quickly read the outputs (focusing on title, abstract, introduction, figures and conclusions so you’re not there all day) and then ask the group (or small groups if there are many of you) to discuss the key factors that they think distinguish between high and low scoring outputs. Get your group(s) to distill the key principles that they think are most useful and disseminate these more widely to the group, so that anyone who wasn’t present can benefit.

It would be great if I could tell you that these are my “three easy ways to get a 4* paper” but doing work that is genuinely original, significant and rigorous is far from easy. If you have done work that is of the highest quality though, I hope that the ideas I’ve suggested here will help you get the credit you deserve for the great research you’ve done.