Posts tagged: REF2021

Preparing for impact peer review

This month we are conducting an internal peer review of our potential impact case studies in preparation for REF2021. Twenty of these will subsequently be selected for external peer review in June.

Each Unit of Assessment (UoA) team has been asked to assign at least two reviewers to each impact case study (1 x Lead reviewer from the same UoA, 1 x Associate reviewer from a different UoA and, optionally, 1 x Non-academic reviewer [industry partner or similar]).

Reviewers have been given 4 weeks to look over the case studies and provide their feedback on a review sheet covering each of the 5 main areas in the impact case study (Summary of the impact; Underpinning research; References to the research; Details of the impact; Sources to corroborate the impact).

Each of the 5 aspects of the case study form are rated using a traffic lights system (red, amber, green) to indicate whether this is:

• an area requiring significant development (red)

• an area requiring some improvement (amber) or

• an area that is well developed and on track for submission (green).

An overall impact case study traffic lights rating is then provided at the bottom of the feedback sheet to indicate:

  1. evidence of reach and significance
  2. potential for submission of the case study to REF2021.

Feedback will be provided to the case study leads during May 2019.

This process will help inform decision-making within each UoA and will also identify where there is a need to focus resource for the final year of the REF process.

Go to www.salford.ac.uk/ref to check out some examples of annotated case studies from our 2018 external peer review.


Refining your impact case studies

Our potential case study leads for REF2021 have recently submitted the second draft of their impact case studies, which will be assessed as part of an internal peer review process in April. Twenty of these will then also be selected for external peer review in June.

At this stage in the process, feedback from colleagues can be key in ensuring that the case studies reach their full potential. Following the internal and external peer reviews we will have just over a year to generate some more impact, collect impact evidence and refine the narrative further before the REF submission deadline.

With this in mind, here are a few key points for our case study leads, or indeed anyone submitting an ‘impact statement’ for funding purposes or similar, to consider.

Key points to remember:

• Convincingly demonstrate the robustness and quality of the underpinning research in the first instance

• Distinguish between the underpinning research and resulting impact: establish the causation and make sure that there is a golden thread running through the narrative

• Do not focus too heavily on dissemination at the expense of resulting impacts: make sure you are not purely describing your pathway to impact

• Clearly articulate each of the impacts claimed, and their apparent significance and reach

• Ensure there is sufficient corroboration of the impact using appropriate evidence (testimonials, quotes from key stakeholders, citations in policy documents or in the media, documented changes to guidelines etc.)

• Where web pages are used, ensure you have preserved them (screen shots etc.) and don’t just use standard links that may become broken over time

Generating impact

If you are looking to generate some further impact to bolster your case study, don’t forget to consider the following:

1. Create a pathways to impact statement to clearly set out the impacts you are hoping to achieve

2. Complete a publics/stakeholder analysis to identify who you are hoping to influence

3. Engage with your stakeholders at every stage in the research process

4. Identify activities to engage with your publics

5. Drive impact online by developing a social media strategy

Go to www.salford.ac.uk/ref to check out some examples of annotated case studies from our 2018 external peer review.


Confirmed Impact Guidance for REF2021

With less than 2 years to go until our REF submission (deadline: Friday, 27th November 2020), the final set of guidance materials for REF2021 (including guidance on submissions and panel guidelines) was published on 31 January 2019 following wide consultation with the sector in late 2018. The final guidance documentation is available at www.ref.ac.uk/publications/

What does this mean for impact?

The salient points to take from the final REF guidance on impact case study submission are as follows:

Definition of impact for the REF

For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

Impact includes, but is not limited to, an effect on, change or benefit to:

• the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding

• of an audience, beneficiary, community, constituency, organisation or individuals

• in any geographic location whether locally, regionally, nationally or internationally

Impact also includes the reduction or prevention of harm, risk, cost or other negative effects.

Impacts will be assessed in terms of their ‘reach and significance’ regardless of the geographic location in which they occurred, whe

ther locally, regionally, nationally or internationally.

Impact also includes the reduction or prevention of harm, risk, cost or other negative effects.

Each of the four main panels (A, B, C, D) have slightly different requirements for the following:

• Continued case studies

• Indicators of quality for underpinning research

UoA Leads/Deputies are therefore encouraged to look closely at the panel guidance for their particular panel when reviewing impact case study drafts.

Submission requirements

• Each submission must include impact case studies (REF3 template) describing specific impacts that have occurred during the assessment period (1 August 2013 to 31 July 2020) that were underpinned by excellent research undertaken in the submitted unit. The underpinning research must have been produced by the submitting HEI during the period 1 January 2000 to 31 December 2020.

• This may include, for example, impacts at an early stage, or impacts that may have started prior to 1 August 2013 but continued into the period 1 August 2013 to 31 July 2020. Case studies will be assessed in terms of the reach and significance of the impact that occurred only during the period 1 August 2013 to 31 July 2020, and not in terms of any impact prior to this period or potential future or anticipated impact after this period.

• When writing case studies, submitting units should refer to the guidelines for presenting quantitative data set out in the ‘Guidelines for standardising quantitative indicators of impact within REF case studies’. These guidelines have been developed to enable more consistent presentation of quantitative evidence in case studies. This document (and a summary thereof) can be found separately at www.salford.ac.uk/ref under Impact Evidence Collection.

• More than one submitted unit (within the same HEI and/or in different HEIs) may include the same impact within their respective case studies, so long as each submitted unit produced excellent research that made a distinct and material contribution to the impact. In such cases, units may provide common descriptions of the impact arising, where they so wish.

• Impact case studies continued from examples submitted in 2014 will be eligible for submission in REF 2021 as long as they meet the 2021 eligibility criteria, including the length of the window for underpinning research and the assessment period for the impact described.

Go to www.salford.ac.uk/ref to check out all the REF guidance and more.


2019 preparations and ‘REF readiness’

With Christmas fast approaching, here in the REF team we are already turning our attention to 2019 and considering how our REF readiness activities will start to take shape. In terms of our preparations around impact, the new year will herald a raft of targeted activities to help ensure that our case study submission in 2020 is the strongest it can possibly be.

Key deadlines

A few key dates in 2019 for our impact case study leads to bear in mind are as follows:

  • 28 February 2019 – Submission of revised impact case study drafts (using new REF2021 template)
  • February/March 2019 – Impact Action Plan mid-point review meetings
  • March 2019 – Recording of short impact videos (via Marketing & External Relations)
  • 8 April 2019 – Chris Simms informal peer review workshop
  • April 2019 – Planned formal internal peer review of impact case studies
  • June 2019 – Planned formal external peer review of impact case studies

The above dates are complemented by the University’s ‘SECRET’ researcher development programme, which is designed to ensure that our researchers are ‘REF ready’. Further details can be found through our Salford Advantage pages at: https://www.salford.ac.uk/staff-development

The programme includes our monthly one-to-one storytelling mentoring sessions with Chris Simms (bookings through research-impact@salford.ac.uk), impact writing retreats, bidding and funding information sessions, REF briefings and much more.

Alternatively, training of interest is also listed on our Training Calendar, which can be found at: www.salford.ac.uk/ref

 


Impact Guidance for REF2021

The initial set of guidance materials for REF2021 (including guidance on submissions and panel guidelines) was published in July 2018 and consultation with the sector was concluded on 15th October 2018.

The final guidance is due to be published in January 2019, after which time the University will hold a series of briefing meetings through our appointed Unit of Assessment Leads to update colleagues on the key details.

In the meantime, the salient points to take from the latest REF consultation documents on impact case study submission is as follows:

 

Definition of impact for the REF

For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

Impact includes, but is not limited to, an effect on, change or benefit to:

  • the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
  • of an audience, beneficiary, community, constituency, organisation or individuals
  • in any geographic location whether locally, regionally, nationally or internationally

Impact also includes the reduction or prevention of harm, risk, cost or other negative effects.

Impacts will be assessed in terms of their ‘reach and significance’ regardless of the geographic location in which they occurred, whether locally, regionally, nationally or internationally.

**Further guidance about how panels will assess the case studies against the criteria of reach and significance is found separately at www.salford.ac.uk/ref under Impact Evidence Collection.**

 

Submission requirements

  • Each submission must include impact case studies (REF3 template) describing specific impacts that have occurred during the assessment period (1 August 2013 to 31 July 2020) that were underpinned by excellent research undertaken in the submitted unit. The impacts may have been at any stage of development or maturity during this period, so long as some effect, change or benefit meeting the definition of impact took place during that period.

 

  • This may include, for example, impacts at an early stage, or impacts that may have started prior to 1 August 2013 but continued into the period 1 August 2013 to 31 July 2020. Case studies will be assessed in terms of the reach and significance of the impact that occurred only during the period 1 August 2013 to 31 July 2020, and not in terms of any impact prior to this period or potential future or anticipated impact after this period.

 

  • The underpinning research must have been produced by the submitting HEI during the period 1 January 2000 to 31 December 2020. Underpinning research may be a body of work produced over a number of years or may be the output(s) of a particular project. It may be produced by one or more individuals.

 

  • When writing case studies, submitting units should refer to the guidelines for presenting quantitative data set out in the ‘Guidelines for standardising quantitative indicators of impact within REF case studies’. These guidelines have been developed to enable more consistent presentation of quantitative evidence in case studies. This document (and a summary thereof) can be found separately at www.salford.ac.uk/ref under Impact Evidence Collection.

 

  • More than one submitted unit (within the same HEI or in different HEIs) may include the same impact within their respective case studies, so long as each submitted unit produced excellent research that made a distinct and material contribution to the impact.

 

  • Impact case studies continued from examples submitted in 2014 will be eligible for submission in REF 2021 as long as they meet the 2021 eligibility criteria.

 

 

Go to www.salford.ac.uk/ref to check out all the latest REF guidance.

 

 

 


Guide to Research Impact Evidence Collection

There is a growing body of opinion that holds that researchers have a responsibility to articulate the impact of their research to non-academic audiences. Indeed, the way in which research funding is allocated now increasingly reflects researchers’ ability to generate, demonstrate and evidence their impact.

A renewed emphasis on the importance of both planning and evidencing research impact requires researchers to develop ever-more robust and integrated ‘pathways to impact’ (as illustrated by Research Councils UK) within research funding bids in order to identify and reward the contribution that high-quality research will make to the economy or to society. 

Although there are as yet no specific guidelines on what constitutes impact evidence in terms of the next REF2021 exercise, the following pointers provide an indication of what researchers should be thinking about as they start to collect and collate evidence to support their impact.

 

General advice on collecting impact evidence

  • If you are looking to generate impact from your research, ensure that you engage from the start with the University Impact, Engagement and Environment Coordinator (Emma Sutton) and your School Impact Coordinator
  • Consider what indicators of impact are going to be used at the onset of the project – how will success be measured throughout and what will need to be captured?
  • Complete a stakeholder analysis for your potential impact
  • Be able to clearly demonstrate the pathway to impact: what were the steps taken, what is the embedded research etc.
  • Be able to articulate the significance of the potential impact (reach, audience, policy change, technology development) – the “Why should we care?” question
  • Use existing and well-understood baselines and gold standards to measure impact
  • Remember to store all impact evidence on an ongoing basis in the Figshare repository (see below for further details)

 

Here are examples of what evidence could look like:

    • Quotations from high profile figures
    • Testimonials, interviews (always including who, when, where and job title and with consent to reproduce)
    • Specific examples e.g. increased value of a company or number of lives saved by a new technology
    • Published reports as a result of research conducted (especially reports commissioned by independent bodies or those external to the immediate project)
    • Delegate lists to key meetings/conferences/exhibitions/events
    • Letters of support from external bodies

**Look to use both qualitative and quantitative data where possible!**

 

Points to remember:

    • Make sure that the evidence will be available in time to meet REF2021 deadlines
    • Ensure that information is robust and credible
    • Ensure that information is independently verifiable
    • Link evidence to clear targets and indicate whether these were met or exceeded
    • Provide evidence of research being widely disseminated, e.g. through tweets, blogs, access to websites, press coverage, broadcastings, downloads, sales
    • Find ways of communicating the research as it progresses to develop wider impact along the way (not just at the end)
    • Conduct exit interviews with the business if ending relationship/researcher if leaving institution – evidence of impact must be captured before departure
    • Be able to demonstrate that without the research, the impact would not have occurred: how has the research made the difference?

 

The University now uses the Figshare data repository alongside USIR in order for researchers to store evidence relating to their research.

It is therefore good practice for all researchers to begin storing all their impact evidence in Figshare from now onwards so that an institutional repository of impact case study evidence can be built upon.

Currently, Figshare accounts have been created for all those researchers within each School who have been identified as potentials for submitting an impact case study to REF2021 and beyond.

Figshare can be accessed at the following link: https://salford.figshare.com/

Figshare also run monthly webinars to help researchers understand the basics of the system and it is highly recommended that you register for one of these at the following address:

https://zoom.us/webinar/register/WN_5kx95ST4RC-XKMYlguumgw

 

 

Remember: the earlier you begin collecting and collating your impact evidence, the easier it will make the final impact case study submission!

 

 


How to Write a 4* Journal Article

Professor Mark Reed, Professor of Socio-Technical Innovation at Newcastle University

In December, Prof Mark Reed, Professor of Socio-Technical Innovation at Newcastle University and the man behind Fast Track Impact, tweeted some thoughts on how to write a 4* paper for the REF and wrote a blog about it. This post is published here with the author’s permission.

How do you write a 4* paper for the Research Excellence Framework (REF)? It is a question I’ve asked myself with some urgency since the Stern Review shredded my REF submission by not allowing me to bring my papers with me this year to my new position at Newcastle University.

Obviously the answer is going to differ depending on your discipline, but I think there are a few simple things that everyone can do to maximize their chances of getting a top graded research output.

I’m going to start with the assumption that you’ve actually done original, significant and rigorous work – if you haven’t then there is no point in reading any further. However, as I am increasingly asked to pre-review papers for colleagues across a range of disciplines, I am seeing examples of people who write up work as a 2* or 3* paper that has the potential to get a better score. I should point out that I believe that there is an important role for 1* and 2* papers, and that I regularly write these on purpose to address a problem of national significance and frame it for the specific, narrow audience that is likely to be able to benefit most from my work. However, whether I like it or not, as a Professor in a research-intensive University, there is an expectation that I will be submitted as a 4* researcher, which means I need a few 4* papers as well.

You can see some more detailed thoughts on what I think makes 4* for different types of paper in this Tweet:

https://twitter.com/profmarkreed/status/801348612345253888/photo/1

As you’ll see from the discussion under that tweet though, my more detailed thoughts probably only apply to Units of Assessment across panels A-C, and probably isn’t relevant to the arts and humanities.

Having said this, I think there are a number of things we can all do to maximize the chances of our work being viewed favourably by REF panelists.

  1. Write to the criteria:when I was learning to drive, my instructor told me that in the test I should make sure I moved my head when I was looking in the rear view mirror, to make sure the examiner noticed I was using my mirrors. We’re all used to writing to the criteria of funding calls, and in fact we are all perfectly used to writing papers to the criteria of our target journals. In the last REF, research outputs were judged against three criteria: originality, significance and rigour. Whatever the interpretation of these criteria in your discipline, have you made it explicit to REF panelists reading your work exactly what is original, and why it is so original? Have you explained and effectively justified the significance of your work? And have you included evidence that your methods, analysis and interpretation is rigorous, even if you have to use supplementary material to include extra detail about your methods and data to get around journal word limits?
  2. Get REF feedback before you submit your work for publication:find out who is going to be reviewing research outputs for REF internally within your Unit of Assessment at your institution and ask them to review your work before you submit it. They may be able to make recommendations about how you might improve the paper in light of the REF criteria. Sometimes a little bit of extra work on the framing of your research in relation to wider contexts and issues can help articulate the significance of your work, and with additional reading and thinking, you may be able to position your work more effectively in relation to previous work to demonstrate its originality more clearly. Adding a few extra details to your methods and results may re-assure readers and reviewers that your approach is indeed rigorous. This is not just about doing world-leading research; it is about demonstrating to the world that your work is indeed world-leading. For me, these criteria are nothing new and are worth paying attention to, whether or not we are interested in REF. Meeting these three criteria will increase the chances that you get through peer-review and will increase the likelihood that your work gets cited.
  3. Analyse and discuss good practice in your own area: the only way to really “get your eye in” for REF is to actually look at examples of good and poor practice in your own area. Below, I’ve described how you can design an exercise to do this with your colleagues. You can do it yourself and learn a lot, but from my own experience, you learn a lot more by doing this as a discussion exercise with colleagues who work in your area. If you can, take notes from your discussion and try and distill some of the key lessons, so you can learn collectively as a group and more effectively review and support each other’s work.

How to organize a discussion to work out what makes a 4* paper in your area:

  • Identify top scoring institutions for your Unit of Assessment (UOA): download the REF2014 results, filter for your UOA (columns E or F), then filter so it only shows you the outputs (column J), and then filter for 4* (column L), showing only the institutions from your UOA that had the highest percentage of 4* outputs. Now for those institutions, look across the table (columns L-P) to see which has the highest proportion of outputs at either 3* or 4*. For example, an institution may have 80% of its outputs graded at 4* and 15% graded at 3*, meaning that 95% of its outputs were graded at 3-4*
  • Download a selection of papers from the top scoring institutions: go to your UOA on the REF website, find and click on the institutions you’ve identified in step 1, under “view submission data”, click on “research outputs”, copy and paste output titles into Google Scholar (or your search engine of choice) and download the articles. You may want to select outputs randomly, or you may want to go through more selectively, identifying outputs that are close to the areas your group specialize in
  • Repeat for low scoring institutions so you can compare and contrast high and low scoring outputs
  • Discuss examples: print copies of the high and low scoring outputs, labeled clearly, and in your next UOA meeting, let everyone choose a high and a low-scoring example. Given them 10-15 minutes to quickly read the outputs (focusing on title, abstract, introduction, figures and conclusions so you’re not there all day) and then ask the group (or small groups if there are many of you) to discuss the key factors that they think distinguish between high and low scoring outputs. Get your group(s) to distill the key principles that they think are most useful and disseminate these more widely to the group, so that anyone who wasn’t present can benefit.

It would be great if I could tell you that these are my “three easy ways to get a 4* paper” but doing work that is genuinely original, significant and rigorous is far from easy. If you have done work that is of the highest quality though, I hope that the ideas I’ve suggested here will help you get the credit you deserve for the great research you’ve done.