By Dr Chris Hewson, Impact Coordinator
Upon its release last Thursday, the Twittersphere became the locus for a series of overlapping debates on Lord Nicholas Stern’s Review of the REF (see:#SternReview) [i]. This was heartening, chiming with my previous post ‘We need to talk about research impact (again)’ on the need for ‘robust discussions’; a refrain I will seek to expand upon in future pieces [ii]. The report presents a considered and balanced perspective, seeking to develop the well-regarded aspects of REF2014, whilst addressing the three blights of disciplinary siloing, resource burden, and permissible yet unprincipled ‘gaming’. In what follows, I consider the wider canvas upon which the report paints, interspersing this with observations on the structure of the proposed REF2010 ‘impact environment’. In a follow up post, I’ll build on these points, considering how the report seeks to reconfigure impact case study submission, and how this may have knock-on effects with respect to how Higher Education Institutions (HEIs) manage, promote, and report knowledge exchange.
Full submissions and portability
The beating heart of the public debate was Recommendation 3 “Outputs should not be portable [rather, tied to] the institution where the output was demonstrably generated” (para. 73); thus bringing outputs into line with existing rules on HEIs claiming impact. A clear worry was rapidly identified, that this would remove one obvious source of leverage still available to early career researchers in a tight and often volatile job market. This concern was tied to a series of parallel discussions around Recommendation 1 “All research active staff should be returned” (para. 65) and the HR manoeuvrings this could unleash in deciding who counts as ‘research active’ (and indeed what counts as ‘research activity’). This has the potential to usher in new modes of marginalisation, and as Richard Watermeyer suggests this “profound problem… could end up diminishing what universities recognise as the role and contribution of the researcher: primarily, the successful procurement of research funds and prominence.” Nevertheless, it is likely that the devil will be in the detail, with the work of high performing academics – who may be able to submit as many as six outputs – conceivably mitigating the risk of including a ‘long tail’ of less prolific researchers [iii]. Much will rest on how individual HEIs establish and interpret ‘who should be doing what’ within given research units, however configured. I will not dwell on these points, except to note that they have been thoughtfully covered by others, including Athene Donald, Martin Eve, Adam Golberg and Liz Morrish [iv].
Enhancing impact
Mark Reed has provided a sympathetic summary of Stern’s impact-focused recommendations, arguing that the report successfully addresses a perceived narrowing of impact that occurred in the post-REF2014 period. To those working in an impact support role this ossification was always largely a matter of perception. Nevertheless, as a contribution towards a concerted ‘push-back’, Stern is of considerable assistance. As Reed notes “it was not HEFCE who constrained the definition of impact; it was the way that the academy interpreted HEFCE’s very broad definition in strategic and instrumentalist terms.” It is probable that the way the REF was/is managed within HEIs adds to this hive of semantic circumspection. One would therefore hope that whilst Stern maintains that “all panels should have the same broad approach to impact”(para.82), in the second iteration of the REF the individual sub-panels – the“number and shape” of which Stern argues “was about right” (para. 63) – are allowed freer rein to define what impact is ‘for them’. The indications, which I will cover in my next post, are that a reassertion of the value of impacts on cultural life, via public engagement, and through pedagogy will bolster this fresh optimism. One is also left with an irony, in that more prescriptive guidance could ‘lock in’ a broader overall interpretation of how research impact can and should be presented.
Unifying templates
Perhaps the area of most considerable interest is Stern’s recognition of the oft-touted suggestion to merge the environment and impact templates. In one move this increases the value of impact without changing the 65%-20%-15% structure of the REF (impact cases studies now worth 20% of an HEI submission, rather than 16%, with the combined templates remaining at 15%). This sits alongside proposals to introduce institutional as well as Unit of Assessment (UoA) Environment templates, removing a previously unavoidable layer of repetition (and institutional boilerplate), and establishing a means of assessing “steps taken to promote interdisciplinary and other joint working internally and externally and to support engagement and impact, beyond that which is just the aggregate of individual units of assessment” (para.88). The proposal to allow the (tick-box) identification interdisciplinary outputs, as well as document the role of ‘interdisciplinary champions’ (para. 100) – whilst probably not assuaging critics such as Derek Sayer – are also clear moves in the right direction. Besides, this latter suggestion will likely provide a welcome springboard for public engagement champions, and those in similar, often semi-official HEI roles.
The writing was on the wall for the impact template, the moment the HEFCE commissioned RAND review detailed respondents who “spoke of the ‘fairy tale’-like nature of the impact templates, which they felt to be ‘a cosmetic exercise’… Whilst the information the template provided was good to have, there was no way of verifying claims that were made without having a site visit, and there was no confidence that the impact template reflected reality” [v]. As an impact manager, one might also view this tweak as a barely concealed attempt to make the REF ‘UKRI ready’, actively fulfilling the need expressed by RCUK to move knowledge exchange out of the periphery, “embedding throughout the research base a culture in which excellent research departments consistently engage with business, the public sector and civil society organisations, and are committed to carrying new ideas through to beneficial outcomes”. On a practical level outlining a UoA’s research and knowledge exchange trajectory within a unified document is also manifestly more straightforward to execute. As Stern notes:“impact and environment should be seen in a more integrated way and at a more institutional level… becom[ing] more strategic and forward looking whilst retaining a strong evidence base in past performance” (para. 126). Come the forthcoming consultation, a key point of contestation will likely be around the credit split between institutional and UoA components, the report only hinting at the discussions to come: “a share of QR funding should be awarded to the institution based on its Institutional Environment statement and the institutional-level impact case studies which it submits. This innovation will require careful testing and we recommend that the funding bodies explore options for piloting the institutional level assessment to test this proposal” (para. 91).
Looking back, looking forward
An allied issue will be how, without increasing the burden [vi], the new template structure can adequately assess “the future research and knowledge exchange strategy of the HEI, as well as the individual Units of Assessment, and the extent to which both have delivered on the strategies set out in the previous REF” (para. 88). In seeking to collapse institutional boundaries, via a two tier submission, opportunities open up for a more rounded expression of an HEI’s medium to long-term aims and objectives. However, at the same time this affords an opportunity to senior managers, who may seek to actively dismantle and distance themselves from existing institutional plans, as the refreshed REF itself becomes a strategic determinant over the preceding four year period. In a surprising – but not unwelcome – sojourn into detail, the report goes as far as proposing fourteen ‘headers’ for the Institutional and UoA Environment statements, including brief mention of ‘progress’ and ‘strategic plans’ previously outlined (para. 94). This could be beefed up, asking HEIs to reflect in more direct and tangible terms on existing – and one might add publicly available – ‘research strategy’ and ‘(impact) strategy and plans’, as contained within REF2014 submissions [vii].
Data sharing
How data is shared across the research system feeds into the strategic benchmarking, implied above. Stern recognises this as a deep-seated challenge, advising that “attention will have to be paid to the quality and comparability of databases… an issue which applies for the sector as a whole and the new UKRI”(para. 126). In the current system the trail from source funding, via published research towards consolidation within a REF impact case, is somewhat haphazard. A recent report from the BBSRC exemplifies this predicament, citing“a disparity between the institutional distribution of BBSRC research funding and the distribution of [REF] case study acknowledgments… a result of local expectations and practice, driven by researchers and research managers drafting case studies” [viii]. Put simply, for the purposes of accountability different parts of the research system require different forms of output (evidence), each founded upon different forms, or configurations of input (resource). In untangling this, and generating a solution that works for all parties including central government, the report is vague, noting only that HESA will be consulted with respect to numbers of outputs required per UoA (para. 70) [ix]. However, the general thrust of the report clearly fits with a stated desire that UKRI and its board should use REF as a strategic catalyst, a debate re-hashed from the years preceding REF2014, where harried HEI strategy departments wondered aloud what exactly David Willetts was going to ‘do’ with c.7,000 impact case studies (answer: not a lot). As Ant Bagshaw, writing for Wonkhe maintains, this premise “is rather cheeky both in its tasking of UKRI to be more ‘imaginative’ – the body doesn’t exist yet… also it’s a clear request for cash[from]… researchers with the authorial paws on the document… In the Brexit context… that request is all the more important.” Whatever the solution, for good or ill more effective data-driven benchmarking could be utilised as a means to ‘efficiently’ exclude some HEIs from funding schemes under the UKRI umbrella (as per the DTC model). This is a point one can be sure the established HEI associations will foreground within written submissions to the forthcoming consultation.
A conclusion, and a beginning
The report was notable in that, akin to an obverse King Canute, Stern was able to drive back the metric tide purportedly backed within some governmental circles. In line with the HEFCE sponsored report, led by James Wilsdon, Stern calls for a responsible approach: “Panels should set out explicitly how they have used bibliometric data in their working methods” (para. 76). It might be for consultation respondents to press on how panels should set out, in advance, how they will use research metrics. As Wilsdon notes, in an excellent summary of the report’s key judgements, Stern’s approach “maintaining the primacy of peer review, using carefully-selected metrics in the environment section of the REF, and improving data infrastructure and interoperability – is completely in line with the findings in The Metric Tide. And a new Forum for Responsible Metrics, involving all the major research funders, will take forward the detailed work needed to get this system up and running for REF 2021.” Any pre-review of the use of metrics would be a key task for the forum, not least as the door has been left ajar for some limited sampling if “subject panels are able to make the case, explicitly supported with reference to robust evidence, that bibliometric data could be used to reduce the workload” (para. 71).
Overall, one is left with a sense that REF2021 will see a greater percentage of research, researchers, and research outcomes submitted within discrete and focussed – rather than strategically engineered – returns; all supported beneath an interdisciplinary superstructure [x]. It is indicated that “by the end of the year a formal consultation should be issued so that the community can offer their views on the proposed process and the future REF formula. The decisions arising from this consultation should be published in the summer of 2017” (para. 117). One imagines that after a brief flurry of commentary, a significant amount of covert legwork will take place, as stakeholders positon themselves for the horse-trading that will follow.
Notes
[i] To give the report its full title “Building on Success and Learning from Experience: An Independent Review of the Research Excellence Framework.”
[ii] Alas, my call for a blanket ban on the word impactful fell on deaf ears – albeit one solitary appearance, in the call “to make the UK research base even more productive and impactful” (para. 59).
[iii] Whether researchers can be submitted with zero outputs, remains a point for discussion. It is projected that UoAs will be required to submit “two outputs on average per submitted full-time equivalent (FTE) individual” (para. 67).
[iv] With every indication that the ‘Stern’ puns will not be going away at any time soon.
[v] Manville et al (March 2015) Evaluating the Assessment Process for the Impact Element of Research Excellence Framework 2014, RAND Europe (p.38)
[vi] The word ‘burden’ appears 29 times, possibly the most instances per page of any published work since ‘The Pilgrim’s Progress’.
[vii] Sections within the REF2014 environment template (section b) and impact template (section c), respectively.
[viii] Digital Science (July 2015) REF 2014 Impact Case Studies and the BBSRC(p.2)
[ix] Whether this means UKRI will seek to replace Researchfish, will presumably be up for discussion.
[x] Marxist wordplay, partly intentional.