Unanswered questions and uncertain trade-offs leave much still up for grabs, says Stephanie Smith
Fortune favours the bold. That seems to be what the team running the Research Excellence Framework was thinking when it launched its initial decisions on REF 2028 on 15 June. Backed up by a two-year consultation process and an international panel of experts, no one could accuse the next REF of embracing incremental reforms only.
But changing the REF is tricky. With over £2 billion of funding at stake each year, every decision seems to involve a Byzantine set of trade-offs, and every change becomes an exercise in balancing the potential for game-playing and burden against the potential benefits of reform.
This REF will be no different, but the prize could be well worth it. No one will disagree that a strong research environment and culture is needed to support research excellence, or that universities must do more to recognise the wider teams responsible for underpinning this work.
The academic community has long struggled with the issue of how to reward the creation of a broader range of outputs, such as systematic reviews and datasets, vital to progress in research, while also making it easier to move between academia and industry. This REF valiantly attempts to address all these issues, while also reducing burden where it can.
Best laid plans
Many previous potential reforms to the REF have come a cropper as laudable aims—such as a fully automated evaluation—have bumped up against the reality of whether changes could be implemented without making the assessment less legitimate or robust.
This time around, the desire to completely break the link between individuals and their outputs could be one such early casualty.
The logic of de-coupling researchers from their outputs makes sense. The REF seeks to assess institutions, not individuals, and the requirement of one output per researcher is seen to hinder movement between academia and industry. But there are downsides.
As has been widely noted, the removal of the range of 1-5 outputs permitted per full-time equivalent in REF 2021 could allow institutions to rely on a handful of research stars, while neglecting the development of other staff, who previously had to be supported to produce at least one output in seven years.
This would run counter to REF 2028’s focus on research culture. But while units will be required to explain how their submitted output pool is representative of their research and researchers in their area, the crucial bit—how this can be evidenced—still isn’t known and will be up for consultation.
If we don’t know where outputs are coming from, and people move around without a census date, this could also increase the risk of outputs being double-counted and could revive the ‘REF poaching’ of top academics between universities seen before the 2014 assessment.
Outputs downgraded
Everything in the REF is a trade-off, of course. To raise the profile of research culture, the weighting of the evaluation of contribution to knowledge and understanding—outputs in old money—will fall to 50 per cent.
This section will also include a new structured explanatory statement for each unit of assessment, accounting for “at least” 10 per cent of the score. So the assessment of research outputs, as currently undertaken, will make up 45 per cent of the total REF assessment at most.
In other words, the next REF will be the first in which outputs account for a minority of the total mark. Given the high stakes, it’s inevitable that such bold changes will be pored over.
If the new people, culture and environment element is going to be taken from 15 per cent up to 25 per cent, then the assessment and metrics underpinning it will have to be credible. Here, too there are no easy answers.
After two years of consultation on the data or indicators that could be used to differentiate between institutional performance, the REF team has conceded that additional consultation is still needed. Further work will be commissioned this summer on how to approach evidence-gathering in this area.
Indeed, the list of questions in Annex A of the report suggests that many of the answers on REF are still a way off. The consultation on how to approach these tricky trade-offs is now open until October, with more workshops likely to come. If we are going to stick to the timetable, which sees draft guidance published by this time next year, the sector will need to get its skates on.
Stephanie Smith is interim deputy director of policy at the Russell Group
A version of this article appeared in Research Fortnight