Environment assessment should gauge progress on careers, integrity and more, say Tanita Casci and colleagues
The UK government’s R&D People and Culture Strategy published in July 2021 recognises that the environment for research is central to its reliability, quality and societal benefit. The strategy pledges to “ensure that the Future Research Assessment Programme builds on the ambitions and actions set out in this strategy”.
That assessment, in the shape of the Research Excellence Framework, currently comprises three pillars: outputs, impact and environment. The REF, which drives up to about 10 per cent of university income, has its downsides, bureaucracy included. But it has enhanced the UK’s international competitiveness by encouraging a focus on quality rather than quantity and establishing the research impact agenda.
With regards to research culture, the effect of the REF has largely been restricted to promoting open access and equity, diversity and inclusion (EDI). Fundamental as these two domains are, other aspects of research culture were too loosely specified to drive focused improvements.
There is now a major opportunity to leverage the influence of the REF over research policy and practice to improve research culture. We believe the REF environment template should be remodelled to help it do so.
What ‘good’ looks like
Focusing on the research environment matters because outputs and impact alone fail to capture the contribution that institutions make to the research community, such as providing skilled people for the workforce within and beyond academia. How an organisation supports and develops people and careers should be assessed with the same rigour that is applied to outputs and impact.
In assessing the research environment, the challenge is to define what ‘good’ looks like while allowing a range of organisational and disciplinary profiles. Here, we could learn from experience, applying a blend of qualitative (narrative) and quantitative indicators in a similar way to that used when assessment of impact was introduced in REF2014.
Environment assessment could be structured around existing national concordats and sector agreements under the following five themes: careers; integrity and open research; EDI; responsible research assessment; and impact and knowledge exchange.
Longitudinal and benchmarked data on these could be put into context by a narrative statement. This would allow institutions to set out their position and direction of travel, and disciplinary units to show how they have contributed to, and benefited from, their institutional environment.
Universities UK, UK Research and Innovation and the Wellcome Trust are currently reviewing the impact and adoption of research concordats and agreements. The outcome should feed into the REF and help build consensus on what constitutes a good environment. Whatever assessment criteria are chosen, they must allow review panels to discriminate across different quality standards.
Different levels
At the basic level, an institution would provide evidence of minimum assurance standards. For research integrity, for example, this would point to public information on basic compliance with the concordat: the identity of the named person, the policy for handling concerns, and an annual integrity assurance report.
At a more mature level, assessment would consider whether an institution has a training structure to disseminate best practice. To achieve a top mark, institutions and units could highlight how they have contributed to national and international best practice. As the standard increases, the weight of evidence would shift from data to narrative.
On open research, an institution could return data on the fraction of outputs that include statements on author contribution and data availability, supported by a narrative describing institutional support for these practices.
Taking career development as a third example, the Researcher Development Concordat already commits universities to providing staff with at least 10 days of continuing professional development training each year. An environment submission could include a description of CPD opportunities, with data on the level of engagement, standardised staff surveys and career destinations.
We are mindful of the need to reduce bureaucracy. Any addition to the REF process requires a reduction of burden elsewhere. This could be achieved by basing data submissions on existing returns to the Higher Education Statistics Agency and others, increasing consistency in the process.
HESA data, for instance, already include a count of doctoral degrees awarded. Supported by a narrative and data on destinations, this would give insight to the vitality of provision for research students.
Looking forward to the next REF, it is exciting to think that the improvements in outputs and impact achieved thus far could be mirrored by improvements to research culture and the careers that universities enable. The environment pillar has great potential to drive these improvements—it’s time to harness that potential.
Tanita Casci and Miles Padgett are at the University of Glasgow. Grace Gottlieb and David Price are at University College London.
A version of this article also appeared in Research Fortnight