A one-size-fits-all approach to research evaluation helps exploitative publishers, warns Emanuel Kulczyck
Research evaluation and scholarly publishing are two of the dominant issues in European research policy. As the February announcement of the Paris Call on Research Assessment shows, there is a strong desire to move away from a publication-oriented research culture and from highly metricised systems towards recognising quality and diversity.
The risk, though, is that the result of this effort to reform and harmonise evaluation will be to take the norms and processes of older, ‘central’ nations with strong research systems and impose them on countries, research systems and disciplines ill-suited for such approaches.
To see the pitfalls of such an approach, one need only look at the national systems that have already taken it. Of the perverse incentives and unintended consequences created in the process, the most striking is the interaction with predatory forms of scholarly publishing.
The debate on predatory publishing focuses almost entirely on English-language journals published in non-English speaking countries. Lists such as the discontinued Beall’s List or Cabells’ Predatory Reports judge journals on their editorial practices and divide the world into goodies, which are published mostly in central countries in English, and baddies, published mostly in English in countries ‘semi-peripheral’ to the global research system, such as India.
This is a false dichotomy. There are many bad journals with aggressive business models in the centre and many good journals published in English and, primarily, local languages at the edges. And some editorially reputable journals from large commercial publishers have predatory business models.
Mislocated centres
To get away from an oversimplified view of scholarly publishing, my colleagues and I have coined the term ‘mislocated centre of scholarly communication’ to describe some publications, without passing judgement on their publishers or authors.
Mislocated centres emerge because peripheral nations take their criteria for legitimacy from the centre while being uncertain about what counts as central and what does not. National research policymakers mistakenly perceive some journals as connected with the central countries and institutions and include them in evaluation regimes. This drives researchers in these nations to publish in these journals.
Strange publishing phenomena can result. In 2018, for example, Elsevier’s Scopus database ceased indexing the European Research Studies Journal, published by a Greek association, due to “publication concerns”. In many researchers’ eyes this causes the journal to lose its bibliometric legitimacy.
In 2019, however, Poland’s national evaluation regime recognised the journal, and Polish authorship skyrocketed. In the first issue of that year, one of 17 papers had a Polish author; in 2020, the journal published three special issues, each containing over 95 per cent Polish authors.
The journal’s publication fees range from €600 to €1,000, depending on the discounts granted. Its delisting by Scopus might raise concerns in some quarters. But the Polish evaluation system, which aims to move the country closer to the centre of the research world by encouraging English-language publication, has driven Polish researchers into its pages while undermining a strong tradition in Polish-language open access publishing. This pattern is repeating globally.
Many peripheral scholars are pushed towards mislocated centres, or actively participate in their creation. Most scholars who publish in mislocated centres do so not by accident or to deceive, but because they are working in an unequal geopolitical system of publishing.
Predatory journals threaten the integrity of, and public trust in, research. But the answer is not ‘educating researchers’ out of using them. It is reforming the incentives that drive researchers to them.
A good approach would be to give more recognition to local-language publications, and to focus more on quality and less on quantity in the assessment of individuals. This could be achieved through narrative-based CVs that, instead of showcasing publication counts or bibliometric indicators, invite researchers to show how they have contributed to ongoing debates, societal challenges and the development of the research community and broader society.
More generally, the EU needs to be more alive to the academic traditions of all member states, and avoid policies that impose a single idea of excellence on R&D systems. Member states in Eastern and Southern Europe also need to engage more with EU efforts to reform research evaluation. It will not be possible to create a new evaluation system tailored to such nations without their participation.
Emanuel Kulczycki is head of the Scholarly Communication Research Group at Adam Mickiewicz University in Poznań, Poland
This article appeared in Research Europe