The challenges of scientific research evaluation

Evaluation of research conducted by academic institutions was one of the topics addressed during the fourth REHES workshop, Research on Higher Education and Science in Switzerland, which brought together some 50 scholars at Università della Svizzera italiana to discuss the challenges of the higher education system.
In particular, the focus was on the impact of evaluations carried out outside the scientific community, mainly those carried out by governments to distribute funds or to accredit institutions of higher learning.

We talked about this with Emanuela Reale, director of the Research Institute on Sustainable Economic Growth of the Italian CNR, who, together with her colleague Antonio Zinilli presented a study conducted on Italian universities. The starting point is the juxtaposition of two ideas of research quality in the scientific literature on the topic. "The first is the one that researchers construct within their community of reference," explained Professor Reale. Sociologists construct their idea of quality within the sociology community, biologists within the biology community, and so on. "It is a homogeneous idea because it refers to the same kind of knowledge, methodologies and procedures." The second idea of quality, on the other hand, is that generated by governments, which, as an entity outside the scientific community, evaluate research according to specific standards and indicators. Tension is thus created between two different evaluation systems, and the purpose of the research presented concerns precisely the effects of this tension. "Will researchers adjust their idea of research quality by looking at the indicators chosen by politics, or will they follow a double track? What we have observed is that in some scientific communities, this effect is powerful: particularly the humanities and social sciences communities react to external evaluations because the indicators proposed by the government conflict with their research practice." Thus, research evaluation leads "to transformative change, although whether for better or worse remains to be seen: the recommendation that comes from our research is that a government should be extremely cautious, in its use of the evaluation tool, because it is not irrelevant," concluded Professor Reale.

Professor Reale added that policymakers bring their idea of quality through different tools. The main ones are competitive funding mechanisms. Another widely used in some European countries is the massive evaluation of universities whose results are linked to resource allocation or accreditation of PhD courses. Both have also been adopted in Switzerland, albeit in a very moderate form, added Professor Benedetto Lepori, organiser of the talk.
Professor Reale also emphasised that we need to question how much this massive evaluation needs to be transformed. For example, a recent activity promoted by the European Commission proposes revisiting these criteria to be more adherent to the goal of open science. Since evaluation practices have an effect, they can be used to direct research toward virtuous behaviour. The government's idea of quality could be changed to be closer to the scientific community's. Moreover, it is unthinkable to do only qualitative evaluation: all evaluations have quantified elements of observation of the phenomena they reveal. But these methods, Professor Reale added, must fit into a theory-based design.
These remarks, Professor Lepori concluded, underscore the issue's relevance at the policy level and for the universities themselves.