By Luiz Augusto Campos
The system as a whole cannot be judged by the cases it overlooks.
*
The following article is a replica of the text “The Invisible Clothing of Peer Review” by Olavo Amaral published on June 19th.
Earlier techniques, expressions such as “peer-reviewed article” or “work available in preprint” can now be found frequently in political speeches and newspaper pages. Not even the pandemic CPI has left out these debates about the criteria for the scientific nature of a given theory. Olavo Amaral’s text on the anonymous peer review system is therefore an excellent service for science and scientific dissemination.
However, if his criticisms are almost all correct, this cannot be said of his vehement rejection of this still hegemonic evaluation mechanism in the world. I believe that this mismatch between criticism and inference can be resolved if we consider two factors: 1) Although opinions can influence the decisions of scientific journals, they are not the decision itself but act more as a subsidy for a consultation; 2) The text occupies the retail process and highlights its specific shortcomings without looking at the general implications for wholesaling.
About three centuries ago the Royal Society of London decided that the publication of scientific articles in their journal had to be approved by a committee. This is considered the starting point for the peer review system, although it has changed dramatically over time. The early texts of the Royal Society were evaluated in face-to-face meetings where staff from the Charles Darwin tribe were publicly discussed about their discoveries.
However, by the end of World War II, the scientific world had gained industrial and global importance. Although the principle of evaluation not only survived but spread around the world, its procedures have changed. In most subjects, manuscripts submitted to a journal were sent to two or three specialists whose anonymity was aimed at ensuring the impartiality of the judgment while protecting the reviewers from possible retaliation.
Opinions, however, were seen less and less as final judgments than subsidies of an editorial decision: their limitations and shortcomings can and should be arranged by editorial offices and the work of editors. It is true that appraisers are not trained for the task that is usually not paid. However, this applies to many scientific activities, from applying for funding from funding agencies to the administrative management of a laboratory.
Undoubtedly, the inherent secrecy of this verification system opens the door to dangerous deviations. Since only editors know the authors and reviewers, the scope for discretion is quite large. However, it is not necessary that the reviewers are selected randomly or that the editorial work is not checked. Scientific journals are collective companies that are overseen by councils and committees. And an important, albeit invisible part of the work of a scientific editor is precisely to establish criteria for the selection of suitable readers for certain subject areas in order to give preference to the best reviewers.
Since statements only serve as subsidies for the editorial decision, low approval rates among the reviewers are not in themselves problematic. For an editor who has doubts about the quality of a survey, the diversity of opinions is frankly positive. Depending on the editorial nature of the journal, articles that have received different ratings from different reviewers may be viewed as immature for publication as important to encourage a particular debate.
Strictly speaking, when predatory articles – meaningless, inaccurate or maliciously written, etc. – are “accepted somewhere without exception” it strictly shows that the peer-reviewed system is to keep the most questionable publications out of the most prestigious journals. Obviously the system is not foolproof (as it is almost nothing in reality). Faulty or even criminal articles have passed the review of large journals as well as alternative scientific evaluation models. With every article read, whether revised or not, we evaluate the quality of the research. Nothing prevents already published texts from being denounced and taken out of the air, it’s part of the game.
The collection of peer-reviewed articles with serious problems or incorrect dates is large and will unfortunately continue to grow. But it is infinitely smaller than good articles that are useful for scientific advancement. The system as a whole cannot be judged by the cases it misses, but by the science it produces in its entirety.
Peer review is not only reserved for science and shapes decisions in a wide variety of areas. In fact, I would not get on an airplane whose quality was measured by two randomly selected reviewers, but such an analogy is inappropriate. My confidence in flying in something heavier than air rests on the often implicit assumption that an airplane is the result of the sum of technologies and scientific discoveries, the validity of which has been proven a thousand times over at different stages of its processing. Auditing systems themselves can be read as peer review strategies, for example, as they rely on impartial specialists who are responsible for evaluating a theory or the operation of a technology.
In the six centuries of its existence, the scientific review has changed fundamentally: It has developed from public and collective judgments to anonymous and secret judgments moderated by scientific editorials. Today it is experiencing a moment of transformation with the proliferation of preprint servers that function as open social networks in which every scientist can “like” or comment on a manuscript. But these new systems seem to complement rather than replace good old anonymous peer review. To draw a somewhat struck parallel, such a review is like a representative democracy: despite their numerous shortcomings, alternatives still require more robust testing (preferably peer reviews).
*
Luiz Augusto Campos is professor of sociology at the UERJ Institute of Social and Political Studies, editor-in-chief of the journal DADOS and researcher on diversity in the academic world.
Subscribe to Serrapilheira’s newsletter for more news from the Institute and the Ciência Fundamental blog. Do you have a proposal for an agenda? This is how you can work together.