By Olavo Amaral
How do you deal with data that is too good to be true?
*
The Covid-19 Early Treatment Choir was hit two weeks ago with news that an Egyptian clinical trial showing ivermectin’s efficacy against the disease had been pulled from Research Square’s preprint platform. The study showed a 90 percent reduction in mortality in patients with severe illness compared to a group who received hydroxychloroquine.
Many people had not taken the work seriously from the start, either because it was done by obscure researchers, because it was written in poor English, or because it produced a result too spectacular to be true. However, none of this prevented them from being included in several meta-analyzes defending ivermectin, which were responsible for much of the beneficial effects seen in them.
The article was only withdrawn from circulation after the English journalist Jack Lawrence decided to investigate it when he noticed signs of plagiarism. One version of the article contained a link to the original data – subject to a charge and password-protected. By chance, Lawrence stepped on a creative little “1234” and watched the Excel spreadsheet materialize on his screen with the raw data.
After that, the work of “data policeman” Nick Brown revealed not only inconsistencies, but also strong evidence of fraud: several patients appeared to be clones created by copy and paste, with some data modified for camouflage. As a result, the platform removed the article and the authors have yet to respond.
The story is graphic to analyze another case that has gained prominence in the Brazilian media. In March, a team of researchers led by endocrinologist Flávio Cadegiani reported at a press conference on spectacular results from proxalutamide, an anti-androgenic drug originally developed to treat prostate cancer that would have resulted in a 92% reduction in mortality among hospitalized patients with Covid – 19 .
The degree of success soon caught the attention of critics, who called it unlikely. The delay in the publication of the data (which did not appear as preprint until more than three months later), the high mortality in the placebo group, the meteoric recruitment of more than 600 patients in less than a month, and indications of deviations from the protocol of approved by the ethics committee.
However, some of the skepticism stems from factors unrelated to the data. Cadegiani had been claiming positive results for early treatment with hydroxychloroquine, ivermectin, nitazoxanide and dutasteride, and proxalutamide, even in outpatients, since the pandemic began – a success that is at least unlikely. His colleague Ricardo Zimerman was a guest of the ruling party at the CPI of the pandemic and became a digital influencer in social networks and right-wing media, who made himself felt on channels such as Osmar Terra. It’s also worth noting that the president’s repeated mentions of proxalutamide are not considered an academic seal.
However, are these good reasons to turn a blind eye to a study that claims over 90% effectiveness for a disease that causes millions of deaths? After all, the Royal Society’s motto is “nullius in verba” (“in nobody’s words”): Scientific data should be more important than who is presenting it.
Judging by the reception of the article, however, the impersonality is at a low point. In an article in Science magazine, cardiologist and digital medicine guru Eric Topol says the results are “too good to be true” and that “there are almost no interventions in the history of medicine with benefits of this magnitude.” The same article mentions that the New England Journal of Medicine rejected the article on the grounds that “the results are unexpectedly good,” which would result in the need to review the primary data – which the journal claims unable to do being.
After its reputation was exposed in the Surgisphere scandal, it is understandable that the New England Journal would not want to take chances with articles that arouse suspicion. Still, the heuristic of the decision doesn’t seem to justify – as does the claim that the world’s largest medical journal is unable to verify the original study data that Cadegiani claims to have presented to the editor.
However, the offer does not seem to apply to everyone. Even if it is stated in the form that the data is available on a justified request, my request to receive it was answered with the answer “The authors prefer not to disclose it at the moment” – an incorrect availability, the password-protected link the file reflects Egyptian. When asked on Twitter, Cadegiani justified the refusal with “injustice in the treatment of various studies” and implied that the fact that I had not requested data from other works challenged my impartiality.
Incredibly, the refusal to provide original data from a study is a common reality in academic science. In the impossibility of accessing them, belief in the statements of an article is almost always based on the words of the authors. The words may sound like no other, but as a viral commercial from a few decades ago put it, “la garantía soy yo”. What defines the speaker’s reputation, and a great deal to decide what to believe.
With that, the debate eventually wanders into investigative journalism – or into social media, where virulent “ad hominem” arguments from both sides try to solve an insoluble problem by attacking the reputations of authors and critics. And as with any topic, in the end, each group finds the truth that suits them, which leads to polarization between doctors and laypeople.
The first step in solving the problem is obvious – anonymized data from a study should be available to anyone who wants to analyze it. While this data is typically requested by regulators and most articles claim it can be obtained, it is rarely actually available.
However, even with open dates, better than roughly copying and pasting the Egyptian article, fraud can be difficult to detect. It is therefore necessary to develop into auditing systems that make it possible to check that what is written in an article reflects reality. In a world where millions of secret votes are counted in hours, it shouldn’t be difficult to verify that people who have used a drug in a study are alive or dead. Strangely enough, however, this doesn’t seem to be a priority at the Academy, which is content with a trust-based system that ultimately sows discord.
At some point we will know if what Cadegiani and his colleagues are saying is true – Proxalutamide has been approved by Anvisa for new tests, and the Paraguayan government has given emergency approval for the drug to be used. Until then, however, we will spend several months harming thousands of people, either by withholding effective treatment or by selling the false hopes and side effects of an ineffective drug and its commercial analogues that are already being prescribed “off-label” ” in Brasil.
Both alternatives are unacceptable and testify to the failure of academic science to perform the most basic level of quality control – namely, to know whether published data is true. Something that should be everyone’s right with no passwords, research, or petitions to the authors.
*
Olavo Amaral is Professor at the Leopoldo de Meis Institute of Medical Biochemistry at UFRJ and Coordinator of the Brazilian Reproducibility Initiative.
Subscribe to Serrapilheira’s newsletter for more news from the Institute and the Ciência Fundamental blog.