Retraction of scientific papers for fraud or bias is just the tip of the iceberg

Taking a closer look at the details. Small print by Shutterstock

Publishing clinical trials in medical journals can help doctors and scientists rise through the ranks of the research hierarchy. While most play the publication game fairly, some cheat. Whereas all misconduct undermines the public’s trust in science – such as the recent retracted paper about gay canvassers – health research scandals put the health of millions of patients around the world in jeopardy.

Professionals and patients depend on results from systematic reviews of clinical trials, which evaluate all the evidence on a particular issue, to know whether or not treatments are safe and effective. However, those of us who coordinate the preparation of Cochrane systematic reviews of treatments for seriously injured patients believe that these types of reviews can no longer be entirely trusted because of research misconduct and publication bias. An argument we recently made in the BMJ.

Falsified reports

Most medical journal editors and systematic reviewers take clinical trial reports at face value with little or no effort to confirm whether a particular trial even took place. A Cochrane systematic review showing that the infusion of high-dose sugar solution prevents death after head injury, for example, was later retracted after our review editors were unable to confirm that any of the included trials took place.

As part of the investigation, the London School of Hygiene & Tropical Medicine editors contacted the editor of the journal that published one of the doubtful trials. His response? That there had been doubts about the data but that doubts were difference from concluding that it was fabricated.

Similarly, the conclusions of a review of starch infusions in critically ill patients changed substantially after excluding seven entirely fabricated trials by Joachim Boldt.

Investigating fraud is hard work, and it is easier for journal editors to ignore the problem and perpetuate the myth that peer review of trial reports ensures their scientific quality. But how can systematic reviews claim to provide “trusted evidence” when all evidence is taken on trust?

Bias in reviews

The second major problem is that the medical literature contains a biased sample of clinical trials. Clinical trials showing that a particular treatment is effective are much more likely to be published than those showing no benefit or harm.

As a result, systematic reviews based on published research are biased – they emphasise the positive and eliminate the negative. Despite decades of exhortation about trial publication, around half of all clinical trials remain unpublished – and so even the most diligent efforts to synthesise the results from all relevant clinical trials are in vain.

Cherry picking the data. Cherries by Shutterstock

Even when trials are published – information on side effects is often neglected from published reports. Clinical trial information is too important to depend on the publication game – after all, patients’ lives depend on it. Reviews of research evidence should be based on the actual trial data obtained from compulsory clinical trial registers. These registers would specify in advance the information that is to be collected in the trial – and when the trial was completed they would provide the actual raw data. Providing all the information collected would avoid the bias associated with publishing only some of the findings.

The public need to know what the trial found rather than a given doctor’s spin on the results – usually designed to make results look more interesting or newsworthy.

However, medical journals and publishers of systematic reviews have no incentive to change. They make good money putting a sophisticated gloss on whatever research manuscripts are available to them. When fraud scandals break they throw up their arms in horror but there is no structural change.

Although clinical trial research is heavily regulated, fabricated research is completely immune to this regulation since all it requires is a doctor who has lost the plot and has access to a computer. There is no shortage of medical journals who will publish – and with no questions asked. Progress will only be made when the general public becomes involved.

Accurate health research information is a public good that is as important as safe clean drinking water. We need a health research information system that delivers pure research information that is untainted by the career interests of scientists or the commercial interests of publishers.