Here we publish some correspondence with JAMA, who have declined to correct the record on a series of trials that they have misreported in their pages.

First, one paragraph of background, for those unfamiliar with the problem of outcome switching, and COMPare’s efforts to address it. It is well established that academic journals routinely permit outcome switching in the trial reports that they publish, despite public commitments to address this problem. This is widely recognised as bad, for reasons we have already outlined. For six weeks from October 2015, the COMPare team analysed every RCT published in the top 5 medical journals to check if they had correctly reported their pre-specified outcomes. Where we found discrepancies between the outcomes that were pre-specified and those reported, we wrote a letter to that journal to correct the record. Our intention was to test whether the structures of science are working as they should, and specifically to see whether academic journals are transparent and self-correcting in the face of legitimate concerns being raised about misreporting of trial results. The responses we have received from journals have been extremely varied: from full public engagement and transparent corrections in the BMJ, to inaccurate and concerning responses from the Annals of Internal Medicine and the New England Journal of Medicine.

JAMA is a very widely read journal, with an impact factor of around 35. So far we have analysed 13 of their trials. Of 22 primary outcomes pre-specified across these papers, 17 were correctly reported. However, we also found 87 additional outcomes reported that were not pre-specified in the protocol or trial registry, and only 34 of these were declared as novel. Furthermore, 70 of the 105 pre-specified secondary outcomes spread across these publications were never reported. Our full results and raw data are available here. Only 2 of the trials we assessed from JAMA were perfectly reported.

[Read more…]

Over 6 weeks the COMPare team analysed 67 trials in the top 5 medical journals to assess the ongoing prevalence of outcome switching in clinical trials, and to find out why it persists. Our methodology can be found here. In this series we will be highlighting the salient lessons learned from this process by describing the analysis of several trials in detail. Each study highlights different ways in which outcomes can be misreported, leading to undetected flaws in the evidence presented in clinical trial reports.

Here we describe our coding of the ERASE trial [1]. The context and overview of this study is described in this editorial. This study was 1 of 13 randomised control trials published in JAMA that we analysed. Of these, 11 contained outcome switching and required a letter of correction. All were rejected by JAMA’s editors; more on that, including full correspondence with JAMA’s editors, here.

 

Our summary analysis of outcome reporting in the ERASE trial was that:

3/3 pre-specified primary outcomes were correctly reported

21/37 pre-specified secondary outcomes were correctly reported

16/37 pre-specified secondary outcomes were not reported at all

6 additional outcomes were reported that were not pre-specified

 

[Read more…]