COMPare aims to fix the prevalent and ongoing problem of outcome switching in clinical trials. In short, outcome switching is when trialists report something different to what they originally said they were going to: it increases the risk of false positive results and exaggerated findings, and is therefore a bad thing.

For 6 weeks we assessed every trial published in the top 5 medical journals for discrepancies between pre-specified and reported outcomes. Of 67 trials, we found that 58 (87%) contain misreported outcomes [1].

That was phase 1 of COMPare. We are now in phase 2: blogging on our findings and engaging the journals and authors in public discussion on our specific assessments and their attitudes and policies on outcome reporting. The range of responses this has generated has been fascinating and enlightening; from best practice demonstrated by the BMJ [2], to extremely concerning views expressed and actions taken by Annals of Internal Medicine [3]. We are beginning to see a set of recurring themes in these responses that shed light on the underlying reasons for this prevalent problem; largely based on a fundamental misunderstanding of the importance of pre-specification and correct reporting of outcomes.

Responses from the authors and Annals’ editors on our analysis of the ATLAS trial are a great example of the problems we have uncovered. We assessed this trial back in November, found a range of problems in the reporting of its outcomes, and submitted a letter to Annals to correct the record. You can see our full analysis here, but in summary: the primary outcome was only reported for 1 of the 3 pre-specified time-points, only 6 of 17 pre-specified secondary outcomes were reported, and lots of new analyses were reported without being declared as non-prespecified. If you’re still unsure about why this is bad, read our website, our protocol, and this great xkcd cartoon.

Following our letter on the Annals website, the lead author of ATLAS, Dr Hugh MacPherson, posted a response, in which he demonstrates a number of misunderstandings of the problem and our methods:

When we registered our primary outcome measure, the Northwick Park Neck Pain Questionnaire, we specified that it would be ‘measured at baseline, 3, 6 and 12 months’. The authors of the Comment incorrectly stated that there were ‘3 pre-specified primary outcomes (one score at 3 timepoints)’

The Registry entry does not ask for a primary end-point.

In the CONSORT guidelines [4], an adequate specification of an outcome includes the time-point; so, if you have 3 different time-points, this is 3 different outcomes. The registry entry does ask for a primary “end-point” (time-point), simply by asking for the primary outcome.

The distinction between “end-points” and “time-points” is not one that we recognise as widely accepted or used: a trial’s registry entry must contain a time-point for each outcome in order for those outcomes to be adequately defined. We are aware of no tradition of pre-specifying three time points, but silently nominating one as the most important, without noting or registering this anywhere. We would be interested to hear from any researcher who disagrees.

In our protocol, published 2 years before the main trial results, we specified that the 12 months outcome on this questionnaire was our primary outcome end-point. Hence, the 12 month end-point was the one we reported as the primary end-point in our published paper. It is important that the primary outcome is identified in advance of analysis of the main trial results to avoid ‘cherry picking’ trial outcomes: and this is what we did.

To reduce the risk of selective outcome reporting and false positive results, all primary and secondary outcomes must be specified before trial commencement, as stated in the CONSORT guidelines [4]. Dr MacPherson is mistaken here in thinking that pre-specification “in advance of analysis of the main trial results” is sufficient. The protocol cited in Dr MacPherson’s comment [5] was published over a year after trial commencement and therefore cannot, by definition, contain pre-specified outcomes.

Of the 17 secondary pre-specified outcome measures, all of them have been or will be reported in the primary paper or in secondary papers that are currently being prepared. These papers will report on the as yet unpublished outcomes that are part of our pre-specified mediator-based analyses that have been described in detail in our published protocol.

Here Dr MacPherson argues it is acceptable for 11 of the 17 pre-specified secondary outcomes to be left unreported because they will be reported in a subsequent publication, and we absolutely agree. However, as CONSORT states, this deferral must be declared in the trial publication. This is so that readers know that the reported outcomes represent only a subset of those pre-specified, and that those remaining will be reported elsewhere. It seems Dr MacPherson and Annals’ editors share this view, because the ATLAS publication [6] states:

Dr. MacPherson (the manuscript’s guarantor) affirms that … any discrepancies from the study as planned (and, if relevant, registered) have been explained.

As we have shown, from comparing the publicly available registry entry and Annals publication, this was not done in the ATLAS trial report, and these discrepancies went unexplained. To reiterate: Dr MacPherson agrees, in his response to our Annals letter, that there are discrepancies from the study as planned which were not explained in the publication; but guarantees in the statement above that all such discrepancies have been explained. He lastly argues:

While we support the CONSORT guidelines on best practice in trial reporting, we are concerned about the ambiguity of Registry entry labelling and scope for misinterpretation either by researchers when entering data at the outset of a trial, or by commentators, as is the case we discuss here.

This is a deeply worrying view of trial registries. Their purpose is to prospectively define, in the public domain, key information on each clinical trial, including its pre-specified outcomes. Trialists and journal editors are responsible for ensuring that information in trial registries and protocols is not “ambiguous”. Where outcomes are poorly prespecified, results could be selected post-hoc from a range of possibilities: this potentiates selective reporting of outcomes, and allows serious distortions in the evidence presented.

We responded in full to the Editors’ response in a previous blog post; our submitted online comment was rejected on the basis of a 400 word limit, although the letter from Annals’ editors to which we were responding was 850 words long. In addition, Annals have said they will not publish our responses in the journal, although they have published their own 850 word letter in full, in the print edition.

We think this is unfortunate. The misunderstandings expressed by authors and Annals editors described above help explain why outcome switching continues to be so prevalent in leading journals, despite claims by editors that they endorse best practice. These issues are vitally important to the accurate reporting of trial results and can only be resolved through clear public discussion.

Henry Drysdale, Ben Goldacre, Carl Heneghan, on behalf of the COMPare team

 

References:

[1] The CEBM Outcome Monitoring Project, www.COMPare-trials.org, last accessed 29/01/2016

[2] Correction on the REEACT trial, published 12/01/2016: https://www.bmj.com/content/352/bmj.i195, last accessed 20/01/2016.

[3] Annals Editors, Discrepancies Between Prespecified and Reported Outcomes, Ann Intern Med. Published online 22 December 2015

[4] Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869.

[5] MacPherson H, Tilbrook HE, Richmond SJ, Atkin K, Ballard K, Bland M, et al. Alexander Technique Lessons, Acupuncture Sessions or usual care for patients with chronic neck pain (ATLAS): study protocol for a randomised controlled trial. Trials. 2013 Jul 10;14(1):209

[6] MacPherson H et al, Alexander Technique Lessons or Acupuncture Sessions for Persons With Chronic Neck Pain: A Randomized Trial, Ann Intern Med. 2015;163:653-662

[7] Annals Editors, Discrepancies Between Prespecified and Reported Outcomes, Ann Intern Med. Published online 22 December 2015.