Poor outcome reporting in medical research; building practice on spoilt grounds

Published medical literature forms an ever-expanding web of information, available to patients, clinicians and all healthcare providers in everyday practice. Data from some of these publications, especially from high-impact journals, may be used to guide therapeutic decisions. Moreover, these data will be used for future studies’ design, such as for sample size calculations based on previously published size effects (1) or for conducting systematic reviews and meta-analyses, that represent the highest level of evidence in Medical Research (2). Unfortunately, a growing volume of evidence suggests high rates of poor selection and reporting of research outcomes; the reasons are either inadequate formulation of the research question or deliberate selection of outcomes presented depending on their observed results (outcome reporting bias), but in any case this is a rather worrisome phenomenon that needs to be properly addressed in order to protect transparency and reliability of health care research.

Outcome reporting in medical research and its pitfalls

A primary outcome (or endpoint) represents the specific research question each study is designed to answer, whereas secondary outcomes provide additional potentially relevant information that can help with the interpretation of the primary outcome. In practice, secondary outcomes are most often reported ‘out of nowhere’, without having been pre-specified or defined on such rigorous terms as the primary endpoint. They are to be interpreted with caution, especially in randomized controlled trials (RCTs), as by definition the study is sufficiently powered to answer only the primary outcome question. Clearly, when conducting a research project, especially for high-level studies like RCTs, efficiency and scientific rigor is mandatory and several data are collected along with the primary outcome results, to complete and explain them. But how reliable and transparent are these reported outcomes? Outcome reporting bias, meaning selective reporting of outcomes depending on the results obtained, has been identified as a major problem even in well-designed and prestigious trials (3). Several studies reviewing the quality of outcome reporting in published RCTs emphasized that only 47% of trials report the same primary outcome in publication as in the initial protocol (4) whereas in 62% of all published studies at least one primary outcome was changed, downgraded to secondary endpoint, added or completely omitted compared to the study protocol (3-5). Moreover, it has been repeatedly observed that significant results and the presence of external funding interfere greatly with the choice of outcomes to be reported in final publication (3,6).

As shown by the above mentioned studies, the scientific community has information about the quality of primary outcome reporting, but very little is known about that of secondary outcomes, deemed to be even worse (3). Furthermore, if endpoint documentation seems so poor for RCTs, where normally a protocol precedes the final publication of the trial, what can be deduced about observational cohort or registry-based studies, where no pre-existing protocol exists? There is no way to know if the secondary outcomes were actually planned or not, or if they come from ‘data dredging’ or pure reporting bias, for example to replace deceiving results of the primary outcome. These considerations along with many others have lead Glasziou et al. to define several types of ‘research waste’, meaning research data that consume a great deal of energy and resources from trial planning until publication, but remain poorly documented or incomplete, and thus, of no value (5,7).

What does poor outcome reporting mean for clinical practice?

Poor outcome reporting, defined here as either reporting bias or poor methodology in the definition and detection of the reported outcomes, may have serious implications both for clinical practice and future research. In addition to reducing each study’s internal validity, its effect is potentialized when data from such studies are combined together in systematic reviews and meta-analyses. The PRISMA guidelines of conduct and reporting of systematic reviews and meta-analyses, require accurate definition of all research questions to be answered on a review level (8). However, on an individual study level, although the risk of outcome reporting bias can be assessed by different methods (e.g., the Cochrane Collaboration tool for assessing risk of bias) (9), the actual quality and relevance of the reported outcomes may not at all be assessed or reported in the final conclusions. As the quality of any meta-analysis can only be as good as the individual studies included, caution is needed for the interpretation of such high-level information if underlying outcome reporting seems to be poor.

In a very insightful recent study, Matthews et al. (10) identified another interesting aspect of poor outcome reporting, the unreliability of data presented as secondary outcomes in RCTs. They assessed the rate of surgical site infections (SSIs) reported in several high-impact surgical RCTs, and found that irrespective of the type of surgery and the level of contamination, when SSIs were reported as secondary outcome they were significantly lower, thus under-reported, compared to when reported as the primary outcome (5.1% vs.12.6% respectively, P<0.001). Definitions and detection methods of SSIs were less rigorously followed for secondary endpoints, explaining this striking difference.

The first potential problem here is the danger of using misleading information in clinical practice, for example when a patient is incorrectly informed about his real risk of SSI based on poor quality data. From a more scientific point of view, if a future study with SSIs as primary endpoint was to be planned, one of these baseline estimations of SSI may be used for sample size calculation For example, if a study was planned for an intervention X that will decrease the rate of postoperative SSIs by 50%, in the first case (baseline SSI =5.1%), 2 groups of 905 patients each will be needed, whereas in the second case (baseline SSI =12.6%), only 377 patients per group are required for a power of 80% and an alpha-error of 0.05 (11). Of course, while planning such a study, the author will seek the most reliable and widely accepted baseline SSI rate, but this is a typical situation where previously published ‘research waste’, re-used without the necessary critical appraisal, might lead to a real waste of resources in future research.

Increasing rigor in outcome reporting; core outcome sets (COS) and research guidelines

As awareness is rising in recent years about methodological problems related to outcome reporting described above, several initiatives have been taken to improve reliability and transparency of research results.

To specifically address the problem of poor outcome reporting, the COS has been developed for many fields of medical research. The COMET initiative (Core Outcome Measures in Effectiveness Trials) launched the process in 2012, to improve both the clinical relevance and transparency of research outcomes reported in published studies (12). As significant heterogeneity is observed in the types of outcomes authors choose to study and report, development of standardized sets of clinically relevant outcomes for specific research fields are a valuable option. The COMET database currently contains 857 references of planned, ongoing and completed work (13) while any researcher can develop a new set of outcomes deemed relevant and reproducible for his field of interest (14). Basically, by defining a standard minimum of outcomes to report in each study, several advantages might be expected; less missing data on key outcome measures, complete and rigorous description of the outcomes concerning a field of interest, ensuring the report of clinically relevant outcomes in each research question, and as a result, limiting the risk of outcome reporting bias.

The EQUATOR initiative (Enhancing the QUAlity and Transparency Of Research results), funded in 2006 by Douglas Altman’s team in Oxford (15), is another pioneer movement aiming to improve and standardize the way medical research is conducted and reported. Today, guidelines and checklists have been developed for all major study types, such as the CONSORT reporting guidelines for RCTs, STROBE for observational studies, PRISMA for systematic review and meta-analyses, SPIRIT for study protocols etc. These guidelines are freely available for researchers (15) and can be of great help both while planning the study and drafting the manuscript. Many high-impact journals require conformity of the manuscript with the above standards to accept submitted manuscripts, however the overall adherence to these guidelines has been reported as low as 20% of published studies (16). Thus, ongoing effort is required from both medical journals and individual researchers to ensure a maximal adherence to these guidelines, leading to increased scientific rigor and reproducibility of results (5).

Pre-publication registration of all studies, especially those with a high expected impact, provides a valuable tool of methodological assessment of published trials. Internet-based registries already exist for RCTs (www.ClinicalTrials.gov, European Clinical Trials Database–EudraCT) and systematic reviews/ meta-analyses (PROSPERO registry) (17). Although registration of those key study types allows some control over the potential risks of bias, observational studies remain a major unsolved problem. As mentioned above, there is no way of controlling potential outcome-related bias in these studies, due to no pre-existing study plan for comparison. Increasing attention is paid to the need of registration and monitoring of observational studies, however to this day nothing has yet changed in this direction (18,19).

In conclusion, poor outcome reporting is getting increased attention from both statisticians and researchers in recent years, with alarming rates of inadequate outcome measures published even in high-impact literature. Acknowledging the problem already leads to the necessary cautious interpretation of published study results, however the real need is taking outcome reporting to a better level. Development and use of COS, transparent registration of trials and adherence to standardized guidelines of study report are some extremely valuable, although clearly underused tools, that should be rigorously implemented in medical research to increase its long-term scientific sustainability.

Acknowledgements

None.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005;365:1348-53. [Crossref] [PubMed]
2. Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg 2011;128:305-10. [Crossref] [PubMed]
3. Dwan K, Gamble C, Williamson PR, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One 2013;8:e66844. [Crossref] [PubMed]
4. Chan AW, Hróbjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457-65. [Crossref] [PubMed]
5. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014;383:267-76. [Crossref] [PubMed]
6. von Elm E, Röllin A, Blümle A, et al. Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss Med Wkly 2008;138:197-203. [PubMed]
7. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86-9. [Crossref] [PubMed]
8. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647. [Crossref] [PubMed]
9. Green S, Higgins JP, Alderson P, et al. Cochrane Handbook for: Systematic Reviews of Interventions. In: Higgins JP, Altman DG, Sterne JA, et al. editors. Chapter 8: Assessing risk of bias in included studies 2011:3-10.
10. Matthews JH, Bhanderi S, Chapman SJ, et al. Underreporting of Secondary Endpoints in Randomized Trials: Cross-sectional, Observational Study. Ann Surg 2016;264:982-6. [Crossref] [PubMed]
11. ClinCalc sample size calculator (Internet). Available online: http://clincalc.com/stats/samplesize.aspx
12. Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials 2012;13:132. [Crossref] [PubMed]
13. COMET initiative webpage (Core Outcome Measures in Effectiveness Trials). Available online: http://www.comet-initiative.org
14. Kirkham JJ, Gorst S, Altman DG, et al. COS-STAR: a reporting guideline for studies developing core outcome sets (protocol). Trials 2015;16:373. [Crossref] [PubMed]
15. EQUATOR Network webpage (Enhancing the QUALity and Transparency Of health Research). Available online: http://www.equator-network.org
16. Samaan Z, Mbuagbaw L, Kosa D, et al. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc 2013;6:169-88. [PubMed]
17. PROSPERO Registry for Systematic Reviews and Meta-analyses (Internet). Available online: https://www.crd.york.ac.uk/PROSPERO/
18. Loder E, Groves T, Macauley D. Registration of observational studies. BMJ 2010;340:c950. [Crossref] [PubMed]
19. Williams RJ, Tse T, Harlan WR, et al. Registration of observational studies: Is it time? CMAJ 2010;182:1638-42. [Crossref] [PubMed]