Evidence of spin in clinical trials in the surgical literature

The pre-eminence of evidence-based medicine has led to a growing emphasis on the requirement for complete, clear and unbiased reporting of research. In particular, a premium has been placed on best research practice within randomized controlled trials, as these constitute a central plank in the assessment of the comparative effectiveness of biomedical interventions. Initiatives to improve the conduct and reporting of clinical trials have encompassed the development of guidelines to enhance overall reporting (1), as well as guidance specific to aspects of trial conduct and reporting including interventions (2) and statistical analysis (3). Notwithstanding this, the quality of trial design and the clarity of reporting has consistently fallen short of these ideals with problems common to diverse areas within medicine and dentistry (4,5).

Among the more controversial issues is the concept of publication bias which may constitute failure to publish or report negative or unwanted scientific findings, particularly in funded research. Other potential forms of reporting bias include selective outcome reporting with more interesting or positive findings preferentially reported, while negative findings may be overlooked (6). Similarly, ‘spin’ whereby trial results are presented in a selective, potentially biased manner risking distortion of the interpretation of trial results has been recognized both in clinical trials and more recently in systematic reviews (7,8). ‘Spin’ may relate to ignorance of the particular issue, unconscious bias, or may be due to wilful intent to deceive.

Previous research has exposed that ‘spin’ may affect the general medical literature with four forms of spin being defined: focus on statistically significant secondary results instead of non-significant primary results; interpreting statistically non-significant results for the primary outcomes as showing treatment equivalence; highlighting the beneficial effect of treatment despite statistically non-significant results; and fixating on within-group improvements over time rather than between-group comparisons as a basis of treatment efficacy when all primary analyses compared the treatment group with a control group. In the present cross-sectional analysis focusing on ten leading surgical journals these four types of ‘spin’ were investigated throughout the trial reports (in the title, abstract results and conclusions, main-text results, discussion and conclusions) over a 2-year period (9). The included studies all had non-significant results for the primary outcomes. The presence of spin was also related to trial funding source, use of a statistician, trial phase, and intervention type.

Overall, 110 articles with non-significant primary outcomes were considered; the majority related to either pharmacological (39%) or surgical (28%) interventions. Funding sources were declared in 60% of these with for-profit funding sources accounting for 23% overall. Spin was commonplace both within the abstracts (40%) and full-text articles (35%). In relation to conclusions sections, spin was found in the abstract conclusions in 27%, and to be high in the abstract and main-text conclusions in 14% and 19%, respectively. Worryingly, the intervention of interest was recommended despite a non-significant primary outcome in 22% of studies. No relationship was found, however, between trial funding source or intervention type and the presence of spin. These findings indicate that there is commonly a mismatch between trial results and their interpretation in high-impact surgical journals. All four ‘spin’ mechanisms were commonly found.

Should these findings surprise us or indeed do they refer to an isolated problem within surgical journals? The authors refer to almost identical findings from general medical and pain literature suggesting that ‘spin’ is ingrained throughout the biomedical literature and that this is therefore not an isolated problem (7,10). Previous research has related reporting that is incompatible with results to commercial and industry funding (11); however, this was not reflected in the present analysis suggesting that other factors are at play. The authors do not speculate as to what these might be but conscious or unconscious reporting bias allied to inadvertent errors may be contributory.

This cross-sectional analysis, therefore, alludes to a further reporting issue in a subset of biomedical journals. Similar findings seem to be common in large meta-epidemiological studies despite the adoption of increasing numbers of reporting guidelines. Clearly, more needs to be done in terms of robust implementation of these tools. This shift from guideline development to implementation has been intensifying in recent years and is likely to continue with increasing emphasis on innovative and bespoke methods of submission, presentation and editorial processes (12-14). The present cross-sectional analysis provides further evidence of the pressing need for these refinements.

Acknowledgements

None.

Footnote

Provenance: This is a Guest Editorial commissioned by Section Editor Jianrong Zhang, MD (Department of Thoracic Surgery, First Affiliated Hospital of Guangzhou Medical University, Guangzhou Institute of Respiratory Disease, Guangzhou, China).

Conflicts of Interest: The author has no conflicts of interest to declare.

Comment on: Arunachalam L, Hunter IA, Killeen S. Reporting of Randomized Controlled Trials With Statistically Nonsignificant Primary Outcomes Published in High-Impact Surgical Journals. Ann Surg 2016. [Epub ahead of print].

References

1. Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med 2010;8:18. [Crossref] [PubMed]
2. Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687. [Crossref] [PubMed]
3. Curran-Everett D, Benos DJ. Guidelines for reporting statistics in journals published by the American Physiological Society: the sequel. Adv Physiol Educ 2007;31:295-8. [Crossref] [PubMed]
4. Turner L, Shamseer L, Altman DG, et al. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev 2012;1:60. [Crossref] [PubMed]
5. Lempesi E, Koletsi D, Fleming PS, et al. The reporting quality of randomized controlled trials in orthodontics. J Evid Based Dent Pract 2014;14:46-52. [Crossref] [PubMed]
6. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 2008;3:e3081. [Crossref] [PubMed]
7. Boutron I, Dutton S, Ravaud P, et al. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 2010;303:2058-64. [Crossref] [PubMed]
8. Yavchitz A, Ravaud P, Altman DG, et al. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol 2016;75:56-65. [Crossref] [PubMed]
9. Arunachalam L, Hunter IA, Killeen S. Reporting of Randomized Controlled Trials With Statistically Nonsignificant Primary Outcomes Published in High-Impact Surgical Journals. Ann Surg 2016. [Epub ahead of print].
10. Gewandter JS, McKeown A, McDermott MP, et al. Data interpretation in analgesic clinical trials with statistically nonsignificant primary analyses: an ACTTION systematic review. J Pain 2015;16:3-10. [Crossref] [PubMed]
11. Lundh A, Sismondo S, Lexchin J, et al. Industry sponsorship and research outcome. Cochrane Database Syst Rev 2012;12:MR000033. [PubMed]
12. Pandis N, Shamseer L, Kokich VG, et al. Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting. J Clin Epidemiol 2014;67:1044-8. [Crossref] [PubMed]
13. Wynne KE, Simpson BJ, Berman L, et al. Results of a longitudinal study of rigorous manuscript submission guidelines designed to improve the quality of clinical research reporting in a peer-reviewed surgical journal. J Pediatr Surg 2011;46:131-7. [Crossref] [PubMed]
14. Barnes C, Boutron I, Giraudeau B, et al. Impact of an online writing aid tool for writing a randomized trial report: the COBWEB (Consort-based WEB tool) randomized controlled trial. BMC Med 2015;13:221. [Crossref] [PubMed]