Methodologies of COLDICE and Cryo-PID studies: details make the difference

On behalf of the COLDICE Team, we are grateful for the opportunity to respond to comments on the methodology of the COLDICE Study, to clarify any confusion about how our findings were generated (1,2).

The manuscript published within this journal by Suehs et al. highlights differences in the statistical analyses of two recently published studies investigating concordance of transbronchial lung cryobiopsy and surgical lung biopsy for interstitial lung disease (ILD) diagnosis (3). The authors acknowledge their involvement with the Cryo-PID study, one of the two studies under scrutiny (4). The stated intention of the commentary is to move forward the domain of cryobiopsy in ILD. The challenge of this exercise is highlighted by the authors in their recognition of substantial methodological differences that render the results of the two studies incomparable.

Ahead of addressing the specific concerns about the applicability of our findings, we reiterate the strengths of our study including precise, clinically relevant endpoints, use of robust processes to minimise bias, and standardised techniques for biopsy collection. In particular, the prospective incorporation of each biopsy result into an individual multi-disciplinary discussion (MDD) followed practice in the clinical setting.

The ATS/ERS/JRS/ALAT Idiopathic Pulmonary Fibrosis (IPF) guideline categories were used to distinguish one of our co-primary endpoints for both pragmatic and clinical reasons (5). These well-defined and widely accepted classifications are clinically useful for decision making, and importantly, allowed for statistical powering of the study to a meaningful endpoint. We emphasise that more nuanced histopathological categories were also determined, as defined by Travis et al., and reported as a key secondary outcome in the primary manuscript (1,6). Both strategies revealed virtually identical agreement at histopathology interpretation (70.8% for the guideline-directed categories, versus 69.2% for specific patterns). This refutes the suggestion that fewer categories contributed to higher likelihood of agreement. We acknowledge that the weighted kappa value used for the IPF guideline categories (κ_w 0.7) was higher than the unweighted kappa value used for the specific histopathologic patterns (κ 0.46) due to fewer categories, and a ranking scale, however this does not alter the validity of the findings, nor the legitimacy of the statistical analysis.

Our utilisation of three expert pathologists to overcome inherent biases of a single expert is brought into question. It is suggested that this approach (which is standard in pathology research) reduced the applicability of our findings to the “reader’s” own practice. Clearly the two studies deviate from most real-world situations, in that the majority of patients will not have simultaneously sampled biopsy specimens to inform their diagnosis. Nor will most centres have the benefit of world-authority ILD pathologists to interpret their specimens. In particular, the inclusion of both samples within a single MDD in the Cryo-PID protocol does not align with usual clinical practice, and thus is of limited utility for the reader. To answer the key clinical question of diagnostic accuracy of cryobiopsy, we chose scientifically rigorous strategies, acknowledging that trial conditions may not always be met in actual clinical practice. We would also add that in many circumstances, pathologists will consult with colleagues to form consensus views, particularly where there are uncertainties.

Beyond histopathological agreement, the COLDICE Study evaluated the performance of each biopsy at MDD, the accepted gold standard for ILD Diagnosis. The agreement for surgical biopsy-MDD and cryobiopsy-MDD diagnoses was 76.9% with an unweighted κ of 0.62 (0.47–0.78). Thus, the good concordance that we reported between cryobiopsy and surgical biopsy was true for both guideline-directed histopathology and for final MDD diagnoses. In our study design, we recognised that assessment of the performance characteristics of the cryobiopsy required the scaffolding of MDD, given that a lung biopsy alone is rarely sufficient for diagnosis.

We also took into account the interaction between “diagnostic confidence” and the assigned diagnosis, reflecting the greyscale that currently exists in ILD classification. Using the ontology recommended by Ryerson et al., each MDD diagnosis was delegated as “Definite”, “High” or “Low” confidence, or “Unclassifiable” (7). For the 39/65 high confidence cryobiopsy-MDD cases, there was 95% concordance with surgical biopsy-MDD diagnoses. Even in the 20/65 low confidence cryobiopsy-MDD cases, there was 60% concordance with the surgical biopsy-MDD findings, and only 4/20 had an alternative high confidence MDD diagnosis with the surgical specimen. The key message from this analysis is that surgical biopsy added very little additional information for the majority of cases, particularly when diagnostic confidence was high.

To avoid any recollection of the patient details at MDD, we took exhaustive measures. The undertaking of a single-session MDD to discuss the 130 biopsies from 65 patients, was in fact held over four days. The cases were discussed in random order, however the order was changed where paired cases were too close in time, as detailed in our methodology (2). Further safeguards against recollection were also taken: two alternating radiologists and two alternating presenters were used for different sessions; and clinical details on proforma slides were kept generic, removing unique identifying aspects except where relevant to the diagnostic process. To ensure there was no systematic bias towards either diagnostic approach, the presenting pathologists showed high-power views of the biopsy only, ensuring the meeting participants were unaware of the scale of the tissue. It is of some surprise to us that the efforts that we undertook to optimise the data quality are implied to be shortcomings in the present manuscript.

We conclude by agreeing with the authors that inherent differences in study design resulted in disparate findings in the COLDICE and Cryo-PID studies. We strongly disagree that methodological manipulation led to the high concordance between biopsy techniques in our study. As Suehs et al. emphasise, there are many questions that remain incompletely resolved, particularly those around safety and the ideal patient characteristics for undergoing cryobiopsy. There is a clear need for further research in this area. We hope to provide some additional insights from our secondary analyses, which are currently underway.

Acknowledgments

Funding: The authors received funding support for the COLDICE study from Erbe Elektromedizin, Medtronic, Cook Medical, Rymed, Karl-Storz, Zeiss and Olympus. Institutional research funding was received from University of Sydney and John Hunter Hospital.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/atm-20-3769). EMTL, CG, TJC, JPW, MJP, PJT, MPV and LKT report grants from Erbe Elektromedizin, non-financial support from Elektromedizin, grants from Medtronic, non-financial support from Cook Medical, grants from Rymed, non-financial support from Karl-Storz, non-financial support from Zeiss, non-financial support from Olympus, during the conduct of the study. WAC reports non-financial support from Zeiss and Olympus. In addition, MPV reports personal fees for consultancy and advisory board work for Medtronic, and TJC reports grants and personal fees from Boehringer Ingelheim, grants and personal fees from Roche, personal fees from BMS, personal fees from Promedior, grants from Gilead, personal fees from Ad Alta, grants from Bayer, grants from Biogen, outside the submitted work. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Troy LK, Grainge C, Corte TJ, et al. Diagnostic accuracy of transbronchial lung cryobiopsy for interstitial lung disease diagnosis (COLDICE): a prospective, comparative study. Lancet Respir Med 2020;8:171-81. [Crossref] [PubMed]
2. Troy LK, Grainge C, Corte T, et al. Cryobiopsy versus open lung biopsy in the diagnosis of interstitial lung disease (COLDICE): protocol of a multicentre study. BMJ Open Respir Res 2019;6:e000443. [PubMed]
3. Suehs C, Bourdin A, Vachier I, et al. Transbronchial cryobiopsy in the diagnosis of interstitial lung diseases: methodologies and perspectives from the Cryo-PID and COLDICE studies. Ann Transl Med 2020. [Crossref]
4. Romagnoli M, Colby TV, Berthet JP, et al. Poor Concordance between Sequential Transbronchial Lung Cryobiopsy and Surgical Lung Biopsy in the Diagnosis of Diffuse Interstitial Lung Diseases. Am J Respir Crit Care Med 2019;199:1249-56. [Crossref] [PubMed]
5. Raghu G, Remy-Jardin M, Myers JL, et al. Diagnosis of Idiopathic Pulmonary Fibrosis. An Official ATS/ERS/JRS/ALAT Clinical Practice Guideline. Am J Respir Crit Care Med 2018;198:e44-68. [Crossref] [PubMed]
6. Travis WD, Costabel U, Hansell DM, et al. An official American Thoracic Society/European Respiratory Society statement: Update of the international multidisciplinary classification of the idiopathic interstitial pneumonias. Am J Respir Crit Care Med 2013;188:733-48. [Crossref] [PubMed]
7. Ryerson CJ, Corte TJ, Lee JS, et al. A Standardized Diagnostic Ontology for Fibrotic Interstitial Lung Disease. An International Working Group Perspective. Am J Respir Crit Care Med 2017;196:1249-54. [Crossref] [PubMed]