If no difference in effectiveness is found between two treatments it may be because the treatments are similarly ineffective

The authors are to be congratulated for undertaking a systematic review and meta-analysis partly following the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) checklist (1,2). It is only by working to internationally agreed standards can surgeons expect to gain scientific credibility for their treatments. The E in EBM stands for evidence, not for eminence, experience, expertise, eloquence or any of the other “E” words that have previously been used to provide authority for the practice of surgery for advanced and metastatic cancer (3,4). Systematic reviewing should be an essential component of the appraisal of any clinical practice: we need to know what is already known before embarking on further studies.

The studies comparing thoracotomy and video-thoracoscopic resection of pulmonary metastases found for systematic review did not include any randomised trials (5). What has been found by the search are surgeons’ case series. These are prone to severe limitations as a source of evidence (6). However even when carried out, randomised trials of surgical interventions may be inconclusive because of small sample size, difficulties in blinding and the trial patients being unrepresentative of so-called “real world” practice. Until we have more robust randomised trials we need to see what can be learnt from critical inspection of non-randomised data, but we should only draw conclusions about comparative effectiveness with extreme caution (7).

In this case it is probably inappropriate to draw any conclusions from these unreliable data about the comparative effectiveness and risks of the two procedures.

The decision to use video-thoracoscopy rather than thoracotomy was not randomised in any of these studies but it is unlikely to have been haphazard. The review unfortunately gives no information about how populations in each study were produced. The surgeons probably will have chosen to use one surgical approach rather than the other for a variety of technical considerations. One will certainly have been the number of metastases. The number of metastases has consistently been found to be associated with length of survival after metastasectomy. The threshold of that effect is between one and more than one metastasis (8). There are many other practical considerations such as the size and the anatomical location of the metastases that lead to a surgeon preferring an open vs. a minimal access approach. There is a tendency to regard bigger treatment as better treatment leading surgeons to assume that the more major operation of open thoracotomy provides a better “oncological” operation. This attitude merits closer examination.

There are two reasons commonly given as oncological justification for thoracotomy rather than using the less invasive video-thoracoscopic approach. One is that it allows the surgeon to feel the lung and find metastases that are beyond the limits of detection by CT. Since the increase in hazard is between one and more than one metastasis, finding the third, fourth or fifth may not make any difference. Some surgeons find and remove large numbers. The greatest recorded number of lung metastases removed of which we are aware is 124 (9). It tests credibility that finding one more or less would have made any difference.

The other oncological justification for open surgery is to enable a more complete systematic dissection of mediastinal nodes that can supposedly be achieved by open surgery. That is not what the evidence in lung cancer surgery shows. Analysis of data in the Surveillance, Epidemiology and End Results (SEER)-Medicare database is that surgeons who have adopted and developed an expertise in video-thoracoscopic surgery removed more not fewer lymph nodes (10). As Migliore and colleagues write, lymphadenectomy can be performed just as well by video-thoracoscopy (5). Again however we should challenge the assumption that removing lymph nodes more completely is necessarily of benefit to the patient. The presence of mediastinal nodal involvement is a major adverse factor for prognosis and again it is unlikely that piecemeal resection of further onward metastatic deposits alters the outcome for patients with obviously disseminated cancer.

However all of this misses the fundamental underlying question. As pointed out by Migliore and colleagues, the underlying belief that lung metastasectomy itself offers survival advantage is in doubt (5). A study that finds no difference in effectiveness between two methods of treatment does not prove that they are equally effective; it includes the possibility that neither is effective. There is inadequate evidence for any survival benefit from metastasectomy. Although randomised trials of surgery are difficult to run, unless there is proof of what surgery is effective and what is not, patients will not be given the most effective treatment. The Pulmonary Metastasectomy in Colorectal Cancer (PulMiCC) trial is designed to resolve that doubt. Following peer review and re-evaluation of the on-going trial by Cancer Research UK the PulMiCC trial continues to recruit internationally.

Although we opened by congratulating the authors on adopting PRISMA as a checklist for their systematic review, there is more that could be reported to come up to top quality EBM standards. The readers should be told what databases were searched. Did the authors select the papers and extract data independently? A full report would give a description of the individual studies. Has the risk of bias been assessed? It may be high in all cases since the choice of thoracotomy and video-thoracoscopic surgery relies on conviction, not evidence. The evidence for effectiveness of lung metastasectomy is lacking and the conclusions about the comparative “effectiveness” are insecure. There is no comment about the small size and (presumably) retrospective nature of the studies but that is the nature of most of the “evidence” on which cancer surgery is based. This study is a step on the road towards better research but there is some distance yet to go.

Acknowledgements

None.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535. [PubMed]
2. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700. [PubMed]
3. Russell RC, Treasure T. Counting the cost of cancer surgery for advanced and metastatic disease. Br J Surg 2012;99:449-50. [PubMed]
4. Treasure T, Macbeth F. Fifteen years in the evaluation of extrapleural pneumonectomy: Lessons to be learned. J Thorac Cardiovasc Surg 2015;149:1382-3. [PubMed]
5. Migliore M, Criscione A, Calvo D, et al. Wider implications of video-assisted thoracic surgery versus open approach for lung metastasectomy. Future Oncol 2015;11:25-9. [PubMed]
6. Treasure T, Utley M. Ten traps for the unwary in surgical series: a case study in mesothelioma reports. J Thorac Cardiovasc Surg 2007;133:1414-8. [PubMed]
7. Anyanwu AC, Treasure T. Surgical research revisited: clinical trials in the cardiothoracic surgical literature. Eur J Cardiothorac Surg 2004;25:299-303. [PubMed]
8. Gonzalez M, Poncet A, Combescure C, et al. Risk factors for survival after lung metastasectomy in colorectal cancer patients: a systematic review and meta-analysis. Ann Surg Oncol 2013;20:572-9. [PubMed]
9. Rolle A, Pereszlenyi A, Koch R, et al. Is surgery for multiple lung metastases reasonable? A total of 328 consecutive patients with multiple-laser metastasectomies with a new 1318-nm Nd:YAG laser. J Thorac Cardiovasc Surg 2006;131:1236-42. [PubMed]
10. Paul S, Isaacs AJ, Treasure T, et al. Long term survival with thoracoscopic versus open lobectomy: propensity matched comparative analysis using SEER-Medicare database. BMJ 2014;349:g5575. [PubMed]