Evaluation of the reporting quality of clinical practice guidelines on melanoma using the RIGHT checklist

Introduction

Melanoma is a malignancy whose incidence and mortality rates across the world are strongly associated with socioeconomic and lifestyle factors (1). In 2018, there were an estimated 290,000 new cases of melanoma and approximately 61,000 deaths caused by melanoma worldwide (2). Once melanoma becomes invasive or spreads, it is life-threatening. In the past decade, increasing knowledge of melanoma has led to significant advances in fighting this disease. After the introduction of the immunotherapy ipilimumab in 2011 (3), patient survival has improved rapidly. Since then, nine new drugs have been approved, including immunotherapy with anti-PD-1/PD-L1 monoclonal antibodies, and targeted therapy with BRAF inhibitors and MEK inhibitors (4,5). Molecular pathogenesis and clinical imaging technologies have also advanced, resulting in a revolution in the melanoma staging system and patient stratification (6). The new imaging technologies do not only inform prognostic assessment, but also improve clinical outcomes. Surgical resection, chemotherapy and radiotherapy remain relevant, and should also be incorporated into clinical decision making (7). The rapidly expanding therapeutic landscape has also led to rising costs (8). Given the multiple therapeutic options and their high costs, the ways to integrate these approaches in the optimal way need to be assessed.

Many clinical practice guidelines (CPGs) have been developed to provide guidance on the screening, diagnosis, surgical and medical treatment approaches, and further management of melanoma. The existing guidelines also address more specific issues, such as melanoma management in special populations (9), genetic testing for hereditary risk factors (10) and the management of the unique toxicity profiles related to novel therapies (11). CPGs can decrease the variability in healthcare procedures and help to choose the appropriate approach in each specific clinical situation (12). This expectation relies on the assumption that the CPG has been rigorously developed, and that the recommendations are easy to understand and implement. The quality of CPGs, therefore, is of particular concern. To our knowledge, the quality of melanoma guidelines has not yet been systematically assessed.

Explicit and clear reporting is an important aspect of the quality of a CPG. The International Reporting Items for Practice Guidelines in Healthcare (RIGHT) instrument, published in 2016, is a tool for assessing reporting quality of guidelines according to 22 key criteria, grouped into seven domains (13). In this study, we systematically searched for guidelines for melanoma and assessed their reporting quality using the RIGHT tool. Our ultimate aim was to identify weaknesses and strengths in reporting to inform the framework of future development of melanoma guidelines.

Methods

Study design

We conducted a systematic review and critical appraisal of the reporting quality of CPGs for melanoma using the RIGHT checklist.

Literature search

We searched Medline (via PubMed), Chinese Biomedical Literature Database (CBM), Wan Fang Database and Chinese National Knowledge Infrastructure (CNKI) from their inception to November 27, 2020, using the following terms: melanoma, nevocarcinoma, melanomatosis, practice guideline, guidance, and recommendation. Our search was restricted to publications in English and Chinese. Additionally, we searched the websites of The National Institute for Health and Care Excellence (NICE, https://www.nice.org.uk/), The National Comprehensive Cancer Network (NCCN, https://www.nccn.org/), World Health Organization guidelines (WHO, https://www.who.int/publications/guidelines/year/en/), Scottish Intercollegiate Guidelines Network (SIGN, https://www.sign.ac.uk/our-guidelines/), and Guidelines International Network (GIN, https://guidelines.ebmportal.com), as well as Google Scholar as supplemental sources.

Inclusion criteria and exclusion criteria

CPGs were eligible for inclusion if they focused on screening, testing, diagnosis, treatment, or management of melanoma; the full-text was accessible; and the most recent updated version was published between 2018 and 2020. We excluded records that were translations, summaries or interpretations of guidelines. Drafts and preprints of guidelines were also excluded.

Screening

All records were exported to a bibliographic file and imported into EndNote X8 (Thomson Reuters) for management and screening. Titles and abstracts were screened by AL Zhang and XH Liu to determine their relevance. Final selection was based on full-text screening of all potentially applicable articles and relevant supplementary materials. Ambiguous articles were examined by a third reviewer (QW Zhang). Disagreements were resolved by discussion with a senior third author (XJ Zhang).

Data extraction of guidelines

All data were extracted in duplicate (KF Liu and S Tang) using a standardized electronic form. Disagreements were settled through consensus or, if necessary, consultation with a third reviewer (J Kang). We extracted the following data from each CPG: first author, region/country where the CPG was developed, developers (institution or working group), publication year, publication language, format of publication (peer-reviewed journal, or website only), impact factor (IF) of the journal according to the Science Citation Index (SCI), and the scope/purpose and target population of the CPG.

Reporting quality assessment using RIGHT checklists

The reporting quality of CPGs was evaluated using the RIGHT tool, which includes 22 key items categorized into seven domains. Some key items were divided into two or three sub-items, resulting in a total of 35 items. The seven domains are: basic information (six items), background (eight items), evidence (five items), recommendations (seven items), review and quality assurance (two items), funding and conflicts of interest statements and management (four items), and other information (three items). We rated items as “reported” (relevant information was fully presented), “unreported” (lacked some relevant information), or “not applicable” (not appropriate for evaluating in the specific guideline), based on the protocol of the RIGHT tool. The evaluation method was checked for validity and consistency by two co-authors (YJ Yang and JL Lu), who subsequently independently evaluated the reporting quality of all CPGs. Disagreements were settled by consultation with a member of RIGHT checklist working group (YF Ma).

Statistical analysis

The numbers and proportions of the RIGHT items that were appropriately reported were calculated for each guideline. We calculated the mean reporting rates over each domain of RIGHT as well as overall weighting all 35 items equally. We present the results as means over all CPGs, as well as stratified by the type of organization and country that led the guideline development, the language of publication, the year of publication, the IF of the journal, and funding support.

Results

Identification of guidelines

We identified 865 potentially relevant records in the literature search (Figure 1). After exclusion of 17 duplicated records, we screened 848 titles and abstracts, and excluded 820 that were not eligible. We retrieved 28 references for full-text assessment. A total of 20 CPGs met the inclusion criteria and were included in the analysis.

Figure 1 The flow chart of the selection process.

Characteristics of selected guidelines

The characteristics of the included 20 CPGs are shown in Table 1. Three CPGs were published in 2018, eight in 2019, and nine in 2020. Most CPGs (n=17) were developed by medical specialty societies, and three by a guideline working group. Eight CPGs were developed in North America, seven in Europe, three in Asia, one in Australia, and one was developed by an international consortium. One CPG reported receiving no funding, nine reported receiving government or society funding, and ten did not report whether they received funding or not. Twelve CPGs were updated versions, six were original versions, and two were adapted from other guidelines. Two CPGs used GRADE to grade the quality of evidence and strength of the recommendations, two CPGs Oxford classification, and three CPGs the Infectious Diseases Society of America-United States Public Health Service Grading System. Only two CPGs were developed following the Appraisal of Guidelines for Research & Evaluation II (AGREE II) framework in their methodology.

Table 1 Characteristics of the included clinical practice guidelines.
Full table

Overall analysis of reporting quality

Five of the seven RIGHT domains had a reporting rate of >50% (Figure 2). The reporting rates were 67.5% for the basic information domain, 81.9% for the background domain, 61.4% for the recommendations domain, and 51.2% for the funding and declaration and management of interests domain. Two domains (other information; review and quality assurance) had a reporting proportion of 35.0%.

Figure 2 The reporting rates of the seven RIGHT checklist domains in the included clinical practice guidelines. RIGHT, Reporting Items for Practice Guidelines in Healthcare.

The overall reporting rates ranged between 28.6% and 82.9% across the guidelines. The mean overall reporting rate over all guidelines was 63.7%, and only two CPGs had rates above 80% (Figure 3). All CPGs reported the items 1a and 3 in the basic information domain, items 7a and 7b about the target population in the background domain, and items 13a and 13b in the recommendations domain (Table 2). No CPG reported item 14c (recommendations domain), only one CPG described the role of funders (item 18b, funding source and declaration and management of interests domain), and two CPGs indicated if and how quality assurance was evaluated (item 17, review and quality assurance domain).

Figure 3 The number of guidelines by overall reporting rates (%).

Table 2 The reporting rates of each RIGHT checklist item across the included clinical practice guidelines (13)
Full table

Stratified analyses

The mean overall reporting proportion improved slightly over time, being 60.0% in guidelines published in 2018, 61.4% in 2019, and 67.3% in 2020 (Table 3). Guidelines published in Chinese had a reporting rate of 37.1%; for guidelines published in English the reporting rate was 65.3%. Guidelines that reported their funding sources had a higher reporting rate (71.1%) than those that either did not report funding, or reported receiving no funding (57.9%).

Table 3 The reporting proportion of included CPGs according to stratified analysis
Full table

Discussion

To our knowledge, this is the first analysis of reporting quality specifically assessing CPGs for melanoma. We identified several patterns in the reporting quality of CPGs for melanoma. First, with the exception of the two guidelines developed in China, the quality of reporting was good, with reporting rates of >50%. Second, we identified two guidelines, one from the USA and one from Australia, which had clearly the highest reporting rates and which therefore could serve as a template for future guideline development. Third, the domains other information and review and quality assurance had the poorest reporting quality. Fourth, no guideline described whether developers had considered factors such as equity, feasibility and acceptability in formulating the statements, which can be expected to affect the dissemination and implementation of the recommendations.

These findings are partly consistent with the recent findings about reporting quality of CPGs for assisted reproductive technology (31) and idiopathic pulmonary fibrosis (32), as well of the WHO guidelines (33). The domains basic information and background had a relatively high reporting rate across all guidelines. However, the publication year and a summary of the recommendations tended to be poorly reported. Of the top 50% of guidelines ranked by overall reporting proportions, none reported the year of publication in the title, and 70% of them did not have a summary.

Most guidelines included in our review had deficiencies in reporting review and quality assurance process. Items pertaining to external review, peer reviewers, review process and management of the feedback were poorly reported. One potential explanation for this finding is that appropriate consideration of quality assurance reflects a greater awareness of developers on the importance of guideline reporting.

Guidelines with funding support had a high overall reporting quality than those not declaring or having funding. Funding is of great importance, because the development, maintenance, effective dissemination and implementation of guidelines is an expensive and labor-intensive endeavor (34). CPG panels with financial support may confer quality benefits. However, few guidelines reported the sources of funding for the different stages of the development, dissemination and implementation of the guideline and recommendations. The results suggest a need for greater transparency and rigor in the role of funders.

The included Chinese CPGs had an overall reporting quality clearly below the average. Poor quality of Chinese CPGs has also been demonstrated in other guideline reviews (35). One potential explanation for this finding is that Chinese-language journals do not have as high a requirement for guidelines as journals indexed by SCI, although this may also reflect a lack of awareness on the importance of guideline reporting among Chinese authors.

Most CPGs for melanoma used formal grading systems to assess the quality of evidence and strength of recommendations. Consequently, items in the evidence and recommendations domains were reported at relatively high rates. Of note, only few guidelines considered the costs and resource implications, and no guideline considered equity, feasibility or acceptability when formulating the recommendations. A previous study has consistently demonstrated inadequacy of cost analysis in guideline development, even for the most highly cited CPGs (36). CPGs should be adapted to different settings by considering local resources and feasibility. Even evidence-based recommendations may not be applicable to all populations, or they may not be implemented due to their high costs.

We acknowledge that our study has several limitations. First, we did not perform a formal statistical assessment. Our descriptive stratified analyses revealed correlations between reporting quality and some characteristics, but we could not assess the influence of potential confounding factors. Second, we only included English- and Chinese-language CPGs. We however expect the analysis to reflect the reporting quality of melanomas relatively well, as it covers CPGs developed by the key oncological societies, such as NCCN, American Society of Clinical Oncology (ASCO), Chinese Society of Clinical Oncology (CSCO) and European Society of Medical Oncology (ESMO).

Questions to be further discussed and considered

Question 1: What impact do you think the low reporting quality of clinical practice guidelines on melanoma will have on clinicians and clinical practices?

Expert opinion: Dr. Sanjiv S. Agarwala

Unfortunately, clinical practice guidelines are not reported adequately by most centers around the world. However, their importance is unquestionable, and efforts need to be made to improve and encourage participation in such projects. The low reporting quality as demonstrated in this paper will likely still have an impact on clinical practices and on clinicians. Even low-quality data on guidelines are still important and can help clinicians choose the appropriate therapy for their patients. Busy clinicians and practices are always looking for data that can help them improve outcomes and reduce costs and guidelines assist in that direction.

Expert opinion: Dr. Dirk Schadendorf

I think guidelines are important tools for broad reference of treating physicians. If diagnosis and treatment is not centralized in specialized centers, it is even more difficult for melanoma care at highest level. Reporting quality is of particular importance if diagnosis and early treatment is wide-spread, and knowledge and penetrance is low. This has consequences for affected patients leading to delayed correct diagnosis and delayed treatment or inappropriate treatment approaches.

Question 2: What do you think the most important aspects needed for developing high-quality clinical practice guidelines on melanoma are?

Expert opinion: Dr. Sanjiv S. Agarwala

The most effective way to develop high-quality guidelines is to involve more clinicians and practices in the data collection process. As this is usually time-consuming and not the highest priority for a clinician, this will take encouragement and assistance with resources. Most practices are busy with taking care of patients and will need financial incentives and the ability to hire dedicated individuals who can spend their time in collecting and reporting the data.

Expert opinion: Dr. Dirk Schadendorf

High-quality is important but similarly usability and accessibility of guidelines is of critical importance also – furthermore language used to understand by non-specialists and potentially also for patients in lay-language help to spread the guidelines – because penetration of guidelines into all levels of clinical management routines is rarely achieved. To unfold the entire power of guidelines quality aspects, but also practical usage at all levels will finally determine the clinical effect on patient outcome.

Question 3: How do you think conflicts of interest in the guidelines should be handled?

Expert opinion: Dr. Sanjiv S. Agarwala

A clear and accurate process for self-reporting of COI is the first step. Random audits of the reporting should be conducted. National data bases for collection of payments to physicians such as the “sunshine act” in the USA need to be established so that there is transparency and accuracy of the data.

Expert opinion: Dr. Dirk Schadendorf

Conflict of interest should clear and fully transparent.

Conclusions

In conclusion, our findings show that CPGs for melanoma tended to be appropriately reported, except two guidelines developed in China. The quality of reporting has improved over time, possibly as a result of the increasing standards of CPG development in recent years. However, the reporting of some aspects, such as funding sources and the role of the funder, still need to be improved. The use of RIGHT tool when developing and updating guidelines ensures rigorous, complete and transparent reporting.

Acknowledgments

The authors appreciate the academic support from the AME Reporting Guideline Collaborative Group.

Funding: This work was supported by the National Science and Technology Major Project of China (grant number 2020ZX09201009).

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/atm-21-2603). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Whiteman DC, Green AC, Olsen CM. The Growing Burden of Invasive Melanoma: Projections of Incidence Rates and Numbers of New Cases in Six Susceptible Populations through 2031. J Invest Dermatol 2016;136:1161-71. [Crossref] [PubMed]
2. Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018;68:394-424. [Crossref] [PubMed]
3. Hodi FS, O'Day SJ, McDermott DF, et al. Improved survival with ipilimumab in patients with metastatic melanoma. N Engl J Med 2010;363:711-23. [Crossref] [PubMed]
4. Luke JJ, Flaherty KT, Ribas A, et al. Targeted agents and immunotherapies: optimizing outcomes in melanoma. Nat Rev Clin Oncol 2017;14:463-82. [Crossref] [PubMed]
5. Seth R, Messersmith H, Kaur V, et al. Systemic Therapy for Melanoma: ASCO Guideline. J Clin Oncol 2020;38:3947-70. [Crossref] [PubMed]
6. Gershenwald JE, Scolyer RA, Hess KR, et al. Melanoma staging: Evidence-based changes in the American Joint Committee on Cancer eighth edition cancer staging manual. CA Cancer J Clin 2017;67:472-92.
7. Saranga-Perry V, Ambe C, Zager JS, et al. Recent developments in the medical and surgical treatment of melanoma. CA Cancer J Clin 2014;64:171-85. [Crossref] [PubMed]
8. Guy GP Jr, Machlin SR, Ekwueme DU, et al. Prevalence and costs of skin cancer treatment in the U.S., 2002-2006 and 2007-2011. Am J Prev Med 2015;48:183-7. [Crossref] [PubMed]
9. Crisan D, Treiber N, Kull T, et al. Surgical treatment of melanoma in pregnancy: a practical guideline. J Dtsch Dermatol Ges 2016;14:585-93. [Crossref] [PubMed]
10. Swetter SM, Tsao H, Bichakjian CK, et al. Guidelines of care for the management of primary cutaneous melanoma. J Am Acad Dermatol 2019;80:208-50. [Crossref] [PubMed]
11. Sullivan RJ, Atkins MB, Kirkwood JM, et al. An update on the Society for Immunotherapy of Cancer consensus statement on tumor immunotherapy for the treatment of cutaneous melanoma: version 2.0. J Immunother Cancer 2018;6:44. [Crossref] [PubMed]
12. Gundersen L. The effect of clinical practice guidelines on variations in care. Ann Intern Med 2000;133:317-8. [Crossref] [PubMed]
13. Chen Y, Yang K, Marusic A, et al. A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. Ann Intern Med 2017;166:128-32. [Crossref] [PubMed]
14. Chinese Society of Clinical Oncology. Guidelines on the Diagnosis and Treatment of Melanoma (2020). Available online: http://meeting.csco.org.cn/MUser/M/1?returnurl=http://www.csco.org.cn/cn/index.aspx
15. Smith HG, Bagwan I, Board RE, et al. Ano-uro-genital mucosal melanoma UK national guidelines. Eur J Cancer 2020;135:22-30. [Crossref] [PubMed]
16. National Health Commission of the People’s Republic of China. Chinese guidelines for diagnosis and treatment of melanoma 2018 (English version). Chin J Cancer Res 2019;31:578-85. [Crossref] [PubMed]
17. Michielin O, van Akkooi ACJ, Ascierto PA, et al. Cutaneous melanoma: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up†. Ann Oncol 2019;30:1884-901. [Crossref] [PubMed]
18. Michielin O, van Akkooi A, Lorigan P, et al. ESMO consensus conference recommendations on the management of locoregional melanoma: under the auspices of the ESMO Guidelines Committee. Ann Oncol 2020;31:1449-61. [Crossref] [PubMed]
19. Keilholz U, Ascierto PA, Dummer R, et al. ESMO consensus conference recommendations on the management of metastatic melanoma: under the auspices of the ESMO Guidelines Committee. Ann Oncol 2020;31:1435-48. [Crossref] [PubMed]
20. Garbe C, Amaral T, Peris K, et al. European consensus-based interdisciplinary guideline for melanoma. Part 1: Diagnostics - Update 2019. Eur J Cancer 2020;126:141-58. [Crossref] [PubMed]
21. Garbe C, Amaral T, Peris K, et al. European consensus-based interdisciplinary guideline for melanoma. Part 2: Treatment - Update 2019. Eur J Cancer 2020;126:159-77. [Crossref] [PubMed]
22. Nenclares P, Ap Dafydd D, Bagwan I, et al. Head and neck mucosal melanoma: The United Kingdom national guidelines. Eur J Cancer 2020;138:11-8. [Crossref] [PubMed]
23. Nakamura Y, Asai J, Igaki H, et al. Japanese Dermatological Association Guidelines: Outlines of guidelines for cutaneous melanoma 2019. J Dermatol 2020;47:89-103. [Crossref] [PubMed]
24. Wright FC, Kellett S, Hong NJL, et al. Locoregional management of in-transit metastasis in melanoma: an Ontario Health (Cancer Care Ontario) clinical practice guideline. Curr Oncol 2020;27:e318-25. [Crossref] [PubMed]
25. Coit DG, Thompson JA, Albertini MR, et al. Cutaneous Melanoma, NCCN Guidelines Version 2.2019, NCCN Clinical Practice Guidelines in Oncology. Available online: https://www.nccn.org/guidelines/category_1
26. Rao PK, Barker C, Coit DG, et al. Uveal Melanoma, NCCN Guidelines Version 1.2019. Available online: https://www.nccn.org/guidelines/category_1
27. Amaria RN, Menzies AM, Burton EM, et al. Neoadjuvant systemic therapy in melanoma: recommendations of the International Neoadjuvant Melanoma Consortium. Lancet Oncol 2019;20:e378-89. [Crossref] [PubMed]
28. Wright FC, Souter LH, Kellett S, et al. Primary excision margins, sentinel lymph node biopsy, and completion lymph node dissection in cutaneous melanoma: a clinical practice guideline. Curr Oncol 2019;26:e541-50. [Crossref] [PubMed]
29. Petrella TM, Fletcher GG, Knight G, et al. Systemic adjuvant therapy for adult patients at high risk for recurrent cutaneous or mucosal melanoma: an Ontario Health (Cancer Care Ontario) clinical practice guideline. Curr Oncol 2020;27:e43-e52. [Crossref] [PubMed]
30. Cancer Council Australia Melanoma Guidelines Working Party. Clinical practice guidelines for the diagnosis and management of melanoma. Sydney: Cancer Council Australia. Available online: https://wiki.cancer.org.au/australia/Guidelines:Melanoma
31. Xiao Y, Jiang L, Tong Y, et al. Evaluation of the quality of guidelines for assisted reproductive technology using the RIGHT checklist: A cross-sectional study. Eur J Obstet Gynecol Reprod Biol 2019;241:42-8. [Crossref] [PubMed]
32. Li X, Yu X, Xie Y, et al. Critical appraisal of the quality of clinical practice guidelines for idiopathic pulmonary fibrosis. Ann Transl Med 2020;8:1405. [Crossref] [PubMed]
33. Wang X, Zhou Q, Chen Y, et al. Using RIGHT (Reporting Items for Practice Guidelines in Healthcare) to evaluate the reporting quality of WHO guidelines. Health Res Policy Syst 2020;18:75. [Crossref] [PubMed]
34. Boyd EA, Akl EA, Baumann M, et al. Guideline funding and conflicts of interest: article 4 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc 2012;9:234-42. [Crossref] [PubMed]
35. Wang Q, Duan Y, Liang J, et al. Reporting quality of 2014-2018 clinical practice guidelines on diabetes according to the RIGHT checklist. Endocrine 2019;65:531-41. [Crossref] [PubMed]
36. Zervou FN, Zacharioudakis IM, Pliakos EE, et al. Adaptation of Cost Analysis Studies in Practice Guidelines. Medicine (Baltimore) 2015;94:e2365 [Crossref] [PubMed]