Evaluation of reporting quality in clinical practice guidelines for acute myeloid leukemia using the RIGHT checklist

Introduction

Acute myeloid leukemia (AML) is a highly heterogeneous molecular phenotype characterized by the malignant transformation of hematopoietic stem or progenitor cells (1). As the most common form of acute leukemia in adults, AML remains a devastating disease, with an annual incidence of approximately 16,000 new cases in China and a 5-year overall survival (OS) of only 10% (2). Effective treatment of AML is very challenging (3). For over 4 decades, therapeutic options for AML were limited to high-dose cytotoxic chemotherapy, followed by either allogeneic stem cell transplant or consolidation chemotherapy (4). An improved understanding of the genomic and molecular landscape of AML has resulted in better molecular characterization, leading to more accurate prognostic stratification and treatment decisions. Additionally, advances in treatment with newly approved drugs have resulted in the updating of clinical practice guidelines and paved the way for a new therapeutic era for AML (5).

Clinical practice guidelines (CPGs), which are based on systematic evaluations of evidence, provide recommendations on the management of diseases to guide, optimize, and establish norms for clinical practice (6). An important tool, CPGs standardize the behavior of clinicians to enhance the quality of medical care and also facilitate the allocation of medical resources and protect the rights and interests of patients. Most importantly, updates in CPGs regarding new drugs and therapies can expand the clinical benefit to a broader patient population through the dissemination of guidelines (7). An increasing number of academic organizations and institutions have formulated CPGs for the management of diseases to optimize medical care.

Reporting checklists can assist in developing CPGs and also be used to evaluate the reporting quality of guidelines (8). The Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist, which contains an elaboration statement with detailed information and examples, has been widely recognized as a standard for reporting criteria and is used to assess the reporting quality of CPGs for different diseases (9-12). However, the reporting quality of CPGs for AML has not been evaluated. We used the RIGHT checklist to evaluate the reporting quality of AML CPGs published between 2016 and 2020. The aim of this study is to support a more comprehensive, clear, and transparent reporting of CPGs in this field and to provide recommendations for guideline development in the future.

Methods

Search for AML CPGs

Two independent researchers performed systematic and detailed searches of Medline (via PubMed), Chinese National Knowledge Infrastructure (CNKI), Wanfang, Chinese Biomedical Literature (CBM), and other databases to obtain AML CPGs for the last 5 years [2016–2020], the PubMed search strategy was shown in Appendix 1. In addition, official websites of relevant organizations and medical associations were also searched, including the World Health Organization (WHO), European Society for Medical Oncology (ESMO), National Comprehensive Cancer Network (NCCN), Guidelines International Network (GIN), National Institute for Health and Care Excellence (NICE), American Society of Clinical Oncology (ASCO), and Chinese Society of Clinical Oncology (CSCO). The searches were conducted in January 2021.

Inclusion criteria

CPGs were extracted based on the following inclusion criteria: (I) they were related to AML, including screening, surveillance, diagnosis, treatment, or follow-up of AML; (II) they were published in English or Chinese; and (III) they were published publicly in a peer-reviewed journal or on a website.

Exclusion criteria

Documents were excluded if they were: (I) older versions of guidelines when newer versions were available and accessible; (II) translations and interpretations of guidelines; (III) expert consensus statements; (IV) repeatedly published guidelines, and (V) guidelines for which the full text was inaccessible.

Data extraction

All eligible studies were imported into EndNote X9 to eliminate duplicates. Based on the inclusion and exclusion criteria, the titles, abstracts, and full texts were screened by two researchers. Two additional researchers independently screened and cross-checked the guidelines again. Disagreements were resolved through consensus or consultation with another expert adjudicator.

The RIGHT checklist consists of 22 key items and employs a clear and detailed implementation process. Some items are further divided into several subitems, for a total of 35 items. These items are organized into 7 domains: basic information; background; evidence; recommendations; review and quality assurance; funding, declaration, and management of interests; and other information (Table S1). Four researchers independently assessed the reporting quality of the included guidelines one by one, with “Yes” indicating full or partial reporting of the necessary information and “No” indicating no reporting; if an item did not apply to a particular guideline, it was assigned “Not applicable” (NA) (Table S2). The decision was referred to another expert adjudicator when disagreement was encountered.

Statistical analysis

We calculated the number of checklist items that were reported and defined the reporting quality as (number of reported)/35. The overall reporting rate for the included guidelines, the reporting rate for each domain, and the reporting rate for each item were calculated. If the reporting rate was lower than 50%, the reporting quality of the guideline was regarded as low (13). The data were calculated and analyzed using SPSS 20.0 and Microsoft Office Excel 2019.

Results

Data overview

In total, 341 relevant documents were extracted from the databases. After reading the titles, abstracts, and full texts of the relevant documents, 16 guidelines that met the study criteria were identified (14-29) (Figure 1). Of the 16 CPGs, 13 (81.3%) were written in English, and 3 (18.8%) were written in Chinese. Four (25%) of the 16 guidelines were from the United States, 3 (18.75%) were from China, 3 (18.75%) were from Europe, 2 (12.5%) were from Canada, 2 (12.5%) were from Japan, 1 (6.25%) was from Brazil, and 1 (6.25%) was from India. For 2 (12.5%) of the guidelines, the author type was individual, and the rest (87.5%) were institutional. One (6.25%) CPG was published in 2016, 5 (31.25%) in 2017, 2 (12.5%) in 2018, 2 (12.5%) in 2019, and 6 (37.5%) in 2020. The characteristics of the 16 CPGs for AML are summarized in Table 1.

Figure 1 Flowchart of systematic search.

Reporting quality of the guidelines

The mean reporting rate of the 16 CPGs for AML was 52.9% (28.6–97.1%). Only 7 out of the 16 CPGs (43.8%) had a reporting rate >50%, and 2 guidelines had a reporting rate above 90% (Table 1). Six guidelines reported <40% of the 35 items. The 13 guidelines in English were published in journals and could be retrieved through PubMed, while the 3 Chinese guidelines were accessible only in Chinese journals. The mean reporting rates of guidelines written in Chinese and in English were 35.2% (34.3–37.1%) and 56.9% (28.6–97.1%), respectively.

Reporting quality of the domains

The mean reporting rates in the 7 RIGHT domains of the 16 CPGs were: 79.2 (50–100%) for basic information; 62.5% (37.5–100%) for background; 38.8% (0–100%) for evidence; 53.6% (28.6–100%) for recommendations; 21.9% (0–100%) for review and quality assurance; 32.8% (0–75%) for funding, declaration, and management of interests; and 43.8% (0–100%) for other information (Figure 2).

Figure 2 The mean reporting rate for each item of the RIGHT checklist.

Reporting quality of the items

The details of the reporting quality of each item are summarized in Figure 3. Most of the guidelines identified their document as a guideline and described its focus (items 1a and 1c), with a reporting rate higher than 80%. Similarly, abbreviations and acronyms (item 3) and corresponding developers (item 4) were also sufficiently reported. Items 7 (target population), 9b (list all individuals involved in developing the guideline), 13a/b (provide clear recommendations and present separate recommendations for subgroups) and 20 (access) had reporting rates higher than 80%. Among them, items 1a, 3, 7b, 9b, and 13b were reported by all 16 CPGs (100%). Item 18b was not reported by any of the guidelines, and items 6 (aim of the guideline and specific objectives), 8 (end-users and settings), 10 (healthcare questions), 12 (assessment of the certainty of the body of evidence), 14a/b (rationale/explanation for recommendations), 15 (evidence to decision processes), 16 (external review), 17 (quality assurance), 18a (specific sources of funding), 19b (conflicts of interest), 21 (suggestions for further research), and 22 (limitations of the guideline) were reported by less than 50% of the guidelines.

Figure 3 The mean reporting rate for each domain of the RIGHT checklist.

Discussion

High-quality guidelines provide information on the latest treatment options using reliable evidence, accurate subgroup classification, and transparency, while low-quality guidelines are difficult to interpret and implement, ultimately leading to worse results. The RIGHT checklist was established to assist developers in creating reporting guidelines and help clinicians understand and implement the guidelines.

We used the RIGHT checklist to evaluate 16 AML CPGs of various types published in the past 5 years and found that the guidelines varied greatly in reporting quality. Of the 16 CPGs included in our study, only 7 guidelines reported >50% of the items in the RIGHT checklist, which suggested that most CPGs for AML were of low reporting quality (13).

Of the 16 guidelines, most were developed by European, American, and Asian organizations, and the overall reporting rate of guidelines in Europe and America was slightly higher than that in Asia. In addition, the completeness of the CPGs written in Chinese was lower than that in English. These results showed that an established norm for guideline developers from different geographical regions is particularly important for developing CPGs with high reporting quality.

Of the 7 domains, the basic information domain had the highest reporting rate at 79.2%. The most poorly reported domain was review and quality assurance, with a reporting rate of 21.9%, suggesting that CPG developers tended to include basic information but ignored the review and quality assurance domain. Although most of the items had high reporting rates in the basic information domain, the publication year was not clear for some guidelines, and a summary of the recommendations was not always included. These items are essential for readers who want to know how up-to-date CPGs are and for clinicians to be able to quickly extract the information they require and assess the scope of the guidelines.

Under the background domain, all 16 of the guidelines included subgroups given special consideration (item 7b), which is essential for individualized treatment. However, few guidelines described the intended users of the guideline (item 8a) or the settings in which the guideline was intended (item 8b). Additionally, the developers often did not report on the contributors in guideline development (9a). Reporting on contributors and their roles and responsibilities (e.g., steering group, guideline panel, external reviewer, systematic review team, methodologist, etc.) could increase the accuracy and reliability of the guidelines. The background of AML guidelines should be fully described in the future.

For the evidence domain, items 10 (healthcare questions), 11 (systematic reviews), and 12 (assessment of the certainty of the body of evidence) had a reporting rate lower than 60%. While guideline development is always based on systematic reviews, reporting these items are essential for improving the transparency and accuracy of the guidelines. For item 10, when a guideline states the basis for recommendations in PICO (population, intervention, comparator, and outcome) format and indicates how outcomes were selected and sorted, readers can easily identify useful information and make evidence-based decisions. Items 11 and 12 can greatly assist readers in understanding the evidence and evaluating the accuracy of CPGs. Moreover, these items are essential for peer review and permit any shortcomings in the guidelines to be identified and remedied.

Under the recommendations domain, most CPGs did not report on item 14 (rationale/explanation for recommendations), which suggested that the developers did not regard issues such as values, preferences, cost, and resource implications, among other factors, as important when developing guidelines. Although different management methods may be suitable for specific subgroups, some new drugs are not approved in every country, and even if approved, not every patient can afford expensive drugs. The guidelines that lacked item 14 did not accurately present recommendations and thus are not applicable for larger populations. This means that hematologists cannot easily use the guidelines in different clinical situations. In addition, item 15 (evidence to decision processes) was not fully reported, which indicated that the development of the guideline was not transparent or rigorous enough.

For the review and quality assurance domain, items 16 (external review) and 17 (quality assurance) had reporting rates lower than 30%. And the reporting rate of these items in other diseases such as gastric cancer is also low (30). The lack of such information may lead readers to doubt the quality of the guidelines.

For the funding, declaration, and management of interests’ domain, the RIGHT checklist showed that the reporting quality was low. Item 18b (describing the roles of funders in the different stages of guideline development and in the dissemination and implementation of the recommendations) was not reported by any of the guidelines, suggesting that developers did not pay enough attention to this domain. Experts of multiple interdisciplinary teams may receive grants or consulting fees from the company that developed the drugs included in guidelines. A lack of funding information may lead to the inference that the guideline recommendations may be influenced by multiple interested parties, reducing the credibility of the guidelines.

For the other information domain, some guidelines failed to describe the gaps in the evidence or provide suggestions for future research (or both). Most of the guidelines failed to describe any limitations in the guideline development process or indicate how these limitations might have affected the validity of the recommendations. Such information could provide a reference for readers to use in assessing the suitability of the recommendations and offer guidance for updates and research in the future.

Strengths and limitations

This is the first study to assess the reporting quality of guidelines for AML using the RIGHT checklist to assist developers in standardizing the reporting quality of future guidelines and to help hematologists understand and implement the guidelines. However, there were some limitations to this study. First, only guidelines written in Chinese and English were included, which might have caused selection bias. Second, only guidelines published in journals and websites were included, while guidelines published in books or government documents were not analyzed. In addition, while the RIGHT checklist is not used to assess methodology quality or the effectiveness of guideline recommendations, it can assist guideline developers with reporting and help readers to better understand and implement guidelines.

Conclusions

Our evaluation of AML guidelines using the RIGHT checklist suggested that reporting quality is poor and varies greatly among guidelines. In most of the guidelines we evaluated, the domains with low reporting quality were review and quality assurance; evidence; and funding, declaration, and management of interests. AML guideline developers should pay more attention to these items to improve the standardization of reporting, resulting in reporting that is clearer, more complete, and more transparent to better disseminate and implement advanced guidelines in the future.

Acknowledgments

Funding: This study was supported by Henan Provincial Science and Technology Research Project (202102310157), Medical Science and Technology Research Plan (Joint Construction) Project of Henan Province (LHGJ20190676), and Health Commission of Henan Province (RKX202002011).

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/atm-21-4323). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Schuringa JJ, Bonifer C. Dissecting Clonal Heterogeneity in AML. Cancer Cell 2020;38:782-4. [Crossref] [PubMed]
2. Wang J, Lü S, Yang J, et al. A homoharringtonine-based induction regimen for the treatment of elderly patients with acute myeloid leukemia: a single center experience from China. J Hematol Oncol 2009;2:32. [Crossref] [PubMed]
3. Li D, Yu Z, Wang T, et al. The role of the novel LincRNA uc002jit.1 in NF-kB-mediated DNA damage repair in acute myeloid leukemia cells. Exp Cell Res 2020;391:111985 [Crossref] [PubMed]
4. Döhner H, Wei AH, Löwenberg B. Towards precision medicine for AML. Nat Rev Clin Oncol 2021;18:577-90. [Crossref] [PubMed]
5. Short NJ, Konopleva M, Kadia TM, et al. Advances in the Treatment of Acute Myeloid Leukemia: New Drugs and New Challenges. Cancer Discov 2020;10:506-25. [Crossref] [PubMed]
6. Murad MH. Clinical Practice Guidelines: A Primer on Development and Dissemination. Mayo Clin Proc 2017;92:423-33. [Crossref] [PubMed]
7. Kredo T, Bernhardsson S, Machingaidze S, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care 2016;28:122-8. [Crossref] [PubMed]
8. Chen Y, Yang K, Marušic A, et al. A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. Ann Intern Med 2017;166:128-32. [Crossref] [PubMed]
9. Wang Q, Duan Y, Liang J, et al. Reporting quality of 2014-2018 clinical practice guidelines on diabetes according to the RIGHT checklist. Endocrine 2019;65:531-41. [Crossref] [PubMed]
10. Wang Z, Zhang Y, Guo W, et al. Reporting specifications regarding epilepsy practice guidelines based on the RIGHT reporting checklist: an analysis. BMJ Open 2019;9:e029589 [Crossref] [PubMed]
11. Wang C, Luo X, Li M, et al. Reporting quality of clinical practice guidelines regarding gout and hyperuricemia according to the RIGHT checklist: systematic review. Syst Rev 2021;10:99. [Crossref] [PubMed]
12. Zhao Y, Li Y, Li J, et al. Reporting quality of chronic kidney disease practice guidelines according to the RIGHT statement: a systematic analysis. Ther Adv Chronic Dis 2020;11:2040622320922017 [Crossref] [PubMed]
13. Xiao Y, Jiang L, Tong Y, et al. Evaluation of the quality of guidelines for assisted reproductive technology using the RIGHT checklist: A cross-sectional study. Eur J Obstet Gynecol Reprod Biol 2019;241:42-8. [Crossref] [PubMed]
14. Klepin HD, Neuendorff NR, Larson RA, et al. Treatment of acute promyelocytic leukemia in older patients: recommendations of an International Society of Geriatric Oncology (SIOG) task force. J Geriatr Oncol 2020;11:1199-209. [Crossref] [PubMed]
15. Heuser M, Ofran Y, Boissel N, et al. Acute myeloid leukaemia in adult patients: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol 2020;31:697-712. [Crossref] [PubMed]
16. Fujita H, Ishikawa Y, Yokoyama Y. JSH practical guidelines for hematological malignancies, 2018: I. Leukemia-2. Acute promyelocytic leukemia (APL). Int J Hematol 2020;111:747-60. [Crossref] [PubMed]
17. Kiyoi H, Yamaguchi H, Maeda Y, et al. JSH Practical Guidelines for Hematological Malignancies, 2018: I. Leukemia-1. Acute myeloid leukemia (AML). Int J Hematol 2020;111:595-613. [Crossref] [PubMed]
18. Sekeres MA, Guyatt G, Abel G, et al. American Society of Hematology 2020 guidelines for treating newly diagnosed acute myeloid leukemia in older adults. Blood Adv 2020;4:3528-49. [Crossref] [PubMed]
19. Available online: https://www.nccn.org/
20. Bansal D, Davidson A, Supriyadi E, et al. SIOP PODC adapted risk stratification and treatment guidelines: Recommendations for acute myeloid leukemia in resource-limited settings. Pediatr Blood Cancer 2019; Epub ahead of print. [Crossref] [PubMed]
21. Sanz MA, Fenaux P, Tallman MS, et al. Management of acute promyelocytic leukemia: updated recommendations from an expert panel of the European LeukemiaNet. Blood 2019;133:1630-43. [Crossref] [PubMed]
22. de Haas V, Ismaila N, Advani A, et al. Initial Diagnostic Work-Up of Acute Leukemia: ASCO Clinical Practice Guideline Endorsement of the College of American Pathologists and American Society of Hematology Guideline. J Clin Oncol 2019;37:239-53. [Crossref] [PubMed]
23. Döhner H, Estey E, Grimwade D, et al. Diagnosis and management of AML in adults: 2017 ELN recommendations from an international expert panel. Blood 2017;129:424-47. [Crossref] [PubMed]
24. Brandwein JM, Zhu N, Kumar R, et al. Treatment of older patients with acute myeloid leukemia (AML): revised Canadian consensus guidelines. Am J Blood Res 2017;7:30-40. [PubMed]
25. Silla L, Dulley F, Saboya R, et al. Brazilian guidelines on hematopoietic stem cell transplantation in acute myeloid leukemia. Eur J Haematol 2017;98:177-83. [Crossref] [PubMed]
26. Abla O, Kutny MA, Testi AM, et al. Management of relapsed and refractory childhood acute promyelocytic leukaemia: recommendations from an international expert panel. Br J Haematol 2016;175:588-601. [Crossref] [PubMed]
27. Ma J. Guidelines for the diagnosis and treatment of acute promyelocytic leukemia in China (2018 edition). Chinese Journal of Hematology 2018;39:179-83. [PubMed]
28. Wei H. Adult acute myeloid leukemia (non-acute promyelocytic leukemia) Chinese diagnosis and treatment guidelines (2017 edition). Chinese Journal of Hematology 2017;38:177-82.
29. Zhang Y. Chinese Journal of Diagnosis and Treatment of Relapsed and Refractory Acute Myeloid Leukemia (2017 Edition). Chinese Journal of Hematology 2017;38:183-4.
30. Wu X, Li D, Chen H, et al. Evaluation of the reporting quality of guidelines for gastric cancer using the RIGHT checklist. Ann Transl Med 2021;9:1003. [Crossref] [PubMed]

(English Language Editor: A. Muijlwijk)