Viewpoints on the PRIOR statement—a reporting guideline for overviews of reviews

Introduction

On August 9, 2022, The BMJ published an article about the development of the Preferred Reporting Items for Overviews of Reviews of healthcare interventions (PRIOR) statement. The article describes the status quo of reporting on overview of reviews of healthcare interventions, introduces the background and process of developing the PRIOR checklist in detail, and provides explanations and examples of each item. PRIOR fills an important gap in the existing reporting guidelines, improving the completeness, transparency, and accuracy of reporting of overviews of reviews (1).

In this commentary, we briefly summarize the development background, approach, and items of PRIOR, analyze the limitations of PRIOR in the context of the real challenges of the applications of overviews of reviews in decision-making, and critically appraise the development methodology and applicability of this new tool. We also discuss the status quo and challenges of PRIOR and other reporting guidelines and give suggestions on how to promote their application in clinical research and decision-making, strengthening the voice of Chinese researchers in the discussion about these reporting guidelines.

Evidence-based decision-making, systematic reviews, and overviews of reviews

Evidence-based decision-making requires the best available evidence, which is collected and synthesized based on a systematic search of the literature. The systematic review, as a key research design, locates at the top of the evidence hierarchy pyramid (2,3). The definition of practice guidelines also indicates guideline development must be based on systematic reviews (4). Systematic reviews have a shorter development cycle than primary studies such as randomized controlled trials and cohort studies. These factors have contributed to the rapid growth in the number of systematic reviews. But at the same time, some problems have arisen, such as the risk of obtaining inconsistent or even contradictory results in different systematic reviews on the same healthcare topic. This does not only result in a waste of research resources but also misleads clinical decision-making (5).

Overviews of reviews, which search, screen evaluate, and synthesize multiple systematic reviews, can address the above problems (6,7). Since an overview of reviews provides a comprehensive search and critical evaluation of the included systematic reviews, it is often regarded as a “systematic review of systematic reviews”. The available studies and the survey conducted by the PRIOR working group indicated a rapid growth in the number of overviews of reviews, but the reporting contents in the overviews tend to be inconsistent. Even the names of the study type are diverse, with “overview of reviews”, “overview”, “overview of systematic reviews”, “review of reviews”, “review of systematic reviews”, or “umbrella review” being all used to denote this kind of review (8-10). Moreover, the reporting of methods in overviews of reviews is also often inadequate, especially for the key steps such as summarizing evidence or evidence synthesis methods, which hinders the application of overviews of reviews in clinical decision-making. Reporting guidelines are therefore urgently needed to standardize reporting in overviews of reviews (11-13).

Introduction to PRIOR

PRISMA 2020 is the latest reporting guideline for systematic reviews (14). Instead of developing an extension of PRISMA (15), the PRIOR working group decided to develop a new checklist, strictly following the approach recommended by the EQUATOR network (16). The development of PRIOR consisted of four steps: (I) establishment of a core team and an expert advisory group to launch the project; (II) formation of the list of initial items by developing a systematic review of the reporting quality and a scoping review of methodological guidance and challenges related to overviews of reviews; (III) recruitment of participants with relevant experience to conduct two online Delphi surveys and the organization of two virtual meetings among the core team members, three expert advisory group members, and ten expert panel members; and (IV) finalizing the draft guideline, conducting a pilot test and confirming the final checklist (1).

The structure of PRIOR is similar to PRISMA 2020 and is divided into seven sections: title, abstract, background, methods, results, discussion, and other information, containing together 27 items and 19 sub-items. PRIOR also provides a glossary of terminology and a flow diagram for developing an overview of reviews. In addition to the different type of studies included in the systemic review, PRIOR differs from PRISMA 2020 in the following ways: (I) two items on effect measures and results of individual studies were omitted; (II) two items on primary study overlap and author information were added; (III) a new sub-item of the definition of “systemic reviews” used in the inclusion criteria was added; (IV) a new sub-item about the overlap of studies during the selection process was added; (V) two new sub-items about the methods used to assess the methodological quality of each included systematic review and risk of bias in supplemental primary studies (if included) were added; and (VI) a sub-item about describing the methods used to tabulate or visually display the results was omitted (1,14).

Limitations of the development process of the PRIOR statement

The development process of the PRIOR statement had also some limitations. First, the expert group spanned five continents and sixteen countries but included no experts from China and the participation from low- and middle-income countries in the working group was in general limited. Chinese researchers have published more than three thousands systematic reviews and over thirty overviews of reviews annually during the past years (17-19), thus having substantial experience in this topic. The inclusion of Chinese experts could have improved the representativeness of the PRIOR expert group. Second, the working group requested those who declined participation to recommend another candidate for potential inclusion, but the response rate of the recruited survey respondents remained low. The working group did not take other measures to recruit more experts. Third, the developers did not set any requirements for the distribution of fields of experience, roles, or settings among the participants, thus potentially causing selection bias (14).

Application of PRIOR in practice

Researchers utilizing PRIOR should pay attention in particular to the following aspects. First, PRIOR is intended to guide and evaluate the writing and reporting of overviews of reviews and cannot be used for the evaluation of methodological or evidence quality. Second, the type of primary studies needs to be confirmed: PRIOR is a reporting guideline for overviews of reviews, not for conventional systematic reviews. It is important to note that network meta-analyses, cumulative meta-analyses, evidence mappings, and scoping reviews are types of evidence reviews, not overviews of reviews. Third, the type of clinical questions needs to be taken into consideration: PRIOR was developed for overviews of reviews on healthcare interventions, not other questions (e.g., qualitative reviews, or reviews of prognostic or diagnostic accuracy). Finally, PRIOR may not be applicable to all types of healthcare interventions such as acupuncture and traditional Chinese medicine (TCM). Chinese researchers have contributed substantially to the adaptation of reporting guidelines for TCM interventions, for example by developing the extension of PRISMA for acupuncture (20), We therefore call Chinese researchers to actively participate in the extension of PRIOR for more types of interventions.

Challenges and recommendations for the application of PRIOR and other reporting guidelines

Based on the analysis of PRIOR and the application of other reporting guidelines, we conclude that reporting guidelines currently face several challenges. First, reporting guidelines are still lacking in some specific areas: for example, no reporting guidelines for some types of studies, conflicts of interest, or qualitative research. Second, in addition to the checklist, reporting guidelines also include explanations and examples of the items written almost entirely in English, making it difficult for authors, editors and reviewers especially in non-English speaking settings to quickly grasp the correct use of the reporting guidelines. Third, the reporting guidelines themselves are of varying quality, and most do not follow the recommendations for developing reporting guidelines (21).

In response to the above challenges, we propose the following recommendations. Researchers developing reporting guidelines should strictly follow the approach recommended by the EQUATOR Network (21), determine whether an extension to existing reporting guidelines or a new reporting guideline is needed, and explore methods for assessing the quality of existing reporting guidelines. Databases of reporting guidelines, such as the EQUATOR database, should collect existing reporting guidelines as comprehensively as possible, promote commonly used reporting guidelines, and provide more explanatory materials for reporting guidelines that have not been used for a long time. Clinical researchers should learn to utilize reporting guidelines when writing research articles actively. Journal editors should select the most appropriate reporting guidelines to review the manuscripts, and journal publishers should provide free educational resources in multiple languages for journal editors and authors to study.

Summary

PRIOR and other reporting guidelines still face many challenges in their application. A joint effort among researchers developing reporting guidelines, authors, journal editors, and journal publishers is needed to advance evidence-based clinical decision-making.

Acknowledgments

We thank China Scholarship Council for funding Nan Yang and Hui Liu.

Funding: NY and HL received funding from China Scholarship Council (No. 202206180007 and No. 202206180006).

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Annals of Translational Medicine. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://atm.amegroups.com/article/view/10.21037/atm-22-5724/coif). NY is funded by China Scholarship Council (No. 202206180007); HL is funded by China Scholarship Council (No. 202206180006). KZ reports that she is a section editor of Annals of Translational Medicine and a full-time staff member of AME Publishing Company, the publisher of Annals of Translational Medicine. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Gates M, Gates A, Pieper D, et al. Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. BMJ 2022;378:e070849. [Crossref] [PubMed]
2. Sackett DL, Rosenberg WM, Gray JA, et al. Evidence based medicine: what it is and what it isn't. BMJ 1996;312:71-2. [Crossref] [PubMed]
3. SUNY Downstate Medical Research Library of Brooklyn: Evidence-Based Medicine Course. Evidence Based Pyramid. Available online: https://guides.downstate.edu/c.php?g=856794&p=6831536. Cited 11.21.2022.
4. Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press, 2011.
5. Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews. CMAJ 1997;156:1411-6. [PubMed]
6. Becker LA, Oxman AD. Overviews of reviews. Cochrane Handbook for Systematic Reviews of Interventions. In: Higgins JPT, Green S, editors. Hoboken: John Wiley & Sons, 2008:607-31.
7. Caird J, Sutcliffe K, Kwan I, et al. Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evidence and Policy 2015;11:16. [Crossref]
8. Hartling L, Chisholm A, Thomson D, et al. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One 2012;7:e49667. [Crossref] [PubMed]
9. Gates M, Gates A, Guitard S, et al. Guidance for overviews of reviews continues to accumulate, but important challenges remain: a scoping review. Syst Rev 2020;9:254. [Crossref] [PubMed]
10. Pieper D, Buechter R, Jerinic P, et al. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol 2012;65:1267-73. [Crossref] [PubMed]
11. Lunny C, Brennan SE, Reid J, et al. Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. J Clin Epidemiol 2020;118:69-85. [Crossref] [PubMed]
12. Lunny C, Brennan SE, McDonald S, et al. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev 2017;6:231. [Crossref] [PubMed]
13. Lunny C, Brennan SE, McDonald S, et al. Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev 2018;7:159. [Crossref] [PubMed]
14. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372: [PubMed]
15. Pollock M, Fernandes RM, Pieper D, et al. Preferred Reporting Items for Overviews of Reviews (PRIOR): a protocol for development of a reporting guideline for overviews of reviews of healthcare interventions. Syst Rev 2019;8:335. [Crossref] [PubMed]
16. Altman DG, Simera I, Hoey J, et al. EQUATOR: reporting guidelines for health research. Open Med 2008;2:e49-50. [PubMed]
17. Li X, Lu Y, Zhao Z, et al. Bibliometric Analysis of Systematic Review/Meta Analysis Literatures from China. The Journal of Evidence-Based Medicine 2015;15:303-8.
18. Tian J, Chen Y, Yang K, et al. Rogresses and challenges for Meta analysis or systematic review Journal of Lanzhou University 2016;42:42-7. (Medical Sciences).
19. Wang PJ, Wang JX, Cai RJ, et al. Current Status of Overview of Systematic Reviews in China. Chinese Journal of Drug Evaluation 2019;36:5-9.
20. Wang X, Chen Y, Liu Y, et al. Reporting items for systematic reviews and meta-analyses of acupuncture: the PRISMA for acupuncture checklist. BMC Complement Altern Med 2019;19:208. [Crossref] [PubMed]
21. Wang X, Chen Y, Yang N, et al. Methodology and reporting quality of reporting guidelines: systematic review. BMC Med Res Methodol 2015;15:74. [Crossref] [PubMed]