A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern

Introduction

In 2019, an outbreak of a new coronavirus, named the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2; previously known as 2019 novel coronavirus, 2019-nCoV) occurred in Wuhan, Hubei province, China. To February 22, 2020, a total of 77,794 patients had been diagnosed with coronavirus disease-19 (COVID-19), the disease caused by SARS-CoV-2, in 29 countries and regions, causing 2,359 deaths (1). World Health Organization (WHO) considers the epidemic as a high-risk Public Health Emergency of International Concern (PHEIC) (1). PHEIC is defined as an extraordinary situation that constitutes a public health risk to member states other than the country where the situation originated through the international spread of disease, and potentially requires a coordinated international response (2). Locally outbreaking infectious diseases, often known as public health emergency at first, without effective control measures may easily lead to PHEIC. A study found that more than 260 local public health emergencies occurred in Africa between 2016 and 2018, and 87% of African countries occurred at least one public health emergency event (3). Among them, the Ebola virus (EBOV), which outbroke in south African in 2014, has killed over 11,000 people in ten countries and regions (4). This was also one of the PHEICs with the widest influence and largest death toll over the past 20 years.

WHO constantly focuses on how to effectively control local public health emergencies and prevent them from developing into PHEICs. In 2013, WHO published the Emergency Response Framework (ERF) (5), which details how to respond to local public health emergencies, and emphasizes the importance of guidelines in the context of public health emergencies. WHO regards providing guidance for the world in case of public health emergencies as its responsibility (5). The key of controlling public health emergencies is to guide and organize people to quickly and effectively respond to health threats (6,7). High-quality guidelines and recommendations are the premise to effectively respond to public health emergencies. To guide guidelines developers to develop high-quality guidelines, WHO published a handbook for guideline development in 2014 (8), and then published a framework and toolkit in 2017 to guide the development process and improve the quality of guidelines developing in context of public health emergencies (9). Some scholars have also discussed the applicability and feasibility of the WHO rapid guideline development standards (10) from the WHO Handbook for Guideline Development (8) under public health emergencies, others investigated the reporting of declarations and conflicts of interest in WHO guidelines (11), and considered that the WHO rapid guideline development process still had some problems, for example related to the roles of different members of the guideline development group. It is necessary to reduce the bias and improve the method and reporting quality of rapid guidelines (10) in particular for reporting of funders and their role, and declaration processes (11) to make sure that they can better guide the response to public health emergencies.

In response to public health emergencies, such as infectious diseases, WHO and many other countries have issued a great number of guidelines to guide clinicians and public health professionals how to treat and control epidemics. Some researchers have investigated the quality of guidelines for viruses that have caused PHEICs (12), but none of them have so far studied the development and reporting quality of SARS-CoV-2 guidelines. We therefore aimed to examine the methodological and reporting quality of guidelines on five selected viruses that had caused PHEIC in the past 20 years: the severe acute respiratory syndrome coronavirus (SARS-CoV; 2003), Ebola virus (EBOV,2014), Middle East respiratory syndrome coronavirus (MERS-CoV; 2015), Zika virus (2016), and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2; 2020) (4), hereinafter referred to as “the five viruses”. The overarching goal of this evaluation was to inform WHO and other organizations developing such guidelines about the weaknesses for producing these types of guidelines.

Methods

Search strategy

We systematically searched MEDLINE (via PubMed), China Biomedical Literature database (CBM), China National Knowledge Infrastructure (CNKI) and Wanfang Data. Websites of international organizations government health institutions developing guidelines were also searched, including National Institute for Health and Care Excellence (NICE), National Guideline Clearinghouse (NGC), Healthcare Improvement Scotland (SIGN), World Health Organization (WHO) and Guidelines International Network (GIN). All databases were searched from their inception to February 02, 2020, and the languages were restricted to Chinese and English. The search terms included “Coronavirus”, “Middle East Respiratory Syndrome Coronavirus”, “MERS”, “SARS”, “2019-nCoV”, “Wuhan-Cov”, “Ebola virus”, “Zika”, “Practice guideline”, “Recommendation*” and “Statement*”. (Full search strategies are presented in Supplementary Appendix 1). In addition, supplementary searches were conducted on Google Scholar, Medlive, the official website of the National Health Commission of the People’s Republic of China (http://www.nhc.gov.cn/), the official website of the Centers for Disease Control and Prevention of the United States (https://www.cdc.gov/), the official website of the Chinese Center for Disease Control and Prevention (http://www.chinacdc.cn/), and preprint platforms.

Inclusion and exclusion criteria

We included practice guidelines for SARS-CoV, Ebola virus, MERS-CoV, Zika virus, SARS-CoV-2 and the diseases they caused. As practice guidelines we considered documents that contain recommendations aiming to help clinicians, public health professionals, patients, susceptible populations, or any other target audience, to make appropriate decisions in a specific clinical or public health emergency background.

The following types of documents were excluded: (I) documents containing diagnosis and treatment consensus or standards; (II) comprehensive guidelines on more general topics, containing information about the five viruses; (III) translations or summaries of guidelines; (IV) protocols of guidelines; (V) guidelines where the full text was not available; and (VI) old versions of guidelines with updated version available.

Screening

The search results were imported into an EndNote library (version X9.1). Four researchers (Siya Zhao; Xufei Luo; Junxian Zhao; Jin Cao) screened the guidelines independently in pairs following the pre-defined exclusion criteria, and the results were cross-checked after screening. Disagreements were discussed and solved with another researcher (Yaolong Chen).

Data extraction

Fifteen researchers (Siya Zhao, Jin Cao, Zijun Wang, Qianling Shi, Junxian Zhao, Shuya Lu, Hairong Zhang, Yangqin Xun, Ling Wang, Jianjian Wang, Qi Wang, Jingyi Zhang, Yunlan Liu, Xiaomin Nie and Xianzhuo Zhang) were divided into five groups of three. Each group extracted the data from the retrieved guidelines independently by using a standardized data collection form. Disagreements were resolved by consensus. Extracted data included title, scope and purpose, publication year, publication format, developing organization, development country or region, development method (evidence-based or not) and funding. The publication year was dichotomized depending on if the guideline was published in the year of the outbreak of the respective virus was confirmed (2003 for SARS-CoV, 2014 for Ebola virus, 2015 for MERS-CoV, 2016 for Zika virus and 2020 for SARS-CoV-2) (4), or thereafter. The guideline development method was defined as evidence-based, if the recommendations were based on a comprehensive search of evidence.

Methodological quality appraisal of guidelines

We employed the latest version of the AGREE-II instrument (13) to evaluate the methodological quality of each included guideline. Five groups of appraisers with three appraisers in each group independently evaluated the guidelines. For each guideline, 23 items within six domains (“scope and purpose”, “stakeholder involvement”, “rigour of development”, “clarity and presentation”, “applicability” and “editorial independence”) were evaluated on a scale ranging from 1 (absence of item) to 7 (reported with exceptional quality). For each guideline, we calculated the quality score for each domain (the scaled domain score) following the AGREE-II manual (13). Then we calculated and reported the mean score over all guidelines for a given virus. Figure 1 show an example calculation for the score of Domain 1 for guidelines on one virus.

Figure 1 Example of three appraisers give the following scores for Domain 1 (Scope & Purpose) for one guideline.

Reporting quality appraisal of guidelines

We employed the RIGHT Statement (14) to evaluate the reporting quality of the included guidelines. Two researchers evaluated the guidelines independently. For each guideline, we evaluated 22 items (35 subitems in total) grouped into the following seven domains: “basic information”, “background”, “evidence”, “recommendations”, “review and quality assurance”, “funding and declaration and management of interests” and “other information”. Each item was evaluated as either “reported” (guideline was reported with exceptional quality containing the majority information), “not reported” (relevant information on the item was lacking), or “not applicable” (the item did not need to be evaluated). In the case of conflict, another researcher (Yaolong Chen) was consulted and agreement was reached by consensus. The domain reporting rate for each guideline was then calculated by dividing the number of items reported by the total number of items in the domain, including those evaluated as “reported”, “not reported” and “not applicable”. Finally, we calculated the mean reporting rate of each domain for guidelines of each virus.

Statistical analysis and quality control

Before the formal extraction and evaluation, 15 researchers were trained on a random sample of two guidelines (from outside of the sample) to improve consistency. Consistency was evaluated by intra-class correlation coefficient (ICC). We carried out subgroup analyses for the publication year since outbreak, country and region, format of publication (peer-reviewed journal, or website only), and developing organization. Statistical analyses were conducted by using SPSS version 25.0 through t value of student’s t-test, variance analysis or Kruskal-Wallis H test depending on the type of data. We analyzed the relationship between evaluation results of methodological and reporting quality for all 81 included guidelines by Spearman correlation.

Results

Search results

We identified 1,752 records from the databases and 41 records from guideline websites, manual searches and other additional sources. Twenty-two duplicates were excluded, and 1,771 records were considered to be potentially relevant. After screening for titles, abstract and full-text, a total of 81 guidelines were included (Figure 2).

Figure 2 Flow diagram of the selection process.

Guideline characteristics

A total of 81 guidelines were included, including 21 SARS-CoV guidelines, 11 EBOV guidelines, 9 MERS-CoV guidelines, 10 Zika virus guidelines and 30 SARS-CoV-2 guidelines. The characteristics of the 81 included guidelines are shown in Table 1.

Table 1 Characteristic of 81 included guidelines
Full table

A total of 63 (77.8%) guidelines were published in the calendar year the respective virus outbreak was confirmed (2003 for SARS-CoV, 2014 for Ebola virus, 2015 for MERS-CoV, 2016 for Zika virus and 2020 for SARS-CoV-2) (4). Only fifteen (18.5%) of the guidelines followed the principles of evidence-based guideline development, meaning that the recommendations were based on a comprehensive search of the available evidence. Only four (4.9%) guidelines reported funding sources and the sponsor.

Quality assessment of guidelines

Methodological quality

The ICC value for the pilot test using AGREE-II was 0.81, indicating a high reliability on the scores.

None of the 81 guidelines scored more than 85.0% in any domain of the AGREE-II assessment. The mean scores of the six domains over all guidelines for a particular virus were 3.7–32.2% (SARS-CoV), 12.7–44.4% (MERS-CoV), 9.1–47.5% (Ebola virus), 9.1–47.5% (Zika virus) and 3.2–34.6% (SARS-CoV-2) (Figure 3).

Figure 3 Mean AGREE-II scores of guidelines for the five viruses in the six domains.

In the domains of “Scope and purpose” and “Clarity and presentation”, most guidelines for the five viruses had relatively high scores, but only up to about 80.0%. The mean score of Zika virus guidelines scored highest in the “Scope and purpose” domain (50.2%), and the EBOV guidelines in the domain of “Clarity and presentation” (47.5%).

In the domains of “Rigour of development” and “Editorial independence”, none of the guidelines scored more than 85.0%. The lowest mean score of the “Scope and purpose” domain belong to the SARS-CoV-2 guidelines (3.2%), and the SARS-CoV guidelines scored (4.8%) the least mean score in the domain of “Clarity and presentation”. In addition, the majority of included guidelines for all five viruses [nineteen (95.0%) of the SARS-CoV, six (66.7%) of the MERS-CoV, eight (72.7%) of the Ebola virus, six (60.0%) of the Zika virus, and twenty-seven (90%) of the SARS-CoV-2 guidelines] did not report any information related to “Editorial independence”.

Reporting quality

In the domains of “Basic information” and “Background”, most of guidelines had relatively high reporting rates, however no more than 90.0%. The mean reporting rates of the Zika virus guidelines are the most in these two domains (41.7% and 60.0%), having scores of 90.0% in the items of “Brief description of the health problem(s)” and “Aim(s) of the guideline and specific objectives” (Figure 4).

Figure 4 Mean RIGHT scores of guidelines for the five viruses in seven domains.

In the domains of “Evidence”, “Review and quality assurance” and “Funding and declaration and management of interests”, most of the guidelines for the five viruses are not over 80.0% (except for a Zika virus guideline and a SARS-CoV-2 guideline scored 100% in the domain of “Review and quality assurance”). The SARS-CoV and SARS-CoV-2 guidelines had the lowest mean reporting rates (3.8% and 4.7%, respectively) in the domain of “Evidence”. None of the SARS-CoV and EBOV guidelines reported any items in the domain of “Review and quality assurance”.

The mean reporting rate of the SARS-CoV-2 guidelines did not exceed 50% for any domain. In the domain of “Review and quality assurance”, only 6.7% of the items were reported in the SARS-CoV-2 guidelines, and for the specific items “Executive summary”, “Guideline development groups” and “Health care questions”, the reporting rates were below 5%.

Subgroup analysis

The results of the subgroup analysis are shown in Tables 2,3. In the subgroup analysis of publication year, we compared the evaluation results for methodological and reporting quality between guidelines published within one year (≤1 year) and more than 1 year (>1 year) from the outbreak. For SARS-CoV, the guidelines published at least one year after the outbreak had a higher quality of methodology (P=0.005) and reporting (P=0.026). In the subgroup analysis of publication country/region, results of the variance analysis showed that both the methodological (P=0.001) and reporting (P=0.005) quality of the SARS-CoV guidelines differed across countries and/regions. Guidelines on SARS-CoV-2 in journals had a higher quality of methodology (P=0.003) and reporting (P=0.001) than guidelines published on websites, and the quality also differed across the developer organizations (P=0.006).

Table 2 Subgroup analysis of AGREE-II scores for the guidelines of the five viruses (mean ± SD)
Full table

Table 3 Subgroup analysis of RIGHT scores for the guidelines of the five viruses (mean ± SD)
Full table

Correlation analysis of evaluation results

The results showed that there was a positive correlation between AGREE-II and RIGHT scores (R=0.716, P<0.001) (Figure 5). The guidelines with high methodological quality therefore tend to also have high reporting quality.

Figure 5 Scatter plot for the correlation between AGREE-II and RIGHT scores.

Discussion

Over the past two decades, serious outbreaks caused by the SARS-CoV, MERS-CoV, EBOV and Zika viruses have occurred around the world and triggered PHEIC (4), and the SARS-CoV-2 is currently threatening to become the next global public health emergency (1). We found that the methodological and reporting quality of the guidelines on these five viruses tended to be low. The mean scores of each domain of AGREE-II for the guidelines of each virus were all below 60%. The mean reporting quality scored also below 60% for all viruses and RIGHT domains. In AGREE-II, the domains of “Rigour of development” and “Editorial independence” scored very low. In RIGHT, the domains of “Evidence”, “Review and quality assurance” and “Funding and interests” also had poor scores. Especially none of the guidelines for SARS-CoV and EBOV reported any information in the domain of “Review and quality assurance” of RIGHT statement. There may be two possible reasons for the above results. First, since the RIGHT Statement was not published until 2017 (15), and all the viruses except for the SARS-CoV-2 emerged before 2017 (4), the guidelines therefore had no reporting standards to refer to. This can thus lead to lower reporting quality. Second, due to the nature of public health emergencies, guidance documents or recommendations need to be developed within a short period of time. The developers may not have time to organize an appropriate guideline development group include methodologists, or conduct systematic reviews of evidence and literature.

In 2017, WHO issued guidance for interim guidelines development of public health emergencies (9), and in the same year, Chen and colleagues developed the RIGHT statement to assist developers in reporting guidelines (15). But the results of our study showed that both methodological and reporting quality in the guidelines of SARS-CoV-2 remain poor, especially in the domains of “Editorial independence” (AGREE-II) and “Evidence” (RIGHT). In addition, we also found that the quality in the guidelines for SARS-CoV were similar to the guidelines of SARS-CoV-2. In a subgroup analysis, we also found that the reporting quality differed across countries for SARS-CoV guidelines, and across developer organizations for SARS-CoV-2 guidelines. Zika virus guidelines, which tended to have higher scores on both methodological and reporting quality, were mostly developed or published by the WHO and the United States CDC (Center for Disease Control and Prevention). The majority of SARS-CoV and SARS-CoV-2 guidelines were developed or published by Chinese institutions, which is understandable as China was most affected by these two outbreaks (4). These results suggest that Chinese guideline developers should strengthen the rigor, applicability and editorial independence in the development of such guidelines, to provide better guidelines to fight the ongoing SARS-CoV-2 epidemic and future public health emergencies in China.

China is experiencing increased health care use and expenditures, without sufficient controls to ensure quality and value, and cost-conscious and patient-centered guidelines based on the best available evidence could help establishing these quality and practice measures (16). So, it is important for China to improve quality of guideline. In view of our results, we put forward the following suggestions for Chinese guideline developers. First, guideline development process should be reported more transparently and according to the RIGHT Statement. Second, evidence-based methods should be applied in guideline development and evidence translation (16), and methodologists should be involved in the development process to support methodological and reporting quality. Third, the process of peer review and quality control should be strengthened in guideline development. Finally, the important role of systematic reviews of existing evidence in the process of formulating the guidance should be emphasized.

There were also some limitations when using the AGREE-II and RIGHT to assess the guidelines for these five viruses. The majority of the guidelines of public health emergencies are issued for the general public, meaning that the users and target populations are often the same. This problem is however not addressed in either AGREE-II or RIGHT (13,14). Since there are still many unknown factors related to SARS-CoV-2 and many studies about it are ongoing, the guidelines are being updated rapidly. The results and conclusion of our study regarding the SARS-CoV-2 guidelines therefore only reflect the early stages of the epidemic, and the quality of guidelines may improve later.

Conclusions

The methodological and reporting quality of guidelines on the five viruses (SARS-CoV, Ebola virus, MERS-CoV, Zika virus, and SARS-CoV-2) that have caused serious public health emergencies tends to be poor. Even in case of a rapidly developing infectious disease epidemic and lack of scientific data, the principles of evidence-based medicine and transparent reporting should be followed in guideline development as much as possible. When developing rapid advice guidelines and emergency guidelines in the case of a public health emergency, it is essential to follow WHO Handbook for Guideline Development (8) and the RIGHT statement to ensure the high quality and transparent, comprehensive and clear reporting.

Supplementary

Supplementary Appendix 1 Search Strategy

PubMed

1 Coronavirus [Title/Abstract]

#2 "Middle East Respiratory Syndrome Coronavirus" [Title/Abstract]

#3 MERS [Title/Abstract]

#4 SARS [Title/Abstract]

#5 CoV [Title/Abstract]

#6 HCoV [Title/Abstract]

#7 "2019-nCoV" [Title/Abstract]

#8 "Wuhan-Cov" [Title/Abstract]

#9 Wuhan Coronaviru* [Title/Abstract]

#10 "Ebola virus" [Title/Abstract]

#11 Zika [Title/Abstract]

#12 OR#1-#11

#13 "Coronavirus" [Mesh]

#14 "Middle East Respiratory Syndrome Coronavirus"[Mesh]

#15 "Coronavirus Infections"[Mesh]

#16 "SARS Virus"[Mesh]

#17 "Ebola virus" [Mesh]

#18 "Zika Virus"[Mesh]

#19 "Zika Virus Infection" [Mesh]

#20 OR/#13-#19

#21 #12 OR #21

#22 "Practice guideline" [Publication Type]

#23 Guideline* [Title/Abstract]

#24 Guidance* [Title/Abstract]

#25 Recommendation* [Title/Abstract]

#26 Statement*[Title/Abstract]

#27 OR #23-#27

#28 #22 AND #28

Acknowledgments

Funding: This work was supported by 2020 Key R&D project of Gansu Province.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform (available at http://dx.doi.org/10.21037/atm.2020.03.130). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Novel Coronavirus(2019-nCoV) Situation Report – 33[Internet]. World Health Organization; c2020 [cited 2020 Feb 22]. Available online: https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200222-sitrep-33-covid-19.pdf?sfvrsn=c9585c8f_2
2. World Health Organization. International health regulations 2005. 2nd edition. Geneva: The Network, 2005.
3. Talisuna AO, Okiro EA, Yahaya AA, et al. Spatial and temporal distribution of infectious disease epidemics, disasters and other potential public health emergencies in the World Health Organisation Africa region, 2016-2018. Globalization Health 2020;16:9. [Crossref] [PubMed]
4. Disease outbreak news [Internet]. World Health Organization; c2020 [cited 2020 Feb 7]. Available online: https://www.who.int/csr/don/archive/year/en/
5. World Health Organization. Emergency response framework. 2nd edition. Geneva: The Network, 2017.
6. PLoS Medicine Editors. How is WHO responding to global public health threats? PLoS Med 2007;4:e197. [Crossref] [PubMed]
7. Rose DA, Murthy S, Brooks J, et al. The Evolution of Public Health Emergency Management as a Field of Practice. Am J Public Health 2017;107:S126-33. [Crossref] [PubMed]
8. World Health Organization. WHO handbook for guideline development. 2nd edition. Geneva: The Network, 2014.
9. World Health Organization. Health emergency interim guidelines: a WHO guideline development framework and toolkit. 1st edition. Geneva: The Network, 2017.
10. Garritty CM, Norris SL, Moher D. Developing WHO rapid advice guidelines in the setting of a public health emergency. J Clin Epidemiol 2017;82:47-60. [Crossref] [PubMed]
11. Wang X, Chen Y, Yao L, et al. Reporting of declarations and conflicts of interest in WHO guidelines can be further improved. J Clin Epidemiol 2018;98:1-8. [Crossref] [PubMed]
12. Norris SL, Sawin VI, Ferri M, et al. An evaluation of emergency guidelines issued by the World Health Organization in response to four infectious disease outbreaks. PLoS One 2018;13:e0198125. [Crossref] [PubMed]
13. APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION II[Internet]. The AGREE Collaboration; c2020 [cited 2020 Feb 5]. Available online: https://www.agreetrust.org/wp-content/uploads/2017/12/AGREE-II-Users-Manual-and-33-item-Instrument-2009-Update-2017.pdf
14. A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. The RIGHT Group; c2020 [cited 2020 Feb 5]. Available online: http://www.right-statement.org/right-statement/checklist
15. Chen Y, Yang K, Marusic A, et al. A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. Ann Intern Med 2017;166:128-32. [Crossref] [PubMed]
16. Yang K, Chen Y, Li Y, et al. Editorial: can China master the guideline challenge? Health Res Policy Syst 2013;11:1. [Crossref] [PubMed]