Use of AMSTAR-2 in the methodological assessment of systematic reviews: protocol for a methodological study
Study Protocol

Use of AMSTAR-2 in the methodological assessment of systematic reviews: protocol for a methodological study

Cuncun Lu1,2, Tingting Lu1,2, Long Ge1,2, Nan Yang1,2, Peijing Yan3, Kehu Yang1,2

1Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China;2Evidence-Based Social Science Center, School of Public Health, Lanzhou University, Lanzhou 730000, China;3Institute of Clinical Research and Evidence Based Medicine, Gansu Provincial Hospital, Lanzhou 730000, China

Contributions: (I) Conception and design: C Lu, T Lu, K Yang; (II) Administrative support: None; (III) Provision of study materials or patients: C Lu, T Lu; (IV) Collection and assembly of data: T Lu, L Ge; (V) Data analysis and interpretation: C Lu, N Yang, P Yan; (VI) Manuscript writing: All authors; (VII) Final approval of manuscript: All authors.

Correspondence to: Kehu Yang. Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China. Email: kehuyangebm2006@126.com.

Background: Systematic reviews (SRs) with or without meta-analyses (MAs) are widely used in resolving questions in various healthcare areas (such as, traditional Chinese medicine, public health and surgery), and they are the cornerstone of evidence-based healthcare. However, the reliability of SRs is typically influenced by their methodological quality. AMSTAR (A Measurement Tool to Assess Systematic Reviews) and AMSTAR-2 tools can assess the methodological quality of SRs, and the use of AMSTAR has been investigated. However, AMSTAR-2 is now widely used to evaluate the methodological quality of SRs, but the use of AMSTAR-2 for determining the methodological quality of SRs has not yet been investigated and assessed thoroughly. Thus, we designed the present study to investigate the use of AMSTAR-2 in studies that assessed the methodological quality of a sample of SRs with the AMSTAR-2 and provide references to potential users of AMSTAR-2 tool.

Methods: Four commonly used electronic databases including PubMed, EmBase, the Cochrane Library, and Web of Science will be searched following a comprehensive search strategy to identify and retrieve studies that have used AMSTAR-2 tool for evaluating the methodological quality of SRs. Two independent authors will retrieve bibliometric information and methodological data, including all author names, time of publication, and journal names, whether a specific score value was given for each item, and whether overall quality assessment was performed. Descriptive statistical analyses will be used to present the study results, e.g., frequencies and percentages, mean and standard deviation (SD) or median and interquartile range (IQR). In addition, subgroup analyses will be performed to identify the methodological differences (e.g., the reporting of study designs included in SRs) between overviews and methodological studies. The risk ratio (RR) with 95% confidence interval (95% CI) will be calculated to measure the methodological differences. Cytoscape 3.7.1 software tool will be used to construct collaboration network maps. Further, Microsoft Office Excel 2016 and Stata 12.0 will be used to manage and analyze data.

Discussion: The results of this study will identify any gaps in the use of AMSTAR-2 and important bibliometric features, such as active researchers and journals, provide guidance to researchers in various healthcare areas (such as, traditional Chinese medicine and public health) for using AMSTAR-2 tool and help them in developing cooperation and submitting their manuscripts.

Keywords: Systematic reviews (SRs); quality; AMSTAR-2; healthcare; traditional Chinese medicine; public health; surgery

Submitted Jan 07, 2020. Accepted for publication Mar 11, 2020.

doi: 10.21037/atm-20-392a

Introduction

According to the National Academy of Medicine, a systematic review (SR) is “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies” (1). The meta-analysis (MA) is a statistic method that aims to quantitatively pool individual results from homogeneous primary studies, and SRs are often but not always accompanied with MAs (2). At present, SRs mainly include qualitative SRs, traditional paired MAs, MAs of individual patient data, and network MAs, among others (3-5). SR and MA had been widely used in resolving questions in various healthcare domains, such as public health, surgery and traditional Chinese medicine (3,6,7). These SRs can provide evidence to support decision-making, and are the cornerstone of evidence-based clinical practice and evidence-informed policymaking. However, it is noteworthy that not all SRs are reliable and valuable, and only well-designed, strictly performed, unbiased SRs are excellent evidence resources that can provide scientific evidence to support evidence-based healthcare (1,8).

Studies have shown that quality of SRs can be influenced by many factors (9-11), e.g., quality of included primary studies, preregistration, predesigned protocol, completeness and clarity of reporting, and control of potential bias. Fortunately, the reporting and methodological quality of SRs can be assessed using corresponding checklists or scale tools, such as PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) (12) and its extensions (13,14), ROBIS (Risk Of Bias In Systematic reviews) (15), AMSTAR (A Measurement Tool to Assess Systematic Reviews) (16) and AMSTAR-2 (17). Thus, all potential targeted users in healthcare domains, such as peer-reviewers, editors, clinicians, and health decision-makers, can judge the reliability of existing SRs using the abovementioned tools. Moreover, authors can refer to these tools to improve the quality of their manuscripts; further, evidence has revealed that the quality of publications has been improved with the use of these relevant tools (18-20).

In particular, AMSTAR and AMSTAR-2 tools focus on the methodological quality of SRs, and their reliability and validity have been verified (21,22). AMSTAR was originally developed for SRs of RCTs for interventions and was introduced in 2007 (16). The tool consists of 11 items, involving prospective design, study selection and information abstraction, study search, grey literature, list of primary studies, study characteristics, quality assessment, combination of study results, publication bias, and conflicts of interest. The answers for these 11 items include “Yes”, “No”, “Cannot Answer”, and “Not Applicable”. AMSTAR was widely used to assess the methodological quality of SRs, but was not only limited to SRs of randomized controlled trials (RCTs) for interventions, such as SRs of non-randomized studies (23). On the use of AMSTAR, some limitations of this tool were identified; therefore, a revised tool, AMSTAR-2 was introduced in 2017 (17). According to the original report (17), AMSTAR-2 was developed to assess SRs of randomized and/or non-randomized interventional studies, but was not intended for use in assessing other types of SRs, such as MAs of diagnostic test accuracy, MAs of individual patient data, network MAs, scoping reviews. On comparing with AMSTAR, which comprises 11 items, AMSTAR-2 comprises 16 items, and seven of them are critical items, including prior protocol, comprehensive literature search, justification of excluding studies, assessment of RoB for individual studies, appropriate meta-analytic methods, consideration of RoB in results and impact of publication bias. One of following four options could be used to answer the questions of items: “Yes”, “Partial Yes”, “No” or “No meta-analysis conducted”. Furthermore, based on critical items, the overall confidence in the results of SRs can be divided into four levels: high, moderate, low, and critically low. After getting published, the AMSTAR-2 tool has been widely employed to evaluate the methodological quality of SRs (6,24-26), for example, Luo et al. (25) assessed the quality of SRs of acupuncture for women with polycystic ovarian syndrome using AMSTAR-2 and PRISMA (Preferred Reporting Items for Systematic review and Meta-Analyses), and authors concluded that identified SRs had the poor methodological and reporting quality.

The correct use and complete reporting of assessment tools are critically important (21,27). For example, whether a specific score value should be given for each item and whether the overall quality assessment of SRs should be based on a total score are always topics of debate (21). Pieper et al. (21) identified 247 publications and investigated the use of AMSTAR in them; the study found that the use of AMSTAR varied greatly, and outreached the original purpose defined by the introduced paper; the study suggested that stakeholders including authors, peer-reviewers, and editors should pay more attention to the correct use and complete reporting of AMSTAR. However, although AMSTAR-2 is now widely used in overviews or methodological studies for assessing the methodological quality of SRs, the use of AMSTAR-2 in the methodological assessment of SRs has not been thoroughly investigated so far. Furthermore, as it is a relatively new tool, early detection and correction of problems while using the tool are particularly important. In addition, bibliometric analysis is an important method that can find the key features of a set of studies, such as, core researchers and their collaboration network (also named as social network), and journals, can provide guidance for developing cooperation and submitting manuscripts for researchers (26,28). Therefore, by combining the bibliometric methods, the present methodological study was designed to investigate the use of AMSTAR-2 in studies that assessed the methodological quality of a sample of SRs with the AMSTAR-2 and provide references to potential users of AMSTAR-2 tool.

Objectives

(I) to analyze bibliometric characteristics (such as authors and journals) of studies that assessed the methodological quality of a sample of SRs with the AMSTAR-2; (II) to investigate how AMSTAR-2 is used in studies that employ AMSTAR-2 for assessing the methodological quality of SRs; and (III) to analyze whether the use of AMSTAR-2 goes beyond the original purpose defined by the original study.

Methods

The reporting of this protocol was according to the PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-analysis Protocols) guideline (see Supplement I) (29), although some items were different from the original tool. Because the present study is a methodological study rather than a SR, we did not register it in an open public website.

Eligibility criteria

The study aims to investigate the use of AMSTAR-2 in the methodological assessment of SRs; therefore, we will include published papers that have used AMSTAR-2 to assess the methodological quality of SRs. Based on our previous experience, we will include two large groups publications, the first group will include overviews, also known as SRs of SRs (5,21), where AMSTAR-2 is often applied to assess the methodological quality of included SRs, and the other group will include methodological studies focusing the methodological quality of SRs for a given topic, such as surgery (6), or a type of SRs (26). In this study, psychometric studies will be classified as methodological studies (22,30). In other words, all publications in English will be considered if they have used AMSTAR-2 to assess the methodological quality of SRs. We will exclude conference abstracts, letters, comments, protocols and other paper documents in which we cannot retrieve the data that we need.

Search strategy

To identify as many eligible publications as possible, four commonly used electronic online databases including PubMed, EmBase, the Cochrane Library, and Web of Science, will be searched systematically using a comprehensive search strategy. An initial search was conducted on October 14, 2019. The initial search yielded 186 records. Further, we will also check the lists of references of these included studies to supplement potential eligible studies. We will then update the search will by scanning these databases again after the completion of data extraction. We have presented the search strategy using PubMed as an example in Table 1. The search strategy will be adapted to fit other online databases as well. There will be no other limitations for search methods.

Table 1
Table 1 Search strategy of PubMed database
Full table

Study selection

The selection of studies will comprise two stages. First, all hits from these four online databases will be downloaded and imported into Endnote X9 software tool (Thomson Reuters, New York, NY, USA) for deduplication. Second, full texts of studies will be retrieved from journal websites or electronic databases after deduplication to assess their eligibility according to the abovementioned inclusion criteria. These two stages will be accomplished by two independent authors, and any disagreement will be solved by a discussion or by consulting a third author.

Data extraction

Two authors will extract relevant information from included studies using a predesigned data form. The data form will be piloted and improved using 5–10 included studies for convenience. The following information will be retrieved from each included study: (I) bibliometric data: all author names, author’s institutions, author’s countries, publication time, journal names, journal impact factor (IF), clinical topics (such as, surgery or acupuncture or psychology), and references of AMSTAR-2 (original paper or not); (II) methodological information: study types included in the present study (overview or methodological study), searched databases and included SR types (SR without MA, MA, network MAs, etc.) in overviews or methodological studies, study designs included in SRs (only RCT, non-randomized studies, or both), rationales of using AMSTAR-2, versions of AMSTAR-2 (original or modified), methods of assessment (independently assessed, or assessed by one group and the results checked by the other group), number of reviewers using AMSTAR-2, inter-rater reliability (IRR), whether a specific score value was given for each item (e.g., “Yes” =1 point, “Partial Yes” =0.5 point, or “No” =0 point), whether and how the overall assessment was conducted, whether comments were provided for assessment, whether the tool was used in conjunction with other tools [such as, PRISMA or GRADE (Grading of Recommendations Assessment, Development and Evaluation) (31), whether the results or conclusions of SRs were reported, whether limitations were considered while scoring, and whether limitations were commented on with regard to the use of AMSTAR-2 (e.g., to assess the quality of network MAs). In this study, all data will be based on published documents; therefore, authors of included studies will not be contacted to acquire any additional information on the included studies.

Data analysis

Descriptive statistical analyses will be used in this study. For categorical data, frequencies and percentages will be used, and for continuous data, mean and standard deviation (SD), or median and interquartile range (IQR) will be presented according to the included data. Because overviews and methodological studies typically have different purposes (21), subgroup analyses will be performed to identify the methodological differences between overviews and methodological studies. The risk ratio (RR) with 95% confidence interval (95% CI) will be calculated and used to measure the methodological differences, e.g., whether the reporting of study designs included in SRs in overviews was different from that included in SRs in methodological studies. Microsoft Office Excel 2016 (Microsoft Corporation, Redmond, WA, USA) will be used to manage and analyze data. Cytoscape 3.7.1 will be used for visualization of social-network relationship (32). Stata 12.0 (Stata Corporation, CollegeStation, Texas, USA) will be used to calculate RR values and present forest plots.

Discussion

To the best of our knowledge, this is the first study to investigate the use of AMSTAR-2 tool for assessing the methodological quality of SRs. The results of this study will identify any gaps in the use of AMSTAR-2 and important bibliometric features, such as active researchers and journals, provide guidance to researchers in various healthcare domains (such as, traditional Chinese medicine, public health and surgery) for using AMSTAR-2 tool to improve their manuscripts’ quality and help them in developing cooperation and submitting their manuscripts. This methodological study will only include publications in English language; therefore, the results of the study may be biased.

Presenting and reporting of results

The results of this study will be reported according to the PRISMA guideline (25). A flow chart will be used to present the process of the screening and selection of the eligible studies.

Supplementary

Supplement IPRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist: recommended items to address in a systematic review protocol*

Acknowledgments

Funding: None.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/atm-20-392a). The authors have no conflicts of interest to declare.

Ethical Statement: All authors are accountable for all aspects of this work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The present study will not involve any patients and/or the public. No ethical approve or informed consent is required for the purposes of the present study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

  1. Le JT, Qureshi R, Twose C, et al. Evaluation of Systematic Reviews of Interventions for Retina and Vitreous Conditions. JAMA Ophthalmol 2019;137:1399-405. [Crossref] [PubMed]
  2. Murad MH, Montori VM, Ioannidis JP, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. JAMA 2014;312:171-9. [Crossref] [PubMed]
  3. Li X, Wei L, Shang W, et al. Trace and evaluation systems for health services quality in rural and remote areas: a systematic review. J Public Health 2018;26:127. [Crossref]
  4. Ioannidis J. Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews. Br J Sports Med 2017;51:1456-58. [Crossref] [PubMed]
  5. Lunny C, Brennan SE, McDonald S, et al. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev 2017;6:231. [Crossref] [PubMed]
  6. Yan P, Yao L, Li H, et al. The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study. J Clin Epidemiol 2019;109:20-9. [Crossref] [PubMed]
  7. Lu C. Comment on: “Ginger for health care – An overview of systematic reviews Complement Ther Med 2019. [Crossref]
  8. Goldkuhle M, Narayan VM, Weigl A, et al. A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer. BMJ Open 2018;8:e020869. [Crossref] [PubMed]
  9. Hopewell S, Boutron I, Altman DG, et al. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ open 2013;3:e003342. [Crossref] [PubMed]
  10. Ge L, Tian JH, Li YN, et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J Clin Epidemiol 2018;93:45-55. [Crossref] [PubMed]
  11. Jørgensen AW, Hilden J, Gotzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 2006;333:782. [Crossref] [PubMed]
  12. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700. [Crossref] [PubMed]
  13. Hutton B, Salanti G, Caldwell DM, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med 2015;162:777-84. [Crossref] [PubMed]
  14. McInnes MDF, Moher D, Thombs BD, et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement. JAMA 2018;319:388-96. [Crossref] [PubMed]
  15. Whiting P, Savovic J, Higgins JP, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225e34.
  16. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. [Crossref] [PubMed]
  17. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008. [Crossref] [PubMed]
  18. Sharma S, Oremus M. PRISMA and AMSTAR show systematic reviews on health literacy and cancer screening are of good quality. J Clin Epidemiol 2018;99:123-31. [Crossref] [PubMed]
  19. Sun X, Zhou X, Yu Y, et al. Exploring reporting quality of systematic reviews and Meta-analyses on nursing interventions in patients with Alzheimer's disease before and after PRISMA introduction. BMC Med Res Methodol 2018;18:154. [Crossref] [PubMed]
  20. Vásquez-Cárdenas J, Zapata-Norena O, Carvajal-Florez A, et al. Systematic reviews in orthodontics: Impact of the PRISMA for Abstracts checklist on completeness of reporting. Am J Orthod Dentofacial Orthop 2019;156:442-52.e12. [Crossref] [PubMed]
  21. Pieper D, Koensgen N, Breuing J, et al. How is AMSTAR applied by authors - a call for better reporting. BMC Med Res Methodol 2018;18:56. [Crossref] [PubMed]
  22. Lorenz RC, Matthias K, Pieper D, et al. A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. J Clin Epidemiol 2019;114:133-40. [Crossref] [PubMed]
  23. Pieper D, Mathes T, Eikermann M. Can AMSTAR also be applied to systematic reviews of non-randomized studies? BMC Res Notes 2014;7:609. [Crossref] [PubMed]
  24. Cortese S, Tomlinson A, Cipriani A. Meta-Review: Network Meta-Analyses in Child and Adolescent Psychiatry. J Am Acad Child Adolesc Psychiatry 2019;58:167-79. [Crossref] [PubMed]
  25. Luo YN, Zheng QH, Liu ZB, et al. Methodological and reporting quality evaluation of systematic reviews on acupuncture in women with polycystic ovarian syndrome: A systematic review. Complement Ther Clin Pract. 2018;33:197-203. [Crossref] [PubMed]
  26. Gao Y, Ge L, Ma X, et al. Improvement needed in the network geometry and inconsistency of Cochrane network meta-analyses: a cross-sectional survey. J Clin Epidemiol 2019;113:214-27. [Crossref] [PubMed]
  27. Johnston A, Kelly SE, Hsieh SC, et al. Systematic reviews of clinical practice guidelines: a methodological guide. J Clin Epidemiol 2019;108:64-76. [Crossref] [PubMed]
  28. Lu C, Bing Z, Bi Z, et al. Top-100 Most Cited Publications Concerning Network Pharmacology: A Bibliometric Analysis. Evid Based Complement Alternat Med 2019;2019:1704816. [Crossref] [PubMed]
  29. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;350:g7647. [Crossref] [PubMed]
  30. Pieper D, Puljak L, Gonzalez-Lorenzo M, et al. Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. J Clin Epidemiol 2019;108:26-33. [Crossref] [PubMed]
  31. Guyatt GH, Oxman AD, Kunz R, et al. Going from evidence to recommendations. BMJ 2008;336:1049-51. [Crossref] [PubMed]
  32. Shannon P, Markiel A, Ozier O, et al. Cytoscape: a software environment for integrated models of biomolecular interaction networks. Genome Res 2003;13:2498-504. [Crossref] [PubMed]
Cite this article as: Lu C, Lu T, Ge L, Yang N, Yan P, Yang K. Use of AMSTAR-2 in the methodological assessment of systematic reviews: protocol for a methodological study. Ann Transl Med 2020;8(10):652. doi: 10.21037/atm-20-392a

Article Options

Download Citation

Share