Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions.
Overviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. It is recommended that authors assess and report the methodological quality of SRs in overviews-for example, using A MeaSurement Tool to Assess systematic Reviews (AMSTAR). Currently, there is variation in whether and how overview authors assess and report SR quality, and limited guidance is available. Our objectives were to: examine methodological considerations involved in using AMSTAR to assess the quality of Cochrane and non-Cochrane SRs in overviews of healthcare interventions; identify challenges (and develop potential decision rules) when using AMSTAR in overviews; and examine the potential impact of considering methodological quality when making inclusion decisions in overviews.



We selected seven overviews of healthcare interventions and included all SRs meeting each overview's inclusion criteria. For each SR, two reviewers independently conducted AMSTAR assessments with consensus and discussed challenges encountered. We also examined the correlation between AMSTAR assessments and SR results/conclusions.



Ninety-five SRs were included (30 Cochrane, 65 non-Cochrane). Mean AMSTAR assessments (9.6/11 vs. 5.5/11; p < 0.001) and inter-rater reliability (AC1 statistic: 0.84 vs. 0.69; "almost perfect" vs. "substantial" using the Landis & Koch criteria) were higher for Cochrane compared to non-Cochrane SRs. Four challenges were identified when applying AMSTAR in overviews: the scope of the SRs and overviews often differed; SRs examining similar topics sometimes made different methodological decisions; reporting of non-Cochrane SRs was sometimes poor; and some non-Cochrane SRs included other SRs as well as primary studies. Decision rules were developed to address each challenge. We found no evidence that AMSTAR assessments were correlated with SR results/conclusions.



Results indicate that the AMSTAR tool can be used successfully in overviews that include Cochrane and non-Cochrane SRs, though decision rules may be useful to circumvent common challenges. Findings support existing recommendations that quality assessments of SRs in overviews be conducted independently, in duplicate, with a process for consensus. Results also suggest that using methodological quality to guide inclusion decisions (e.g., to exclude poorly conducted and reported SRs) may not introduce bias into the overview process.
Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Canada.
Michelle
M
Pollock
Clinical Pharmacology Unit, Instituto de Medicina Molecular, University of Lisbon, Lisbon, Portugal.|Department of Pediatrics, Santa Maria Hospital, Lisbon, Portugal.
Ricardo M
RM
Fernandes
Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Canada. hartling@ualberta.ca.|, 4-472 Edmonton Clinic Health Academy, 11405 87 Avenue NW, Edmonton, AB, T6G-1C9, Canada. hartling@ualberta.ca.
Lisa
0000-0001-8341-3991
L
Hartling
D003657:Decision Making; D019317:Evidence-Based Medicine; D006801:Humans; D011787:Quality of Health Care; D015203:Reproducibility of Results; D016896:Treatment Outcome
Bias; Knowledge synthesis; Methodological quality; Observer agreement; Overview of reviews; Systematic review