External validation of a measurement tool to assess systematic reviews (AMSTAR).
BACKGROUND
Thousands of systematic reviews have been conducted in all areas of health care. However, the methodological quality of these reviews is variable and should routinely be appraised. AMSTAR is a measurement tool to assess systematic reviews.


METHODS
AMSTAR was used to appraise 42 reviews focusing on therapies to treat gastro-esophageal reflux disease, peptic ulcer disease, and other acid-related diseases. Two assessors applied the AMSTAR to each review. Two other assessors, plus a clinician and/or methodologist applied a global assessment to each review independently.


CONCLUSIONS
The sample of 42 reviews covered a wide range of methodological quality. The overall scores on AMSTAR ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6) and median 4.0 (range 2.0 to 6.0). The inter-observer agreement of the individual items ranged from moderate to almost perfect agreement. Nine items scored a kappa of >0.75 (95% CI: 0.55 to 0.96). The reliability of the total AMSTAR score was excellent: kappa 0.84 (95% CI: 0.67 to 1.00) and Pearson's R 0.96 (95% CI: 0.92 to 0.98). The overall scores for the global assessment ranged from 2 to 7 (out of a maximum score of 7) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.25 to 5.75). The agreement was lower with a kappa of 0.63 (95% CI: 0.40 to 0.88). Construct validity was shown by AMSTAR convergence with the results of the global assessment: Pearson's R 0.72 (95% CI: 0.53 to 0.84). For the AMSTAR total score, the limits of agreement were -0.19+/-1.38. This translates to a minimum detectable difference between reviews of 0.64 'AMSTAR points'. Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews.
Shea
Beverley J
BJ
Community Information and Epidemiological Technologies, Ottawa, Ontario, Canada. bshea@ciet.org
Bouter
Lex M
LM
Peterson
Joan
J
Boers
Maarten
M
Andersson
Neil
N
Ortiz
Zulma
Z
Ramsay
Tim
T
Bai
Annie
A
Shukla
Vijay K
VK
Grimshaw
Jeremy M
JM
D005744:Gastric Acid; D005764:Gastroesophageal Reflux / Q000628:therapy; D006801:Humans; D015201:Meta-Analysis as Topic; D010437:Peptic Ulcer / Q000628:therapy; D015203:Reproducibility of Results