title: Interrater reliability of grading strength of evidence varies with the complexity of the evidence in systematic reviews.
abstract: OBJECTIVE
  To examine consistency (interrater reliability) of applying guidance for grading strength of evidence in systematic reviews for the Agency for Healthcare Research and Quality Evidence-based Practice Center program.
  
  
  METHODS
  Using data from two systematic reviews, authors tested the main components of the approach: (1) scoring evidence on the four required domains (risk of bias, consistency, directness, and precision) separately for randomized controlled trials (RCTs) and observational studies and (2) developing an overall strength of evidence grade, given the scores for each of these domains.
  
  
  RESULTS
  Conclusions about overall strength of evidence reached by experienced systematic reviewers based on the same evidence can differ greatly, especially for complex bodies of evidence. Current instructions may be sufficient for straightforward quantitative evaluations that use meta-analysis for summarizing RCT findings. In contrast, agreement suffered when evaluations did not lend themselves to meta-analysis and reviewers needed to rely on their own qualitative judgment. Three areas raised particular concern: (1) evidence from a combination of RCTs and observational studies, (2) outcomes with differing measurement, and (3) evidence that appeared to show no differences in outcomes.
  
  
  CONCLUSIONS
  Interrater reliability was highly variable for scoring strength of evidence domains and combining scores to reach overall strength of evidence grades. Future research can help in establishing improved methods for evaluating these complex bodies of evidence.
authors.affiliation: Division of Social Policy, Health, and Economics Research, RTI International Research Triangle Institute, Research Triangle Park, NC 27709-2194, USA. berkman@rti.org
authors.forename: Nancy D
authors.initials: ND
authors.lastname: Berkman
authors.forename: Kathleen N
authors.initials: KN
authors.lastname: Lohr
authors.forename: Laura C
authors.initials: LC
authors.lastname: Morgan
authors.forename: Tzy-Mey
authors.initials: TM
authors.lastname: Kuo
authors.forename: Sally C
authors.initials: SC
authors.lastname: Morton
mesh_terms: D006801:Humans; D015201:Meta-Analysis as Topic; D064887:Observational Studies as Topic; D015588:Observer Variation; D016032:Randomized Controlled Trials as Topic; D015203:Reproducibility of Results; D012107:Research Design; D012196:Review Literature as Topic
keywords: Agency for Healthcare Research and Quality; Comparative effectiveness; Evidence-based practice; Interrater reliability; Strength of evidence; Systematic review methodology