title: Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs.
abstract: OBJECTIVE
  To assess the reliability of the Cochrane Risk of Bias (ROB) tool between individual raters and across consensus agreements of pairs of reviewers and examine the impact of study-level factors on reliability.
  
  
  METHODS
  Two reviewers assessed risk of bias for 154 randomized controlled trials (RCTs). For 30 RCTs, two reviewers from each of four centers assessed risk of bias and reached consensus. We assessed interrater agreement using kappas and the impact of study-level factors through subgroup analyses.
  
  
  RESULTS
  Reliability between two reviewers was fair for most domains (κ=0.24-0.37), except sequence generation (κ=0.79, substantial). Reliability results across reviewer pairs: sequence generation, moderate (κ=0.60); allocation concealment and "other sources of bias," fair (κ=0.37-0.27); and other domains, slight (κ=0.05-0.09). Reliability was influenced by the nature of the outcome, nature of the intervention, study design, trial hypothesis, and funding source. Variability resulted from different interpretation of the tool rather than different information identified in the study reports.
  
  
  CONCLUSIONS
  Low agreement has implications for interpreting systematic reviews. These findings suggest the need for detailed guidance in assessing the risk of bias.
authors.affiliation: Department of Pediatrics, Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, University of Alberta, 4-472 Edmonton Clinic Health Academy, 11405-87 Avenue, Edmonton, Alberta, Canada. hartling@ualberta.ca
authors.forename: Lisa
authors.initials: L
authors.lastname: Hartling
authors.forename: Michele P
authors.initials: MP
authors.lastname: Hamm
authors.forename: Andrea
authors.initials: A
authors.lastname: Milne
authors.forename: Ben
authors.initials: B
authors.lastname: Vandermeer
authors.forename: P Lina
authors.initials: PL
authors.lastname: Santaguida
authors.forename: Mohammed
authors.initials: M
authors.lastname: Ansari
authors.forename: Alexander
authors.initials: A
authors.lastname: Tsertsvadze
authors.forename: Susanne
authors.initials: S
authors.lastname: Hempel
authors.forename: Paul
authors.initials: P
authors.lastname: Shekelle
authors.forename: Donna M
authors.initials: DM
authors.lastname: Dryden
mesh_terms: D015982:Bias; D032921:Consensus; D006801:Humans; D015588:Observer Variation; D016032:Randomized Controlled Trials as Topic; D015203:Reproducibility of Results; D012107:Research Design; D012196:Review Literature as Topic; D018570:Risk Assessment