title: Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study.
abstract: The Cochrane risk of bias tool is commonly criticized for having a low reliability. We aimed to investigate whether training of raters, with objective and standardized instructions on how to assess risk of bias, can improve the reliability of the Cochrane risk of bias tool.
  
  
  
  In this pilot study, four raters inexperienced in risk of bias assessment were randomly allocated to minimal or intensive standardized training for risk of bias assessment of randomized trials of physical therapy treatments for patients with knee osteoarthritis pain. Two raters were experienced risk of bias assessors who served as reference. The primary outcome of our study was between-group reliability, defined as the agreement of the risk of bias assessments of inexperienced raters with the reference assessments of experienced raters. Consensus-based assessments were used for this purpose. The secondary outcome was within-group reliability, defined as the agreement of assessments within pairs of inexperienced raters. We calculated the chance-corrected weighted Kappa to quantify agreement within and between groups of raters for each of the domains of the risk of bias tool.
  
  
  
  A total of 56 trials were included in our analysis. The Kappa for the agreement of inexperienced raters with reference across items of the risk of bias tool ranged from 0.10 to 0.81 for the minimal training group and from 0.41 to 0.90 for the standardized training group. The Kappa values for the agreement within pairs of inexperienced raters across the items of the risk of bias tool ranged from 0 to 0.38 for the minimal training group and from 0.93 to 1 for the standardized training group. Between-group differences in Kappa for the agreement of inexperienced raters with reference always favored the standardized training group and was most pronounced for incomplete outcome data (difference in Kappa 0.52, p < 0.001) and allocation concealment (difference in Kappa 0.30, p = 0.004).
  
  
  
  Intensive, standardized training on risk of bias assessment may significantly improve the reliability of the Cochrane risk of bias tool.
authors.affiliation: Institute of Primary Health Care (BIHAM), University of Bern, Gesellschaftsstrasse 49, Bern, 3012, Switzerland. bruno.dacosta@biham.unibe.ch.
authors.forename: Bruno R
authors.initials: BR
authors.lastname: da Costa
authors.affiliation: Department of Physical Therapy, Florida International University, AHC3-430 11200 8 St, Miami, USA.
authors.forename: Brooke
authors.initials: B
authors.lastname: Beckett
authors.affiliation: Department of Physical Therapy, Florida International University, AHC3-430 11200 8 St, Miami, USA.
authors.forename: Alison
authors.initials: A
authors.lastname: Diaz
authors.affiliation: Department of Physical Therapy, Florida International University, AHC3-430 11200 8 St, Miami, USA.
authors.forename: Nina M
authors.initials: NM
authors.lastname: Resta
authors.affiliation: Department of Anesthesia and Pain Medicine, and Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada.
authors.forename: Bradley C
authors.initials: BC
authors.lastname: Johnston
authors.affiliation: Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland.
authors.forename: Matthias
authors.initials: M
authors.lastname: Egger
authors.affiliation: Applied Health Research Centre (AHRC), Li Ka Shing Knowledge Institute of St. Michael's Hospital, Department of Medicine, University of Toronto, Toronto, Canada.
authors.forename: Peter
authors.initials: P
authors.lastname: Jüni
authors.affiliation: Faculty of Rehabilitation Medicine, Department of Physical Therapy, University of Alberta, Edmonton, AB, Canada.
authors.forename: Susan
authors.initials: S
authors.lastname: Armijo-Olivo
mesh_terms: D015982:Bias; D006801:Humans; D010865:Pilot Projects; D011446:Prospective Studies; D016032:Randomized Controlled Trials as Topic / Q000592:standards*; D015203:Reproducibility of Results; D012196:Review Literature as Topic; D012307:Risk Factors; D013223:Statistics as Topic / Q000193:education*