title: Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non-randomised study designs applied in a health technology assessment setting.
abstract: We describe our experience of using a modified version of the Cochrane risk of bias (RoB) tool for randomised and non-randomised comparative studies.
  
  
  OBJECTIVE
  To assess time to complete RoB assessment. To assess inter-rater agreement. To explore the association between RoB and treatment effect size
  
  
  METHODS
  Cochrane risk of bias assessment was performed on a sample of full text primary reports included in a systematic review comparing operative techniques for radical prostatectomy. Inter-rater agreement was assessed using the kappa statistic.
  
  
  RESULTS
  Twenty-four studies were judged as high overall RoB, 13 were judged as low RoB and 11 were unclear. The weighted Kappa value was 0.35 indicating fair agreement. The median (range) time taken to rate each study was 30 min (10-49). The effect estimate for all studies was 0.61 (95% credible interval (CrI) 0.46-0.83) and 0.73 (95% CrI 0.29-1.75) for low risk studies.
  
  
  CONCLUSIONS
  Although the process was time consuming, using a modified version of the RoB tool proved useful for demonstrating conservative effect estimates. That we only achieved a fair agreement between reviewers demonstrates the urgent need for further validation to improve inter-rater agreement. We suggest additional RoB levels could improve inter-rater reliability.
authors.affiliation: Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
authors.forename: Clare
authors.initials: C
authors.lastname: Robertson
authors.affiliation: Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
authors.forename: Craig
authors.initials: C
authors.lastname: Ramsay
authors.affiliation: Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
authors.forename: Tara
authors.initials: T
authors.lastname: Gurung
authors.affiliation: Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
authors.forename: Graham
authors.initials: G
authors.lastname: Mowatt
authors.affiliation: Institute of Cellular Medicine, Newcastle University, Newcastle upon Tyne, UK.
authors.forename: Robert
authors.initials: R
authors.lastname: Pickard
authors.affiliation: Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
authors.forename: Pawana
authors.initials: P
authors.lastname: Sharma
mesh_terms: D015982:Bias; D015588:Observer Variation; D017063:Outcome Assessment, Health Care / Q000379:methods*; D016032:Randomized Controlled Trials as Topic; D015203:Reproducibility of Results; D012107:Research Design; D012196:Review Literature as Topic; D018570:Risk Assessment / Q000379:methods; D012680:Sensitivity and Specificity; D012984:Software; D013673:Technology Assessment, Biomedical / Q000379:methods*
keywords: Cochrane Collaboration; non-randomised; risk of bias; systematic review