title: Blinded versus unblinded assessments of risk of bias in studies included in a systematic review.
abstract: BACKGROUND
  The importance of appraising the risk of bias of studies included in systematic reviews is well-established. However, uncertainty remains surrounding the method by which risk of bias assessments should be conducted. Specifically, no summary of evidence exists as to whether blinded (i.e. the assessor is unaware of the study author's name, institution, sponsorship, journal, etc.) versus unblinded assessments of risk of bias yield systematically different assessments in a systematic review.
  
  
  OBJECTIVE
  To determine whether blinded versus unblinded assessments of risk of bias yield systematically different assessments in a systematic review.
  
  
  METHODS
  We searched MEDLINE (1966 to September week 4 2009), CINAHL (1982 to May week 3 2008), All EBM Reviews (inception to 6 October 2009), EMBASE (1980 to 2009 week 40) and HealthStar (1966 to September week 4 2009) (all Ovid interface). We applied no restrictions regarding language of publication, publication status or study design. We examined reference lists of included studies and contacted experts for potentially relevant literature.
  
  
  METHODS
  We included any study that examined blinded versus unblinded assessments of risk of bias included within a systematic review.
  
  
  METHODS
  We extracted information from each of the included studies using a pre-specified 16-item form. We summarized the level of agreement between blinded and unblinded assessments of risk of bias descriptively. We calculated the standardized mean difference whenever possible.
  
  
  RESULTS
  We included six randomized controlled trials (RCTs). Four studies had unclear risk of bias and two had high risk of bias. The results of these RCTs were not consistent; two demonstrated no differences between blinded and unblinded assessments, two found that blinded assessments had significantly lower quality scores, and another observed significantly higher quality scores for blinded assessments. The remaining study did not report the level of significance. We pooled five studies reporting sufficient information in a meta-analysis. We observed no statistically significant difference in risk of bias assessments between blinded or unblinded assessments (standardized mean difference -0.13, 95% confidence interval -0.42 to 0.16). The mean difference might be slightly inaccurate, as we did not adjust for clustering in our meta-analysis. We observed inconsistency of results visually and noted statistical heterogeneity.
  
  
  CONCLUSIONS
  Our review highlights that discordance exists between studies examining blinded versus unblinded risk of bias assessments at the systematic review level. The best approach to risk of bias assessment remains unclear, however, given the increased time and resources required to conceal reports effectively, it may not be necessary for risk of bias assessments to be conducted under blinded conditions in a systematic review.
authors: Health Canada, Ottawa, Ontario, Canada.
authors: Kate
authors: K
authors: Morissette
authors: Andrea C
authors: AC
authors: Tricco
authors: Tanya
authors: T
authors: Horsley
authors: Maggie H
authors: MH
authors: Chen
authors: David
authors: D
authors: Moher
mesh_terms: D017594:Publication Bias; D016032:Randomized Controlled Trials as Topic; D012196:Review Literature as Topic; D018570:Risk Assessment