title: Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study.
abstract: BACKGROUND
  Assessing methodological quality of primary studies is an essential component of systematic reviews. Following a systematic review which used a domain based system [United States Preventative Services Task Force (USPSTF)] to assess methodological quality, a commonly used numerical rating scale (Downs and Black) was also used to evaluate the included studies and comparisons were made between quality ratings assigned using the two different methods. Both tools were used to assess the 20 randomized and quasi-randomized controlled trials examining an exercise intervention for chronic musculoskeletal pain which were included in the review. Inter-rater reliability and levels of agreement were determined using intraclass correlation coefficients (ICC). Influence of quality on pooled effect size was examined by calculating the between group standardized mean difference (SMD).
  
  
  RESULTS
  Inter-rater reliability indicated at least substantial levels of agreement for the USPSTF system (ICC 0.85; 95% CI 0.66, 0.94) and Downs and Black scale (ICC 0.94; 95% CI 0.84, 0.97). Overall level of agreement between tools (ICC 0.80; 95% CI 0.57, 0.92) was also good. However, the USPSTF system identified a number of studies (n = 3/20) as "poor" due to potential risks of bias. Analysis revealed substantially greater pooled effect sizes in these studies (SMD -2.51; 95% CI -4.21, -0.82) compared to those rated as "fair" (SMD -0.45; 95% CI -0.65, -0.25) or "good" (SMD -0.38; 95% CI -0.69, -0.08).
  
  
  CONCLUSIONS
  In this example, use of a numerical rating scale failed to identify studies at increased risk of bias, and could have potentially led to imprecise estimates of treatment effect. Although based on a small number of included studies within an existing systematic review, we found the domain based system provided a more structured framework by which qualitative decisions concerning overall quality could be made, and was useful for detecting potential sources of bias in the available evidence.
authors.affiliation: Centre for Public Health, Queen's University Belfast, Belfast, UK. s.oconnor@qub.ac.uk.|UKCRC Centre of Excellence for Public Health (Northern Ireland), Belfast, UK. s.oconnor@qub.ac.uk.|Centre for Health and Rehabilitation Technologies, Institute of Nursing and Health Research, School of Health Sciences, University of Ulster, Belfast, UK. s.oconnor@qub.ac.uk.
authors.forename: Seán R
authors.initials: SR
authors.lastname: O'Connor
authors.affiliation: Centre for Public Health, Queen's University Belfast, Belfast, UK. m.tully@qub.ac.uk.|UKCRC Centre of Excellence for Public Health (Northern Ireland), Belfast, UK. m.tully@qub.ac.uk.
authors.forename: Mark A
authors.initials: MA
authors.lastname: Tully
authors.affiliation: Centre for Health, Activity and Rehabilitation Research, University of Otago, Dunedin, New Zealand. brigid.ryan@otago.ac.nz.
authors.forename: Brigid
authors.initials: B
authors.lastname: Ryan
authors.affiliation: Centre for Health and Rehabilitation Technologies, Institute of Nursing and Health Research, School of Health Sciences, University of Ulster, Belfast, UK. jm.bradley@ulster.ac.uk.
authors.forename: Judy M
authors.initials: JM
authors.lastname: Bradley
authors.affiliation: Centre for Health, Activity and Rehabilitation Research, University of Otago, Dunedin, New Zealand. david.baxter@otago.ac.nz.
authors.forename: George D
authors.initials: GD
authors.lastname: Baxter
authors.affiliation: UKCRC Centre of Excellence for Public Health (Northern Ireland), Belfast, UK. s.mcdonough@ulster.ac.uk.|Centre for Health and Rehabilitation Technologies, Institute of Nursing and Health Research, School of Health Sciences, University of Ulster, Belfast, UK. s.mcdonough@ulster.ac.uk.
authors.forename: Suzanne M
authors.initials: SM
authors.lastname: McDonough
mesh_terms: D015982:Bias; D016009:Chi-Square Distribution; D059350:Chronic Pain / Q000175:diagnosis / Q000503:physiopathology / Q000628:therapy*; D003627:Data Interpretation, Statistical; D005081:Exercise Therapy; D006801:Humans; D015233:Models, Statistical; D059352:Musculoskeletal Pain / Q000175:diagnosis / Q000503:physiopathology / Q000628:therapy; D016017:Odds Ratio; D016032:Randomized Controlled Trials as Topic / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D012107:Research Design / Q000706:statistics & numerical data*; D016896:Treatment Outcome