title: Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers.
abstract: OBJECTIVE
  To assess inter-rater reliability and validity of the Newcastle Ottawa Scale (NOS) used for methodological quality assessment of cohort studies included in systematic reviews.
  
  
  METHODS
  Two reviewers independently applied the NOS to 131 cohort studies included in eight meta-analyses. Inter-rater reliability was calculated using kappa (κ) statistics. To assess validity, within each meta-analysis, we generated a ratio of pooled estimates for each quality domain. Using a random-effects model, the ratios of odds ratios for each meta-analysis were combined to give an overall estimate of differences in effect estimates.
  
  
  RESULTS
  Inter-rater reliability varied from substantial for length of follow-up (κ = 0.68, 95% confidence interval [CI] = 0.47, 0.89) to poor for selection of the nonexposed cohort and demonstration that the outcome was not present at the outset of the study (κ = -0.03, 95% CI = -0.06, 0.00; κ = -0.06, 95% CI = -0.20, 0.07). Reliability for overall score was fair (κ = 0.29, 95% CI = 0.10, 0.47). In general, reviewers found the tool difficult to use and the decision rules vague even with additional information provided as part of this study. We found no association between individual items or overall score and effect estimates.
  
  
  CONCLUSIONS
  Variable agreement and lack of evidence that the NOS can identify studies with biased results underscore the need for revisions and more detailed guidance for systematic reviewers using the NOS.
authors.affiliation: Department of Pediatrics, Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, University of Alberta, 4-472 Edmonton Clinic Health Academy, 11405-87 Avenue, Edmonton, Alberta, Canada T5G 1C9. hartling@ualberta.ca
authors.forename: Lisa
authors.initials: L
authors.lastname: Hartling
authors.forename: Andrea
authors.initials: A
authors.lastname: Milne
authors.forename: Michele P
authors.initials: MP
authors.lastname: Hamm
authors.forename: Ben
authors.initials: B
authors.lastname: Vandermeer
authors.forename: Mohammed
authors.initials: M
authors.lastname: Ansari
authors.forename: Alexander
authors.initials: A
authors.lastname: Tsertsvadze
authors.forename: Donna M
authors.initials: DM
authors.lastname: Dryden
mesh_terms: D015331:Cohort Studies; D006801:Humans; D015201:Meta-Analysis as Topic; D015588:Observer Variation; D015203:Reproducibility of Results; D012107:Research Design; D012196:Review Literature as Topic
keywords: Cohort studies; Internal validity; Methodological quality; Reliability; Systematic reviews; Validity