title: Concerns for the reliability and validity of the National Stroke Project Stroke Severity Scale.
abstract: BACKGROUND
  The National Stroke Project (NSP) was a retrospective cohort study of US Medicare beneficiaries hospitalized with stroke or transient ischemic attack (TIA). The NSP included a simple assessment of stroke severity (NSP-Stroke Scale, NSP-SS). Used for risk adjustment in outcome studies, the reliability and validity of the NSP-SS have not been assessed. We determined the reliability, concurrent and construct validity of the NSP-SS.
  
  
  METHODS
  The initial neurologic examinations of 100 consecutive patients hospitalized with ischemic stroke/TIA in a single academic medical center were reviewed. The NSP-SS was retrospectively scored twice by the same rater and independently by a second rater to assess reliability. The National Institutes of Health Stroke Scale (NIH-SS) was also scored retrospectively and used as the criterion standard for concurrent validity. Construct validity was based on discharge status.
  
  
  RESULTS
  The NSP-SS had moderate-substantial inter-rater (weighted kappa, κ(w) = 0.66, 95% CI 0.55-0.77) and intra-rater (κ(w) = 0.63, 95% CI 0.52-0.75) reliability. Correlation between NSP-SS and NIH-SS scores was moderate (Spearman r = 0.65, 95% CI 0.52-0.75, p < 0.0001) but some categorizations in the NSP-SS seemed inappropriate reflecting poor content validity. Each NSP-SS point was associated with a greater likelihood of poor outcome (OR = 2.1, 95% CI 1.1-3.7, p = 0.016). Based on dichotomized scores (NSP 0-2 and NIH-SS <6; mild deficits), the NSP-SS sensitivity was 70.9% (95% CI 57.9-81.2%), specificity 82.2% (95% CI 68.7-90.7%), likelihood ratio for severe stroke 4.0 (95% CI 2.1-7.6) and likelihood ratio for mild stroke 0.3 (95% CI 0.20-0.5). The dichotomized NSP-SS and NIH-SS similarly predicted poor outcome (NSP-SS >2, OR = 4.7, 95% CI 1.7-13.0, p = 0.003 vs. NIH-SS ≥6, OR = 4.4, 95% CI 1.5-13.0, p = 0.006) with excellent discrimination (C = 0.827 and 0.826, respectively).
  
  
  CONCLUSIONS
  The NSP-SS has moderate-substantial reliability but poor content validity and poor to moderate concurrent validity as compared with the NIH-SS. In addition, it is not clear that the NSP-SS is easier to extract from medical records than the NIH-SS. Given this, and its other limitations, the utility of this scale for risk adjustment in future stroke outcome studies is questionable.
authors: Department of Medicine (Neurology), Duke Stroke Center, Duke University Medical Center, Durham, NC 27710, USA.
authors: Nada
authors: N
authors: El Husseini
authors: Kenneth J
authors: KJ
authors: Shea
authors: Larry B
authors: LB
authors: Goldstein
mesh_terms: D000368:Aged; D000369:Aged, 80 and over; D015331:Cohort Studies; D005260:Female; D006801:Humans; D002546:Ischemic Attack, Transient / Q000175:diagnosis*; D008297:Male; D006278:Medicare; D008875:Middle Aged; D009313:National Health Programs; D009316:National Institutes of Health (U.S.); D015588:Observer Variation; D011446:Prospective Studies; D012042:Registries; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012680:Sensitivity and Specificity; D012720:Severity of Illness Index; D020521:Stroke / Q000175:diagnosis*; D014481:United States