title: Reliability adjustment for reporting hospital outcomes with surgery.
abstract: BACKGROUND
  Reliability adjustment, a novel technique for quantifying and removing statistical "noise" from quality rankings, is becoming more widely used outside surgery. We sought to evaluate its impact on hospital outcomes assessed with the American College of Surgeons' National Surgical Quality Improvement Program (ACS-NSQIP).
  
  
  METHODS
  We used prospective, clinical data from the ACS-NSQIP to identify all patients undergoing colon resection in 2007 (n = 181 hospitals, n = 18,455 patients). We first used standard NSQIP techniques to generate risk-adjusted mortality and morbidity rates for each hospital. Using hierarchical logistic regression models, we then adjusted these for reliability using empirical Bayes techniques. To evaluate the impact of reliability adjustment, we first estimated the extent to which hospital-level variation was reduced. We then compared hospital mortality and morbidity rankings and outlier status before and after reliability adjustment.
  
  
  RESULTS
  Reliability adjustment greatly diminished apparent variation in hospital outcomes. For risk-adjusted mortality, there was a 6-fold difference before (1.4%-7.8%) and less than a 2-fold difference (3.2% to 5.7%) after reliability adjustment. For risk-adjusted morbidity, there was a 2-fold difference (18.0%-38.2%) before and a 1.5-fold difference (20.8%-34.8%) after reliability adjustment. Reliability adjustment had a large impact on hospital mortality and morbidity rankings. For example, with rankings based on mortality, 44% (16 hospitals) of the "best" hospitals (top 20%) were reclassified after reliability adjustment. Similarly, 22% (8 hospitals) of the "worst" hospitals (bottom 20%) were reclassified after reliability adjustment.
  
  
  CONCLUSIONS
  Reliability adjustment reduces variation due to statistical noise and results in more accurate estimates of risk-adjusted hospital outcomes. Given the risk of misclassifying hospitals and surgeons using standard approaches, this technique should be considered when reporting surgical outcomes.
authors.affiliation: Center for Healthcare Outcomes and Policy and the Department of Surgery, University of Michigan, Ann Arbor, MI 48109, USA. jdimick@umich.edu
authors.forename: Justin B
authors.initials: JB
authors.lastname: Dimick
authors.forename: Amir A
authors.initials: AA
authors.lastname: Ghaferi
authors.forename: Nicholas H
authors.initials: NH
authors.lastname: Osborne
authors.forename: Clifford Y
authors.initials: CY
authors.lastname: Ko
authors.forename: Bruce L
authors.initials: BL
authors.lastname: Hall
mesh_terms: D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D001499:Bayes Theorem; D003082:Colectomy / Q000401:mortality / Q000592:standards*; D006761:Hospitals / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D016015:Logistic Models; D008875:Middle Aged; D017063:Outcome Assessment, Health Care / Q000379:methods* / Q000706:statistics & numerical data; D011183:Postoperative Complications / Q000453:epidemiology*; D011787:Quality of Health Care; D012042:Registries; D015203:Reproducibility of Results; D020379:Risk Adjustment / Q000379:methods* / Q000706:statistics & numerical data