title: A Comprehensive Evaluation of Statistical Reliability in ACS NSQIP Profiling Models.
abstract: OBJECTIVE
  To assess statistical reliability of hospital profiling models in ACS NSQIP (American College of Surgeons' National Surgical Quality Improvement Program).
  
  
  BACKGROUND
  The ACS NSQIP January 2013 Semiannual Report provided risk-adjusted hospital quality assessments for 137 models.
  
  
  METHODS
  Median reliability and percentage of hospitals achieving acceptable reliability were computed for each model. Average median reliability was computed across models with common outcomes.
  
  
  RESULTS
  Median reliability varied across the 137 models, from a high of 0.91 for "All Cases Morbidity" to a low of 0.005 for "Procedure-Targeted Total Hip Arthroplasty Surgical Site Infection." Generally, reliability was greatest for models with larger sample sizes and higher outcome event rates. Among "Essentials" models, 72% attained a median reliability of 0.40 or more, and 24% of 0.70 or more. Among "Procedure-Targeted" models, 29% attained a median reliability of 0.40 or more, and 3% of 0.70 or more. Percentage of hospitals achieving an acceptable reliability of 0.40 ranged from 98% for "All Cases Morbidity" to 0% for "Procedure-Targeted Pancreatectomy Mortality." For Essentials models, average median reliability for each outcome, except mortality, was more than 0.40. However, for Procedure-Targeted models the average median was less than 0.40.
  
  
  CONCLUSIONS
  For a large proportion of ACS NSQIP Essentials models, statistical reliability is adequate for assessing surgical quality and differentiating hospital performance. The Procedure-Targeted program is evolving in terms of statistical reliability, with promising results to date. These results also argue for broader discussions of statistical reliability in performance assessments for the profession.
authors.affiliation: *Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL †Department of Surgery, University of California Los Angeles David Geffen School of Medicine, and VA Greater Los Angeles Healthcare System, Los Angeles, CA; and ‡Department of Surgery, Washington University in St. Louis; Center for Health Policy and the Olin Business School at Washington University in St Louis; John Cochran Veterans Affairs Medical Center; and BJC Healthcare, St. Louis, MO.
authors.forename: Kristopher M
authors.initials: KM
authors.lastname: Huffman
authors.forename: Mark E
authors.initials: ME
authors.lastname: Cohen
authors.forename: Clifford Y
authors.initials: CY
authors.lastname: Ko
authors.forename: Bruce Lee
authors.initials: BL
authors.lastname: Hall
mesh_terms: D006761:Hospitals / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D015233:Models, Statistical; D017063:Outcome Assessment, Health Care; D011785:Quality Assurance, Health Care; D058996:Quality Improvement; D015203:Reproducibility of Results; D013514:Surgical Procedures, Operative / Q000592:standards* / Q000706:statistics & numerical data; D014481:United States / Q000453:epidemiology