title: Reliability of risk-adjusted outcomes for profiling hospital surgical quality.
abstract: OBJECTIVE
  Quality improvement platforms commonly use risk-adjusted morbidity and mortality to profile hospital performance. However, given small hospital caseloads and low event rates for some procedures, it is unclear whether these outcomes reliably reflect hospital performance.
  
  
  OBJECTIVE
  To determine the reliability of risk-adjusted morbidity and mortality for hospital performance profiling using clinical registry data.
  
  
  METHODS
  A retrospective cohort study was conducted using data from the American College of Surgeons National Surgical Quality Improvement Program, 2009. Participants included all patients (N = 55,466) who underwent colon resection, pancreatic resection, laparoscopic gastric bypass, ventral hernia repair, abdominal aortic aneurysm repair, and lower extremity bypass.
  
  
  METHODS
  Outcomes included risk-adjusted overall morbidity, severe morbidity, and mortality. We assessed reliability (0-1 scale: 0, completely unreliable; and 1, perfectly reliable) for all 3 outcomes. We also quantified the number of hospitals meeting minimum acceptable reliability thresholds (>0.70, good reliability; and >0.50, fair reliability) for each outcome.
  
  
  RESULTS
  For overall morbidity, the most common outcome studied, the mean reliability depended on sample size (ie, how high the hospital caseload was) and the event rate (ie, how frequently the outcome occurred). For example, mean reliability for overall morbidity was low for abdominal aortic aneurysm repair (reliability, 0.29; sample size, 25 cases per year; and event rate, 18.3%). In contrast, mean reliability for overall morbidity was higher for colon resection (reliability, 0.61; sample size, 114 cases per year; and event rate, 26.8%). Colon resection (37.7% of hospitals), pancreatic resection (7.1% of hospitals), and laparoscopic gastric bypass (11.5% of hospitals) were the only procedures for which any hospitals met a reliability threshold of 0.70 for overall morbidity. Because severe morbidity and mortality are less frequent outcomes, their mean reliability was lower, and even fewer hospitals met the thresholds for minimum reliability.
  
  
  CONCLUSIONS
  Most commonly reported outcome measures have low reliability for differentiating hospital performance. This is especially important for clinical registries that sample rather than collect 100% of cases, which can limit hospital case accrual. Eliminating sampling to achieve the highest possible caseloads, adjusting for reliability, and using advanced modeling strategies (eg, hierarchical modeling) are necessary for clinical registries to increase their benchmarking reliability.
authors.affiliation: Department of Surgery, University of Michigan Health System, Ann Arbor.
authors.forename: Robert W
authors.initials: RW
authors.lastname: Krell
authors.affiliation: Department of Surgery, Michigan State University College of Human Medicine, East Lansing.
authors.forename: Ahmed
authors.initials: A
authors.lastname: Hozain
authors.affiliation: Department of Surgery, The University of Texas at Houston Medical School, Houston.
authors.forename: Lillian S
authors.initials: LS
authors.lastname: Kao
authors.affiliation: Department of Surgery, University of Michigan Health System, Ann Arbor.
authors.forename: Justin B
authors.initials: JB
authors.lastname: Dimick
mesh_terms: D019985:Benchmarking / Q000592:standards / Q000706:statistics & numerical data; D015331:Cohort Studies; D017052:Hospital Mortality; D006801:Humans; D017063:Outcome Assessment, Health Care / Q000592:standards* / Q000706:statistics & numerical data*; D058996:Quality Improvement / Q000592:standards* / Q000706:statistics & numerical data*; D011787:Quality of Health Care / Q000592:standards* / Q000706:statistics & numerical data*; D012042:Registries / Q000592:standards / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D012189:Retrospective Studies; D020379:Risk Adjustment / Q000592:standards* / Q000706:statistics & numerical data*; D018401:Sample Size; D013514:Surgical Procedures, Operative / Q000401:mortality* / Q000592:standards*; D014481:United States