title: How best to measure surgical quality? Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicators (AHRQ-PSI) and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) postoperative adverse events at a single institution.
abstract: BACKGROUND
  Evaluating surgical outcomes is an important tool to compare providers and institutions and to drive process improvements. Differing methodologies, however, may provide conflicting measurements of similar clinical outcomes making comparisons difficult. ACS-NSQIP is a validated, risk-adjusted, clinically derived data methodology to compare observed to expected outcomes after a wide variety of operations. The AHRQ-PSI are a set of computer algorithms to identify potential adverse in-patient events using secondary ICD-9-CM diagnosis and procedure codes from hospital discharge abstracts.
  
  
  METHODS
  We compared the ACS-NSQIP and AHRQ-PSI methods for hospital general surgical (n = 6565) or vascular surgical inpatients procedures (n = 1041) at a tertiary-care academic institution from April 2006 to June 2009 on 7 adverse event types.
  
  
  RESULTS
  ACS-NSQIP inpatient adverse events were identified in 564 (7.4%) patients. AHRQ-PSIs were identified in 268 (3.5%) patients. Only 159 (2.1%) patients had inpatient events identified by both methods. Using ACS-NSQIP as the clinically based standard the sensitivity of the specific AHRQ-PSI ranged from 0.030 for infections to 0.535 for PE/DVT. Positive predictive values of AHRQ-PSI ranged from 18% for hemorrhage/hematoma to 89% for renal failure. Greater agreement at greater ASA class and wound classification was observed.
  
  
  CONCLUSIONS
  AHRQ-PSI algorithms identified less than a third of the ACS-NSQIP clinically important adverse events. Furthermore, the AHRQ-PSI identified a large number of events with no corresponding clinically important adverse outcomes. The sensitivity of the AHRQ-PSI for detecting clinically relevant adverse events identified by the ACS-NSQIP varied widely. The AHRQ-PSI as applied to postoperative patients is a poor measure of quality performance.
authors.affiliation: Department of Surgery, Mayo Clinic College of Medicine, Rochester, MN 55905, USA. cima.robert@mayo.edu
authors.forename: Robert R
authors.initials: RR
authors.lastname: Cima
authors.forename: Kandace A
authors.initials: KA
authors.lastname: Lackore
authors.forename: Sharon A
authors.initials: SA
authors.lastname: Nehring
authors.forename: Stephen D
authors.initials: SD
authors.lastname: Cassivi
authors.forename: John H
authors.initials: JH
authors.lastname: Donohue
authors.forename: Claude
authors.initials: C
authors.lastname: Deschamps
authors.forename: Monica
authors.initials: M
authors.lastname: Vansuch
authors.forename: James M
authors.initials: JM
authors.lastname: Naessens
mesh_terms: D000046:Academic Medical Centers / Q000592:standards; D000465:Algorithms; D013502:General Surgery / Q000592:standards*; D006302:Health Services Research / Q000379:methods / Q000592:standards; D006801:Humans; D010351:Patient Discharge / Q000706:statistics & numerical data; D011183:Postoperative Complications / Q000453:epidemiology*; D011785:Quality Assurance, Health Care / Q000379:methods* / Q000592:standards*; D019984:Quality Indicators, Health Care / Q000592:standards; D015203:Reproducibility of Results; D017751:Safety Management / Q000379:methods* / Q000592:standards*; D012955:Societies, Medical / Q000592:standards; D014481:United States; D016270:United States Agency for Healthcare Research and Quality / Q000592:standards