title: Comparison of outlier identification methods in hospital surgical quality improvement programs.
abstract: BACKGROUND
  Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set.
  
  
  METHODS
  Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared.
  
  
  RESULTS
  The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P < 0.01, P < 0.05, P < 0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality.
  
  
  CONCLUSIONS
  There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, 633 N. St. Clair Street, 22nd Floor, Chicago, IL 60611, USA. kbilimoria@facs.org
authors.forename: Karl Y
authors.initials: KY
authors.lastname: Bilimoria
authors.forename: Mark E
authors.initials: ME
authors.lastname: Cohen
authors.forename: Ryan P
authors.initials: RP
authors.lastname: Merkow
authors.forename: Xue
authors.initials: X
authors.lastname: Wang
authors.forename: David J
authors.initials: DJ
authors.lastname: Bentrem
authors.forename: Angela M
authors.initials: AM
authors.lastname: Ingraham
authors.forename: Karen
authors.initials: K
authors.lastname: Richards
authors.forename: Bruce L
authors.initials: BL
authors.lastname: Hall
authors.forename: Clifford Y
authors.initials: CY
authors.lastname: Ko
mesh_terms: D006801:Humans; D011785:Quality Assurance, Health Care; D058996:Quality Improvement; D020379:Risk Adjustment; D013503:Surgery Department, Hospital / Q000592:standards*; D013514:Surgical Procedures, Operative / Q000592:standards* / Q000706:statistics & numerical data; D014481:United States
references.citation: Ann Surg. 1998 Oct;228(4):491-507
references.pmid: 9790339
references.citation: J Am Coll Surg. 2007 Apr;204(4):550-60
references.pmid: 17382213
references.citation: J Clin Epidemiol. 2008 Mar;61(3):232-240
references.pmid: 18226745
references.citation: N Engl J Med. 2009 Oct 1;361(14):1368-75
references.pmid: 19797283
references.citation: Ann Surg. 2010 Jul;252(1):183-90
references.pmid: 20531000
references.citation: Ann Intern Med. 2008 Jan 15;148(2):111-23
references.pmid: 18195336
references.citation: J Am Coll Surg. 2007 Jun;204(6):1089-102
references.pmid: 17544068
references.citation: J Am Coll Surg. 2009 Dec;209(6):687-93
references.pmid: 19959035
references.citation: Med Care. 2009 Jan;47(1):1-8
references.pmid: 19106724
references.citation: Br J Surg. 2003 May;90(5):527-32
references.pmid: 12734856
references.citation: J Am Coll Surg. 2010 Feb;210(2):125-139.e2
references.pmid: 20113932
references.citation: Ann Surg. 2008 Aug;248(2):329-36
references.pmid: 18650645
references.citation: J Clin Epidemiol. 2007 Aug;60(8):858-62
references.pmid: 17606184
references.citation: Med Care. 2006 Apr;44(4):311-9
references.pmid: 16565631
references.citation: J Eval Clin Pract. 2001 Feb;7(1):35-45
references.pmid: 11240838
references.citation: J Am Coll Surg. 1997 Oct;185(4):328-40
references.pmid: 9328381