Comparison of outlier identification methods in hospital surgical quality improvement programs.
BACKGROUND
Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set.


METHODS
Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared.


RESULTS
The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P < 0.01, P < 0.05, P < 0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality.


CONCLUSIONS
There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.
Bilimoria
Karl Y
KY
Division of Research and Optimal Patient Care, American College of Surgeons, 633 N. St. Clair Street, 22nd Floor, Chicago, IL 60611, USA. kbilimoria@facs.org
Cohen
Mark E
ME
Merkow
Ryan P
RP
Wang
Xue
X
Bentrem
David J
DJ
Ingraham
Angela M
AM
Richards
Karen
K
Hall
Bruce L
BL
Ko
Clifford Y
CY
D006801:Humans; D011785:Quality Assurance, Health Care; D058996:Quality Improvement; D020379:Risk Adjustment; D013503:Surgery Department, Hospital / Q000592:standards*; D013514:Surgical Procedures, Operative / Q000592:standards* / Q000706:statistics & numerical data; D014481:United States
Ann Surg. 1998 Oct;228(4):491-507
9790339
J Am Coll Surg. 2007 Apr;204(4):550-60
17382213
J Clin Epidemiol. 2008 Mar;61(3):232-240
18226745
N Engl J Med. 2009 Oct 1;361(14):1368-75
19797283
Ann Surg. 2010 Jul;252(1):183-90
20531000
Ann Intern Med. 2008 Jan 15;148(2):111-23
18195336
J Am Coll Surg. 2007 Jun;204(6):1089-102
17544068
J Am Coll Surg. 2009 Dec;209(6):687-93
19959035
Med Care. 2009 Jan;47(1):1-8
19106724
Br J Surg. 2003 May;90(5):527-32
12734856
J Am Coll Surg. 2010 Feb;210(2):125-139.e2
20113932
Ann Surg. 2008 Aug;248(2):329-36
18650645
J Clin Epidemiol. 2007 Aug;60(8):858-62
17606184
Med Care. 2006 Apr;44(4):311-9
16565631
J Eval Clin Pract. 2001 Feb;7(1):35-45
11240838
J Am Coll Surg. 1997 Oct;185(4):328-40
9328381