title: Failure of Colorectal Surgical Site Infection Predictive Models Applied to an Independent Dataset: Do They Add Value or Just Confusion?
abstract: BACKGROUND
  Colorectal surgical site infections (C-SSIs) are a major source of postoperative morbidity. Institutional C-SSI rates are modeled and scrutinized, and there is increasing movement in the direction of public reporting. External validation of C-SSI risk prediction models is lacking. Factors governing C-SSI occurrence are complicated and multifactorial. We hypothesized that existing C-SSI prediction models have limited ability to accurately predict C-SSI in independent data.
  
  
  METHODS
  Colorectal resections identified from our institutional ACS-NSQIP dataset (2006 to 2014) were reviewed. The primary outcome was any C-SSI according to the ACS-NSQIP definition. Emergency cases were excluded. Published C-SSI risk scores: the National Nosocomial Infection Surveillance (NNIS), Contamination, Obesity, Laparotomy, and American Society of Anesthesiologists (ASA) class (COLA), Preventie Ziekenhuisinfecties door Surveillance (PREZIES), and NSQIP-based models were compared with receiver operating characteristic (ROC) analysis to evaluate discriminatory quality.
  
  
  RESULTS
  There were 2,376 cases included, with an overall C-SSI rate of 9% (213 cases). None of the models produced reliable and high quality C-SSI predictions. For any C-SSI, the NNIS c-index was 0.57 vs 0.61 for COLA, 0.58 for PREZIES, and 0.62 for NSQIP: all well below the minimum "reasonably" predictive c-index of 0.7. Predictions for superficial, deep, and organ space SSI were similarly poor.
  
  
  CONCLUSIONS
  Published C-SSI risk prediction models do not accurately predict C-SSI in our independent institutional dataset. Application of externally developed prediction models to any individual practice must be validated or modified to account for institution and case-mix specific factors. This questions the validity of using externally or nationally developed models for "expected" outcomes and interhospital comparisons.
authors: Department of Surgery, Mayo Clinic, Rochester, MN; Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN.
authors: John R
authors: JR
authors: Bergquist
authors: Department of Surgery, Mayo Clinic, Rochester, MN; Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN.
authors: Cornelius A
authors: CA
authors: Thiels
authors: Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN; Division of Colon and Rectal Surgery, Mayo Clinic, Scottsdale, AZ.
authors: David A
authors: DA
authors: Etzioni
authors: Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN; Division of Health Care Policy and Research, Mayo Clinic, Rochester, MN.
authors: Elizabeth B
authors: EB
authors: Habermann
authors: Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN; Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, MN. Electronic address: cima.robert@mayo.edu.
authors: Robert R
authors: RR
authors: Cima
mesh_terms: D000328:Adult; D000368:Aged; D003108:Colonic Diseases / Q000401:mortality / Q000473:pathology / Q000601:surgery*; D016208:Databases, Factual; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D011237:Predictive Value of Tests; D012372:ROC Curve; D012002:Rectal Diseases / Q000401:mortality / Q000473:pathology / Q000601:surgery*; D012189:Retrospective Studies; D012307:Risk Factors; D013530:Surgical Wound Infection / Q000175:diagnosis* / Q000453:epidemiology*; D014481:United States / Q000453:epidemiology