title: Evaluation and Enhancement of Calibration in the American College of Surgeons NSQIP Surgical Risk Calculator.
abstract: The American College of Surgeon (ACS) NSQIP Surgical Risk Calculator has been widely adopted as a decision aid and informed consent tool by surgeons and patients. Previous evaluations showed excellent discrimination and combined discrimination and calibration, but model calibration alone, and potential benefits of recalibration, were not explored. Because lack of calibration can lead to systematic errors in assessing surgical risk, our objective was to assess calibration and determine whether spline-based adjustments could improve it.
  
  
  
  We evaluated Surgical Risk Calculator model calibration, as well as discrimination, for each of 11 outcomes modeled from nearly 3 million patients (2010 to 2014). Using independent random subsets of data, we evaluated model performance for the Development (60% of records), Validation (20%), and Test (20%) datasets, where prediction equations from the Development dataset were recalibrated using restricted cubic splines estimated from the Validation dataset. We also evaluated performance on data subsets composed of higher-risk operations.
  
  
  
  The nonrecalibrated Surgical Risk Calculator performed well, but there was a slight tendency for predicted risk to be overestimated for lowest- and highest-risk patients and underestimated for moderate-risk patients. After recalibration, this distortion was eliminated, and p values for miscalibration were most often nonsignificant. Calibration was also excellent for subsets of higher-risk operations, though observed calibration was reduced due to instability associated with smaller sample sizes.
  
  
  
  Performance of NSQIP Surgical Risk Calculator models was shown to be excellent and improved with recalibration. Surgeons and patients can rely on the calculator to provide accurate estimates of surgical risk.
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Yaoming
authors.initials: Y
authors.lastname: Liu
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. Electronic address: markcohen@facs.org.
authors.forename: Mark E
authors.initials: ME
authors.lastname: Cohen
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, Washington University in St. Louis; Center for Health Policy and the Olin Business School at Washington University in St Louis; John Cochran Veterans Affairs Medical Center; and BJC Healthcare, St Louis, MO.
authors.forename: Bruce L
authors.initials: BL
authors.lastname: Hall
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California Los Angeles David Geffen School of Medicine and the VA Greater Los Angeles Healthcare System, Los Angeles, CA.
authors.forename: Clifford Y
authors.initials: CY
authors.lastname: Ko
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Surgical Outcomes and Quality Improvement Center (SOQIC), Department of Surgery and Center for Healthcare Studies, Feinberg School of Medicine and Northwestern Medicine, Northwestern University, Chicago, IL.
authors.forename: Karl Y
authors.initials: KY
authors.lastname: Bilimoria
mesh_terms: D002138:Calibration; D000066491:Clinical Decision-Making / Q000379:methods*; D003661:Decision Support Techniques; D006801:Humans; D016015:Logistic Models; D011183:Postoperative Complications / Q000175:diagnosis* / Q000209:etiology / Q000401:mortality; D018570:Risk Assessment / Q000379:methods; D012307:Risk Factors; D012955:Societies, Medical; D013514:Surgical Procedures, Operative / Q000401:mortality*; D014481:United States