title: How to improve the performance of intraoperative risk models: an example with vital signs using the surgical apgar score.
abstract: BACKGROUND
  Computerized reviews of patient data promise to improve patient care through early and accurate identification of at-risk and well patients. The significance of sampling strategy for patient vital signs data is not known. In the instance of the surgical Apgar score (SAS), we hypothesized that larger sampling intervals would improve the specificity and overall predictive ability of this tool.
  
  
  METHODS
  We used electronic intraoperative data from general and vascular surgical patients in a single-institution registry of the American College of Surgeons National Surgical Quality Improvement Program. The SAS, consisting of lowest heart rate, lowest mean arterial blood pressure, and estimated blood loss between incision and skin closure, was calculated using 5 methods: instantaneously and using intervals of of 5 and 10 minutes with and without interval overlap. Major complications including death were assessed at 30 days postoperatively.
  
  
  RESULTS
  Among 3000 patients, 272 (9.1%) experienced major complications or death. As the sampling interval increased from instantaneous (shortest) to 10 minutes without overlap (largest), the sensitivity, positive predictive value, and negative predictive value did not change significantly, but significant improvements were noted for specificity (79.5% to 82.9% across methods, P for trend <0.001) and accuracy (76.0% to 79.3% across methods, P for trend <0.01). In multivariate modeling, the predictive utility of the SAS as measured by the c-statistic nearly increased from Δc = +0.012 (P = 0.038) to Δc = +0.021 (P < 0.002) between the shortest and largest sampling intervals, respectively. Compared with a preoperative risk model, the net reclassification improvement and integrated discrimination improvement for the shortest versus largest sampling intervals of the SAS were net reclassification improvement 0.01 (P = 0.8) vs 0.06 (P = 0.02), and for integrated discrimination improvement, they were 0.008 (P < 0.01) vs 0.015 (P < 0.001).
  
  
  CONCLUSIONS
  When vital signs data are recorded in compliance with American Society of Anesthesiologists' standards, the sampling strategy for vital signs significantly influences performance of the SAS. Computerized reviews of patient data are subject to the choice of sampling methods for vital signs and may have the potential to be optimized for safe, efficient patient care.
authors.affiliation: From the Departments of *Anesthesiology, †Anesthesiology, Division of Critical Care Medicine, and ‡Surgery, Mayo Clinic, Rochester, MN.
authors.forename: Joseph A
authors.initials: JA
authors.lastname: Hyder
authors.forename: Daryl J
authors.initials: DJ
authors.lastname: Kor
authors.forename: Robert R
authors.initials: RR
authors.lastname: Cima
authors.forename: Arun
authors.initials: A
authors.lastname: Subramanian
mesh_terms: D000328:Adult; D000368:Aged; D062186:Arterial Pressure; D016063:Blood Loss, Surgical; D016009:Chi-Square Distribution; D005260:Female; D006339:Heart Rate; D006801:Humans; D016014:Linear Models; D016015:Logistic Models; D008297:Male; D008875:Middle Aged; D008910:Minnesota; D016343:Monitoring, Intraoperative; D015999:Multivariate Analysis; D016017:Odds Ratio; D011183:Postoperative Complications / Q000209:etiology* / Q000401:mortality; D011237:Predictive Value of Tests; D012042:Registries; D018570:Risk Assessment; D012307:Risk Factors; D013514:Surgical Procedures, Operative / Q000009:adverse effects* / Q000401:mortality; D013997:Time Factors; D055986:Vital Signs
references.citation: J Am Coll Surg. 1995 May;180(5):519-31
references.pmid: 7749526
references.citation: Anesthesiol Clin. 2011 Mar;29(1):57-69
references.pmid: 21295752
references.citation: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):639-44
references.pmid: 21508415
references.citation: Ann Surg. 2009 Sep;250(3):363-76
references.pmid: 19644350
references.citation: Crit Care. 2011;15(2):217
references.pmid: 21457500
references.citation: J Am Coll Surg. 2007 Feb;204(2):201-8
references.pmid: 17254923
references.citation: Zentralbl Chir. 2008 Apr;133(2):156-63
references.pmid: 18415903
references.citation: Surgery. 2010 Sep;148(3):559-66
references.pmid: 20227100
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:3803-6
references.pmid: 19163540
references.citation: Ann Surg. 2008 Aug;248(2):320-8
references.pmid: 18650644
references.citation: Acta Anaesthesiol Scand. 1996 Aug;40(7):783-91
references.pmid: 8874563
references.citation: J Trauma. 2009 Dec;67(6):1426-30
references.pmid: 20009697
references.citation: Ann Surg. 1998 Oct;228(4):491-507
references.pmid: 9790339
references.citation: Intern Med. 2012;51(8):871-6
references.pmid: 22504241
references.citation: Am J Epidemiol. 2011 Jun 1;173(11):1327-35
references.pmid: 21555714
references.citation: Stat Med. 2012 Jan 30;31(2):101-13
references.pmid: 22147389
references.citation: Acta Anaesthesiol Scand. 1994 Oct;38(7):679-90
references.pmid: 7839777
references.citation: Br J Surg. 2012 Feb;99(2):192-7
references.pmid: 22183685
references.citation: JAMA. 2010 Apr 28;303(16):1646-7
references.pmid: 20424257
references.citation: J Neurosurg Anesthesiol. 2012 Jan;24(1):58-62
references.pmid: 21862931
references.citation: Anesthesiology. 2007 Aug;107(2):213-20
references.pmid: 17667564
references.citation: Arch Surg. 2009 Jan;144(1):30-6; discussion 37
references.pmid: 19153322
references.citation: Am J Epidemiol. 2011 Aug 1;174(3):364-74
references.pmid: 21673124
references.citation: J Am Coll Surg. 2008 Nov;207(5):717-30
references.pmid: 18954785
references.citation: Anesthesiology. 2009 Jan;110(1):58-66
references.pmid: 19104171
references.citation: N Engl J Med. 2011 Nov 10;365(19):1756-8
references.pmid: 22070474
references.citation: J Gastrointest Surg. 2009 Aug;13(8):1529-38
references.pmid: 19319612
references.citation: Ann Surg. 2002 Sep;236(3):344-53; discussion 353-4
references.pmid: 12192321
references.citation: Epidemiology. 2010 Jan;21(1):128-38
references.pmid: 20010215
references.citation: Stat Med. 1997 May 15;16(9):965-80
references.pmid: 9160492
references.citation: J Emerg Med. 1988 Sep-Oct;6(5):417-9
references.pmid: 3066820
references.citation: Anesthesiology. 2012 Jun;116(6):1195-203
references.pmid: 22546967
references.citation: Int J Epidemiol. 2011 Aug;40(4):1106-8
references.pmid: 21571811
references.citation: J Am Coll Surg. 2010 Apr;210(4):503-8
references.pmid: 20347744