title: A bootstrap approach for assessing the uncertainty of outcome probabilities when using a scoring system.
abstract: BACKGROUND
  Scoring systems are a very attractive family of clinical predictive models, because the patient score can be calculated without using any data processing system. Their weakness lies in the difficulty of associating a reliable prognostic probability with each score. In this study a bootstrap approach for estimating confidence intervals of outcome probabilities is described and applied to design and optimize the performance of a scoring system for morbidity in intensive care units after heart surgery.
  
  
  METHODS
  The bias-corrected and accelerated bootstrap method was used to estimate the 95% confidence intervals of outcome probabilities associated with a scoring system. These confidence intervals were calculated for each score and each step of the scoring-system design by means of one thousand bootstrapped samples. 1090 consecutive adult patients who underwent coronary artery bypass graft were assigned at random to two groups of equal size, so as to define random training and testing sets with equal percentage morbidities. A collection of 78 preoperative, intraoperative and postoperative variables were considered as likely morbidity predictors.
  
  
  RESULTS
  Several competing scoring systems were compared on the basis of discrimination, generalization and uncertainty associated with the prognostic probabilities. The results showed that confidence intervals corresponding to different scores often overlapped, making it convenient to unite and thus reduce the score classes. After uniting two adjacent classes, a model with six score groups not only gave a satisfactory trade-off between discrimination and generalization, but also enabled patients to be allocated to classes, most of which were characterized by well separated confidence intervals of prognostic probabilities.
  
  
  CONCLUSIONS
  Scoring systems are often designed solely on the basis of discrimination and generalization characteristics, to the detriment of prediction of a trustworthy outcome probability. The present example demonstrates that using a bootstrap method for the estimation of outcome-probability confidence intervals provides useful additional information about score-class statistics, guiding physicians towards the most convenient model for predicting morbidity outcomes in their clinical context.
authors.affiliation: Department of Surgery and Bioengineering, University of Siena, Siena, Italy.
authors.forename: Gabriele
authors.initials: G
authors.lastname: Cevenini
authors.forename: Paolo
authors.initials: P
authors.lastname: Barbini
mesh_terms: D000328:Adult; D001499:Bayes Theorem; D016001:Confidence Intervals; D001026:Coronary Artery Bypass / Q000009:adverse effects*; D016002:Discriminant Analysis; D005260:Female; D006801:Humans; D007362:Intensive Care Units; D007430:Intraoperative Care; D007558:Italy; D008297:Male; D015233:Models, Statistical; D009017:Morbidity; D011183:Postoperative Complications / Q000453:epidemiology; D011237:Predictive Value of Tests; D011300:Preoperative Care; D018570:Risk Assessment; D016896:Treatment Outcome; D035501:Uncertainty
references.citation: BMC Med Inform Decis Mak. 2007 Nov 22;7:36
references.pmid: 18034873
references.citation: Stat Med. 2008 Jan 30;27(2):213-23
references.pmid: 17357988
references.citation: Crit Care. 2006;10(3):R94
references.pmid: 16813658
references.citation: Ann Thorac Surg. 2000 Dec;70(6):2082-6
references.pmid: 11156124
references.citation: J Intensive Care Med. 2007 May-Jun;22(3):141-56
references.pmid: 17562738
references.citation: Crit Care Med. 1990 Oct;18(10):1111-8
references.pmid: 2209038
references.citation: Chest. 1997 Oct;112(4):1035-42
references.pmid: 9377914
references.citation: Clin Chem. 2008 Jan;54(1):17-23
references.pmid: 18024533
references.citation: Chest. 1985 May;87(5):580-4
references.pmid: 3987371
references.citation: BMC Med Inform Decis Mak. 2007 Nov 22;7:35
references.pmid: 18034872
references.citation: Eur J Cardiothorac Surg. 1999 Feb;15(2):166-72
references.pmid: 10219549
references.citation: Acta Anaesthesiol Scand. 1997 Aug;41(7):810-7
references.pmid: 9265921
references.citation: J Biomed Inform. 2002 Oct-Dec;35(5-6):352-9
references.pmid: 12968784
references.citation: Ann Thorac Surg. 1997 Oct;64(4):1050-8
references.pmid: 9354526
references.citation: Circulation. 1970 Jun;41(6):989-1001
references.pmid: 5482913
references.citation: Prof Case Manag. 2008 Jan-Feb;13(1):19-23; Quiz 24-5
references.pmid: 18162967
references.citation: Crit Care Med. 1996 Dec;24(12):1968-73
references.pmid: 8968263
references.citation: Stat Med. 2000 May 15;19(9):1141-64
references.pmid: 10797513
references.citation: Crit Care. 2007;11(1):109
references.pmid: 17316458
references.citation: Eur J Anaesthesiol. 2001 Sep;18(9):576-84
references.pmid: 11553252
references.citation: Crit Care. 2006;10(6):R167
references.pmid: 17134504
references.citation: Crit Care. 2005 Aug;9(4):R458-63
references.pmid: 16137361
references.citation: Biometrics. 1999 Jun;55(2):597-602
references.pmid: 11318220