A bootstrap approach for assessing the uncertainty of outcome probabilities when using a scoring system.
BACKGROUND
Scoring systems are a very attractive family of clinical predictive models, because the patient score can be calculated without using any data processing system. Their weakness lies in the difficulty of associating a reliable prognostic probability with each score. In this study a bootstrap approach for estimating confidence intervals of outcome probabilities is described and applied to design and optimize the performance of a scoring system for morbidity in intensive care units after heart surgery.


METHODS
The bias-corrected and accelerated bootstrap method was used to estimate the 95% confidence intervals of outcome probabilities associated with a scoring system. These confidence intervals were calculated for each score and each step of the scoring-system design by means of one thousand bootstrapped samples. 1090 consecutive adult patients who underwent coronary artery bypass graft were assigned at random to two groups of equal size, so as to define random training and testing sets with equal percentage morbidities. A collection of 78 preoperative, intraoperative and postoperative variables were considered as likely morbidity predictors.


RESULTS
Several competing scoring systems were compared on the basis of discrimination, generalization and uncertainty associated with the prognostic probabilities. The results showed that confidence intervals corresponding to different scores often overlapped, making it convenient to unite and thus reduce the score classes. After uniting two adjacent classes, a model with six score groups not only gave a satisfactory trade-off between discrimination and generalization, but also enabled patients to be allocated to classes, most of which were characterized by well separated confidence intervals of prognostic probabilities.


CONCLUSIONS
Scoring systems are often designed solely on the basis of discrimination and generalization characteristics, to the detriment of prediction of a trustworthy outcome probability. The present example demonstrates that using a bootstrap method for the estimation of outcome-probability confidence intervals provides useful additional information about score-class statistics, guiding physicians towards the most convenient model for predicting morbidity outcomes in their clinical context.
Cevenini
Gabriele
G
Department of Surgery and Bioengineering, University of Siena, Siena, Italy.
Barbini
Paolo
P
D000328:Adult; D001499:Bayes Theorem; D016001:Confidence Intervals; D001026:Coronary Artery Bypass / Q000009:adverse effects*; D016002:Discriminant Analysis; D005260:Female; D006801:Humans; D007362:Intensive Care Units; D007430:Intraoperative Care; D007558:Italy; D008297:Male; D015233:Models, Statistical; D009017:Morbidity; D011183:Postoperative Complications / Q000453:epidemiology; D011237:Predictive Value of Tests; D011300:Preoperative Care; D018570:Risk Assessment; D016896:Treatment Outcome; D035501:Uncertainty
BMC Med Inform Decis Mak. 2007 Nov 22;7:36
18034873
Stat Med. 2008 Jan 30;27(2):213-23
17357988
Crit Care. 2006;10(3):R94
16813658
Ann Thorac Surg. 2000 Dec;70(6):2082-6
11156124
J Intensive Care Med. 2007 May-Jun;22(3):141-56
17562738
Crit Care Med. 1990 Oct;18(10):1111-8
2209038
Chest. 1997 Oct;112(4):1035-42
9377914
Clin Chem. 2008 Jan;54(1):17-23
18024533
Chest. 1985 May;87(5):580-4
3987371
BMC Med Inform Decis Mak. 2007 Nov 22;7:35
18034872
Eur J Cardiothorac Surg. 1999 Feb;15(2):166-72
10219549
Acta Anaesthesiol Scand. 1997 Aug;41(7):810-7
9265921
J Biomed Inform. 2002 Oct-Dec;35(5-6):352-9
12968784
Ann Thorac Surg. 1997 Oct;64(4):1050-8
9354526
Circulation. 1970 Jun;41(6):989-1001
5482913
Prof Case Manag. 2008 Jan-Feb;13(1):19-23; Quiz 24-5
18162967
Crit Care Med. 1996 Dec;24(12):1968-73
8968263
Stat Med. 2000 May 15;19(9):1141-64
10797513
Crit Care. 2007;11(1):109
17316458
Eur J Anaesthesiol. 2001 Sep;18(9):576-84
11553252
Crit Care. 2006;10(6):R167
17134504
Crit Care. 2005 Aug;9(4):R458-63
16137361
Biometrics. 1999 Jun;55(2):597-602
11318220