Caution when using prognostic models: a prospective comparison of 3 recent prognostic models.
OBJECTIVE
Prognostic models have been developed to estimate mortality and to compare outcomes in different intensive care units. However, these models need to be validated before their use in different populations. In this study, we assessed the performance of 3 recently developed general prognostic models (Acute Physiologic and Chronic Health Evaluation [APACHE] IV, Simplified Acute Physiology Score [SAPS] 3 and Mortality Probability Model III [MPM(0)-III]) in a population admitted at 3 medical-surgical Brazilian intensive care units.


METHODS
All patients admitted from July 2008 to December 2009 were evaluated for inclusion in the study. Standardized mortality ratios were calculated for all models. Calibration was assessed by the Hosmer-Lemeshow goodness-of-fit test. Discrimination was evaluated using the area under the receiver operator curve.


RESULTS
A total of 5780 patients were included. Inhospital mortality was 9.1%. Discrimination was very good for all models (area under the receiver operator curve for APACHE IV, SAPS 3 and MPM(0)-III was 0.883, 0.855 and 0.840, respectively). APACHE IV showed better discrimination than SAPS 3 and MPM(0)-III (P < .001 for both comparisons). All models calibrated poorly and overestimated hospital mortality (Hosmer-Lemeshow statistic was 53.7, 134.2, 226.6 for APACHE IV, MPM(0)-III, and SAPS 3, respectively; P < .001 for all).


CONCLUSIONS
In this study, all models showed poor calibration, while discrimination was very good for all of them. As this has been a common finding in validation studies, caution is warranted when using prognostic models for benchmarking.
Adult Intensive Care Units—Hospital e Maternidade São Camilo—São Paulo, Brazil. paulo_nassar@yahoo.com.br
Antonio Paulo
AP
Nassar
Amilcar Oshiro
AO
Mocelin
André Luiz Baptiston
AL
Nunes
Fabio Poianas
FP
Giannini
Leonardo
L
Brauer
Fabio Moreira
FM
Andrade
Carlos Augusto
CA
Dias
D000293:Adolescent; D000328:Adult; D000367:Age Factors; D000368:Aged; D015331:Cohort Studies; D016638:Critical Illness / Q000401:mortality*; D003625:Data Collection; D003627:Data Interpretation, Statistical; D005260:Female; D006305:Health Status Indicators; D006801:Humans; D007362:Intensive Care Units / Q000706:statistics & numerical data*; D008297:Male; D008875:Middle Aged; D008962:Models, Theoretical; D017063:Outcome Assessment, Health Care; D011379:Prognosis; D012372:ROC Curve; D020379:Risk Adjustment; D012737:Sex Factors; D012959:Socioeconomic Factors; D016019:Survival Analysis; D014739:Victoria / Q000453:epidemiology; D055815:Young Adult