Why the C-statistic is not informative to evaluate early warning scores and what metrics to use.
Metrics typically used to report the performance of an early warning score (EWS), such as the area under the receiver operator characteristic curve or C-statistic, are not useful for pre-implementation analyses. Because physiological deterioration has an extremely low prevalence of 0.02 per patient-day, these metrics can be misleading. We discuss the statistical reasoning behind this statement and present a novel alternative metric more adequate to operationalize an EWS. We suggest that pre-implementation evaluation of EWSs should include at least two metrics: sensitivity; and either the positive predictive value, number needed to evaluate, or estimated rate of alerts. We also argue the importance of reporting each individual cutoff value.
Healthcare Systems Engineering Program, Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, 200 First Street SW, Rochester, MN, 55905, USA. RomeroBrufau.Santiago@mayo.edu.|Division of Health Care Policy and Research, Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA. RomeroBrufau.Santiago@mayo.edu.
Santiago
S
Romero-Brufau
Healthcare Systems Engineering Program, Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, 200 First Street SW, Rochester, MN, 55905, USA. Huddleston.jeanne@mayo.edu.|Division of Health Care Policy and Research, Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA. Huddleston.jeanne@mayo.edu.|Division of Hospital Internal Medicine, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA. Huddleston.jeanne@mayo.edu.
Jeanne M
JM
Huddleston
Kaiser Permanente Division of Research, 2000 Broadway Avenue, 032 R01, Oakland, CA, 94612, USA. Gabriel.Escobar@kp.org.
Gabriel J
GJ
Escobar
Division of General Internal Medicine, Mayo Clinic College of Medicine, 200 First Street SW, Rochester, MN, 55905, USA. mliebow@mayo.edu.
Mark
M
Liebow
D003422:Critical Care / Q000379:methods* / Q000592:standards; D003661:Decision Support Techniques; D006801:Humans; D012372:ROC Curve; D013223:Statistics as Topic