title: Alternative performance measures for prediction models.
abstract: As a performance measure for a prediction model, the area under the receiver operating characteristic curve (AUC) is insensitive to the addition of strong markers. A number of measures sensitive to performance change have recently been proposed; however, these relative-performance measures may lead to self-contradictory conclusions. This paper examines alternative performance measures for prediction models: the Lorenz curve-based Gini and Pietra indices, and a standardized version of the Brier score, the scaled Brier. Computer simulations are performed in order to study the sensitivity of these measures to performance change when a new marker is added to a baseline model. When the discrimination power of the added marker is concentrated in the gray zone of the baseline model, the AUC and the Gini show minimal performance improvements. The Pietra and the scaled Brier show more significant improvements in the same situation, comparatively. The Pietra and the scaled Brier indices are therefore recommended for prediction model performance measurement, in light of their ease of interpretation, clinical relevance and sensitivity to gray-zone resolving markers.
authors.affiliation: Institute of Epidemiology and Preventive Medicine, College of Public Health, National Taiwan University, Taipei, Taiwan.
authors.forename: Yun-Chun
authors.initials: YC
authors.lastname: Wu
authors.affiliation: Institute of Epidemiology and Preventive Medicine, College of Public Health, National Taiwan University, Taipei, Taiwan; Research Center for Genes, Environment and Human Health, College of Public Health, National Taiwan University, Taipei, Taiwan.
authors.forename: Wen-Chung
authors.initials: WC
authors.lastname: Lee
mesh_terms: D019540:Area Under Curve; D015415:Biomarkers / Q000378:metabolism; D004194:Disease; D006801:Humans; D015233:Models, Statistical; D016017:Odds Ratio; D018570:Risk Assessment
chemical_list: D015415:Biomarkers