Assessing the discriminative ability of risk models for more than two outcome categories.
The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a "discrimination plot" as a tool to visualize the model's discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the 'conditional-risk' method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.
Department of Development and Regeneration, KU Leuven, University of Leuven, Leuven, Belgium. ben.vancalster@med.kuleuven.be
Ben
B
Van Calster
Yvonne
Y
Vergouwe
Caspar W N
CW
Looman
Vanya
V
Van Belle
Dirk
D
Timmerman
Ewout W
EW
Steyerberg
D003627:Data Interpretation, Statistical; D016002:Discriminant Analysis; D005260:Female; D006801:Humans; D016015:Logistic Models; D008297:Male; D015233:Models, Statistical; D010051:Ovarian Neoplasms / Q000175:diagnosis / Q000453:epidemiology; D015995:Prevalence; D011379:Prognosis; D012372:ROC Curve; D018570:Risk Assessment; D013736:Testicular Neoplasms / Q000175:diagnosis / Q000453:epidemiology