title: A spline-based tool to assess and visualize the calibration of multiclass risk predictions.
abstract: When validating risk models (or probabilistic classifiers), calibration is often overlooked. Calibration refers to the reliability of the predicted risks, i.e. whether the predicted risks correspond to observed probabilities. In medical applications this is important because treatment decisions often rely on the estimated risk of disease. The aim of this paper is to present generic tools to assess the calibration of multiclass risk models. We describe a calibration framework based on a vector spline multinomial logistic regression model. This framework can be used to generate calibration plots and calculate the estimated calibration index (ECI) to quantify lack of calibration. We illustrate these tools in relation to risk models used to characterize ovarian tumors. The outcome of the study is the surgical stage of the tumor when relevant and the final histological outcome, which is divided into five classes: benign, borderline malignant, stage I, stage II-IV, and secondary metastatic cancer. The 5909 patients included in the study are randomly split into equally large training and test sets. We developed and tested models using the following algorithms: logistic regression, support vector machines, k nearest neighbors, random forest, naive Bayes and nearest shrunken centroids. Multiclass calibration plots are interesting as an approach to visualizing the reliability of predicted risks. The ECI is a convenient tool for comparing models, but is less informative and interpretable than calibration plots. In our case study, logistic regression and random forest showed the highest degree of calibration, and the naive Bayes the lowest.
authors.affiliation: KU Leuven, Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics, Leuven, Belgium; KU Leuven, iMinds Medical Information Technologies, Leuven, Belgium.
authors.forename: K
authors.initials: K
authors.lastname: Van Hoorde
authors.affiliation: KU Leuven, Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics, Leuven, Belgium; KU Leuven, iMinds Medical Information Technologies, Leuven, Belgium.
authors.forename: S
authors.initials: S
authors.lastname: Van Huffel
authors.affiliation: Department of Obstetrics & Gynecology, University Hospitals Leuven, Leuven, Belgium; KU Leuven, Department of Development & Regeneration, Leuven, Belgium.
authors.forename: D
authors.initials: D
authors.lastname: Timmerman
authors.affiliation: Department of Obstetrics & Gynecology, University Hospitals Leuven, Leuven, Belgium; KU Leuven, Department of Development & Regeneration, Leuven, Belgium; Queen Charlotte's & Chelsea Hospital, Imperial College, Du Cane Road, London W12 0HS, UK.
authors.forename: T
authors.initials: T
authors.lastname: Bourne
authors.affiliation: KU Leuven, Department of Development & Regeneration, Leuven, Belgium. Electronic address: ben.vancalster@med.kuleuven.be.
authors.forename: B
authors.initials: B
authors.lastname: Van Calster
mesh_terms: D000328:Adult; D000368:Aged; D000465:Algorithms; D020000:Decision Support Systems, Clinical; D005260:Female; D006801:Humans; D016015:Logistic Models; D000069550:Machine Learning; D008875:Middle Aged; D015233:Models, Statistical; D010051:Ovarian Neoplasms / Q000145:classification; D018570:Risk Assessment / Q000379:methods*
keywords: Calibration; Logistic regression; Machine learning; Multiclass; Probability estimation; Risk models