title: Calibration of medical diagnostic classifier scores to the probability of disease.
abstract: Scores produced by statistical classifiers in many clinical decision support systems and other medical diagnostic devices are generally on an arbitrary scale, so the clinical meaning of these scores is unclear. Calibration of classifier scores to a meaningful scale such as the probability of disease is potentially useful when such scores are used by a physician. In this work, we investigated three methods (parametric, semi-parametric, and non-parametric) for calibrating classifier scores to the probability of disease scale and developed uncertainty estimation techniques for these methods. We showed that classifier scores on arbitrary scales can be calibrated to the probability of disease scale without affecting their discrimination performance. With a finite dataset to train the calibration function, it is important to accompany the probability estimate with its confidence interval. Our simulations indicate that, when a dataset used for finding the transformation for calibration is also used for estimating the performance of calibration, the resubstitution bias exists for a performance metric involving the truth states in evaluating the calibration performance. However, the bias is small for the parametric and semi-parametric methods when the sample size is moderate to large (>100 per class).
authors: Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, USA.
authors: Weijie
authors: W
authors: Chen
authors: Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, USA.
authors: Berkman
authors: B
authors: Sahiner
authors: Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, USA.
authors: Frank
authors: F
authors: Samuelson
authors: Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, USA.
authors: Aria
authors: A
authors: Pezeshk
authors: Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, USA.
authors: Nicholas
authors: N
authors: Petrick
mesh_terms: D002138:Calibration; D016001:Confidence Intervals; D003933:Diagnosis; D004194:Disease / Q000145:classification*; D006801:Humans; D011336:Probability; D018401:Sample Size; D013223:Statistics as Topic; D018709:Statistics, Nonparametric
keywords: Calibration; classifier; probability of disease; rationality