Obtaining Well Calibrated Probabilities Using Bayesian Binning.
Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in artificial intelligence. In this paper we present a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) which addresses key limitations of existing calibration methods. The method post processes the output of a binary classification algorithm; thus, it can be readily combined with many existing classification algorithms. The method is computationally tractable, and empirically accurate, as evidenced by the set of experiments reported here on both real and simulated datasets.
Intelligent Systems Program, University of Pittsburgh, PA, USA.
Mahdi Pakdaman
MP
Naeini
Intelligent Systems Program, University of Pittsburgh, PA, USA ; Department of Biomedical Informatics, University of Pittsburgh, PA, USA.
Gregory F
GF
Cooper
Intelligent Systems Program, University of Pittsburgh, PA, USA ; Computer Science Department, University of Pittsburgh, PA, USA.
Milos
M
Hauskrecht