title: Binary Classifier Calibration Using a Bayesian Non-Parametric Approach.
abstract: Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in Data mining. This paper presents two new non-parametric methods for calibrating outputs of binary classification models: a method based on the Bayes optimal selection and a method based on the Bayesian model averaging. The advantage of these methods is that they are independent of the algorithm used to learn a predictive model, and they can be applied in a post-processing step, after the model is learned. This makes them applicable to a wide variety of machine learning models and methods. These calibration methods, as well as other methods, are tested on a variety of datasets in terms of both discrimination and calibration performance. The results show the methods either outperform or are comparable in performance to the state-of-the-art calibration methods.
authors.affiliation: Intelligent Systems Program, University of Pittsburgh.
authors.forename: Mahdi Pakdaman
authors.initials: MP
authors.lastname: Naeini
authors.affiliation: Department of Biomedical Informatics, University of Pittsburgh.
authors.forename: Gregory F
authors.initials: GF
authors.lastname: Cooper
authors.affiliation: Computer Science Department, University of Pittsburgh.
authors.forename: Milos
authors.initials: M
authors.lastname: Hauskrecht