title: Deriving the expected utility of a predictive model when the utilities are uncertain.
abstract: Predictive models are often constructed from clinical databases with the goal of eventually helping make better clinical decisions. Evaluating models using decision theory is therefore natural. When constructing a model using statistical and machine learning methods, however, we are often uncertain about precisely how the model will be used. Thus, decision-independent measures of classification performance, such as the area under an ROC curve, are popular. As a complementary method of evaluation, we investigate techniques for deriving the expected utility of a model under uncertainty about the model's utilities. We demonstrate an example of the application of this approach to the evaluation of two models that diagnose coronary artery disease.
authors.affiliation: Center for Biomedical Informatics and the Intelligent Systems Program, University of Pittsburgh, Pennsylvania, USA.
authors.forename: Gregory F
authors.initials: GF
authors.lastname: Cooper
authors.forename: Shyam
authors.initials: S
authors.lastname: Visweswaran
mesh_terms: D001185:Artificial Intelligence; D001499:Bayes Theorem; D003657:Decision Making; D003661:Decision Support Techniques; D003663:Decision Trees; D005069:Evaluation Studies as Topic; D015233:Models, Statistical; D012372:ROC Curve