title: Learning classification models from multiple experts.
abstract: Building classification models from clinical data using machine learning methods often relies on labeling of patient examples by human experts. Standard machine learning framework assumes the labels are assigned by a homogeneous process. However, in reality the labels may come from multiple experts and it may be difficult to obtain a set of class labels everybody agrees on; it is not uncommon that different experts have different subjective opinions on how a specific patient example should be classified. In this work we propose and study a new multi-expert learning framework that assumes the class labels are provided by multiple experts and that these experts may differ in their class label assessments. The framework explicitly models different sources of disagreements and lets us naturally combine labels from different human experts to obtain: (1) a consensus classification model representing the model the group of experts converge to, as well as, and (2) individual expert models. We test the proposed framework by building a model for the problem of detection of the Heparin Induced Thrombocytopenia (HIT) where examples are labeled by three experts. We show that our framework is superior to multiple baselines (including standard machine learning framework in which expert differences are ignored) and that our framework leads to both improved consensus and individual expert models.
authors.affiliation: Department of Computer Science, University of Pittsburgh, United States. Electronic address: hamed@cs.pitt.edu.
authors.forename: Hamed
authors.initials: H
authors.lastname: Valizadegan
authors.forename: Quang
authors.initials: Q
authors.lastname: Nguyen
authors.forename: Milos
authors.initials: M
authors.lastname: Hauskrecht
mesh_terms: D001185:Artificial Intelligence; D006801:Humans; D008962:Models, Theoretical
keywords: Classification learning with multiple experts; Consensus models
references.citation: Proceedings (IEEE Int Conf Bioinformatics Biomed). 2011 Nov 12;2011:358-365
references.pmid: 22267987
references.citation: AMIA Annu Symp Proc. 2010 Nov 13;2010:286-90
references.pmid: 21346986
references.citation: Blood. 2000 Sep 1;96(5):1703-8
references.pmid: 10961867
references.citation: J Biomed Inform. 2013 Feb;46(1):47-55
references.pmid: 22944172
references.citation: Proc AMIA Symp. 1998;:538-42
references.pmid: 9929277
references.citation: KDD. 2012;2012:280-288
references.pmid: 25937993
references.citation: Br J Haematol. 2003 May;121(4):535-55
references.pmid: 12752095
references.citation: Stud Health Technol Inform. 2010;160(Pt 2):861-5
references.pmid: 20841808