title: How to get the most out of your curation effort.
abstract: Large-scale annotation efforts typically involve several experts who may disagree with each other. We propose an approach for modeling disagreements among experts that allows providing each annotation with a confidence value (i.e., the posterior probability that it is correct). Our approach allows computing certainty-level for individual annotations, given annotator-specific parameters estimated from data. We developed two probabilistic models for performing this analysis, compared these models using computer simulation, and tested each model's actual performance, based on a large data set generated by human annotators specifically for this study. We show that even in the worst-case scenario, when all annotators disagree, our approach allows us to significantly increase the probability of choosing the correct annotation. Along with this publication we make publicly available a corpus of 10,000 sentences annotated according to several cardinal dimensions that we have introduced in earlier work. The 10,000 sentences were all 3-fold annotated by a group of eight experts, while a 1,000-sentence subset was further 5-fold annotated by five new experts. While the presented data represent a specialized curation task, our modeling approach is general; most data annotation studies could benefit from our methodology.
authors.affiliation: Department of Medicine, University of Chicago, Chicago, Illinois, USA.
authors.forename: Andrey
authors.initials: A
authors.lastname: Rzhetsky
authors.forename: Hagit
authors.initials: H
authors.lastname: Shatkay
authors.forename: W John
authors.initials: WJ
authors.lastname: Wilbur
mesh_terms: D000043:Abstracting and Indexing; D019295:Computational Biology / Q000379:methods*; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D003628:Database Management Systems; D016208:Databases, Factual; D008962:Models, Theoretical; D010363:Pattern Recognition, Automated; D009626:Terminology as Topic