title: Crowd control: Effectively utilizing unscreened crowd workers for biomedical data annotation.
abstract: Annotating unstructured texts in Electronic Health Records data is usually a necessary step for conducting machine learning research on such datasets. Manual annotation by domain experts provides data of the best quality, but has become increasingly impractical given the rapid increase in the volume of EHR data. In this article, we examine the effectiveness of crowdsourcing with unscreened online workers as an alternative for transforming unstructured texts in EHRs into annotated data that are directly usable in supervised learning models. We find the crowdsourced annotation data to be just as effective as expert data in training a sentence classification model to detect the mentioning of abnormal ear anatomy in radiology reports of audiology. Furthermore, we have discovered that enabling workers to self-report a confidence level associated with each annotation can help researchers pinpoint less-accurate annotations requiring expert scrutiny. Our findings suggest that even crowd workers without specific domain knowledge can contribute effectively to the task of annotating unstructured EHR datasets.
authors.affiliation: Department of Biomedical and Health Informatics, The Children's Hospital of Philadelphia, United States; Computer and Information Sciences Department, University of Pennsylvania, United States. Electronic address: acocos@seas.upenn.edu.
authors.forename: Anne
authors.initials: A
authors.lastname: Cocos
authors.affiliation: Department of Biomedical and Health Informatics, The Children's Hospital of Philadelphia, United States. Electronic address: qiant@email.chop.edu.
authors.forename: Ting
authors.initials: T
authors.lastname: Qian
authors.affiliation: Computer and Information Sciences Department, University of Pennsylvania, United States.
authors.forename: Chris
authors.initials: C
authors.lastname: Callison-Burch
authors.affiliation: Department of Biomedical and Health Informatics, The Children's Hospital of Philadelphia, United States. Electronic address: masinoa@email.chop.edu.
authors.forename: Aaron J
authors.initials: AJ
authors.lastname: Masino
mesh_terms: D001298:Audiology; D063045:Crowdsourcing; D066289:Data Curation; D057286:Electronic Health Records; D006801:Humans; D011871:Radiology
keywords: Crowdsourcing; EHR data; Logistic regression; Sentence classification; Text annotations