title: Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing.
abstract: BACKGROUND
  A high-quality gold standard is vital for supervised, machine learning-based, clinical natural language processing (NLP) systems. In clinical NLP projects, expert annotators traditionally create the gold standard. However, traditional annotation is expensive and time-consuming. To reduce the cost of annotation, general NLP projects have turned to crowdsourcing based on Web 2.0 technology, which involves submitting smaller subtasks to a coordinated marketplace of workers on the Internet. Many studies have been conducted in the area of crowdsourcing, but only a few have focused on tasks in the general NLP field and only a handful in the biomedical domain, usually based upon very small pilot sample sizes. In addition, the quality of the crowdsourced biomedical NLP corpora were never exceptional when compared to traditionally-developed gold standards. The previously reported results on medical named entity annotation task showed a 0.68 F-measure based agreement between crowdsourced and traditionally-developed corpora.
  
  
  OBJECTIVE
  Building upon previous work from the general crowdsourcing research, this study investigated the usability of crowdsourcing in the clinical NLP domain with special emphasis on achieving high agreement between crowdsourced and traditionally-developed corpora.
  
  
  METHODS
  To build the gold standard for evaluating the crowdsourcing workers' performance, 1042 clinical trial announcements (CTAs) from the ClinicalTrials.gov website were randomly selected and double annotated for medication names, medication types, and linked attributes. For the experiments, we used CrowdFlower, an Amazon Mechanical Turk-based crowdsourcing platform. We calculated sensitivity, precision, and F-measure to evaluate the quality of the crowd's work and tested the statistical significance (P<.001, chi-square test) to detect differences between the crowdsourced and traditionally-developed annotations.
  
  
  RESULTS
  The agreement between the crowd's annotations and the traditionally-generated corpora was high for: (1) annotations (0.87, F-measure for medication names; 0.73, medication types), (2) correction of previous annotations (0.90, medication names; 0.76, medication types), and excellent for (3) linking medications with their attributes (0.96). Simple voting provided the best judgment aggregation approach. There was no statistically significant difference between the crowd and traditionally-generated corpora. Our results showed a 27.9% improvement over previously reported results on medication named entity annotation task.
  
  
  CONCLUSIONS
  This study offers three contributions. First, we proved that crowdsourcing is a feasible, inexpensive, fast, and practical approach to collect high-quality annotations for clinical text (when protected health information was excluded). We believe that well-designed user interfaces and rigorous quality control strategy for entity annotation and linking were critical to the success of this work. Second, as a further contribution to the Internet-based crowdsourcing field, we will publicly release the JavaScript and CrowdFlower Markup Language infrastructure code that is necessary to utilize CrowdFlower's quality control and crowdsourcing interfaces for named entity annotations. Finally, to spur future research, we will release the CTA annotations that were generated by traditional and crowdsourced approaches.
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH 45229, USA.
authors.forename: Haijun
authors.initials: H
authors.lastname: Zhai
authors.forename: Todd
authors.initials: T
authors.lastname: Lingren
authors.forename: Louise
authors.initials: L
authors.lastname: Deleger
authors.forename: Qi
authors.initials: Q
authors.lastname: Li
authors.forename: Megan
authors.initials: M
authors.lastname: Kaiser
authors.forename: Laura
authors.initials: L
authors.lastname: Stoutenborough
authors.forename: Imre
authors.initials: I
authors.lastname: Solti
mesh_terms: D002986:Clinical Trials as Topic / Q000706:statistics & numerical data; D063045:Crowdsourcing / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D020407:Internet; D009323:Natural Language Processing; D010865:Pilot Projects; D011786:Quality Control; D061108:Social Media; D017216:Telemedicine / Q000592:standards* / Q000706:statistics & numerical data
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):922-30
references.pmid: 23355458
references.citation: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):540-3
references.pmid: 21846785
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):915-21
references.pmid: 23268488
references.citation: AMIA Annu Symp Proc. 2012;2012:144-53
references.pmid: 23304283
references.citation: Sci Transl Med. 2011 Jun 22;3(88):88mr1
references.pmid: 21697527
references.citation: J Med Internet Res. 2012 Nov 29;14(6):e167
references.pmid: 23196001
references.citation: J Med Internet Res. 2012 Jun 04;14(3):e79
references.pmid: 22664384
references.citation: AMIA Annu Symp Proc. 2010;2010:1316
references.pmid: 21785667