title: Community annotation experiment for ground truth generation for the i2b2 medication challenge.
abstract: OBJECTIVE
  Within the context of the Third i2b2 Workshop on Natural Language Processing Challenges for Clinical Records, the authors (also referred to as 'the i2b2 medication challenge team' or 'the i2b2 team' for short) organized a community annotation experiment.
  
  
  METHODS
  For this experiment, the authors released annotation guidelines and a small set of annotated discharge summaries. They asked the participants of the Third i2b2 Workshop to annotate 10 discharge summaries per person; each discharge summary was annotated by two annotators from two different teams, and a third annotator from a third team resolved disagreements.
  
  
  METHODS
  In order to evaluate the reliability of the annotations thus produced, the authors measured community inter-annotator agreement and compared it with the inter-annotator agreement of expert annotators when both the community and the expert annotators generated ground truth based on pooled system outputs. For this purpose, the pool consisted of the three most densely populated automatic annotations of each record. The authors also compared the community inter-annotator agreement with expert inter-annotator agreement when the experts annotated raw records without using the pool. Finally, they measured the quality of the community ground truth by comparing it with the expert ground truth.
  
  
  CONCLUSIONS
  The authors found that the community annotators achieved comparable inter-annotator agreement to expert annotators, regardless of whether the experts annotated from the pool. Furthermore, the ground truth generated by the community obtained F-measures above 0.90 against the ground truth of the experts, indicating the value of the community as a source of high-quality ground truth even on intricate and domain-specific annotation tasks.
authors.affiliation: Department of Information Studies, University at Albany, State University of New York, Albany, NY, USA. ouzuner@albany.edu
authors.forename: Ozlem
authors.initials: O
authors.lastname: Uzuner
authors.forename: Imre
authors.initials: I
authors.lastname: Solti
authors.forename: Fei
authors.initials: F
authors.lastname: Xia
authors.forename: Eithon
authors.initials: E
authors.lastname: Cadag
mesh_terms: D057286:Electronic Health Records; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*; D009323:Natural Language Processing; D010351:Patient Discharge; D004364:Pharmaceutical Preparations
chemical_list: D004364:Pharmaceutical Preparations
references.citation: J Am Med Inform Assoc. 2009 Jul-Aug;16(4):561-70
references.pmid: 19390096
references.citation: Proc AMIA Symp. 1998;:855-9
references.pmid: 9929340
references.citation: J Am Med Inform Assoc. 2005 May-Jun;12(3):296-8
references.pmid: 15684123
references.citation: J Am Med Inform Assoc. 2008 Jan-Feb;15(1):14-24
references.pmid: 17947624
references.citation: BMC Bioinformatics. 2005;6 Suppl 1:S3
references.pmid: 15960837
references.citation: J Am Med Inform Assoc. 2007 Sep-Oct;14(5):550-63
references.pmid: 17600094
references.citation: J Am Med Inform Assoc. 2010 Sep-Oct;17(5):514-8
references.pmid: 20819854