Community annotation experiment for ground truth generation for the i2b2 medication challenge.
OBJECTIVE
Within the context of the Third i2b2 Workshop on Natural Language Processing Challenges for Clinical Records, the authors (also referred to as 'the i2b2 medication challenge team' or 'the i2b2 team' for short) organized a community annotation experiment.


METHODS
For this experiment, the authors released annotation guidelines and a small set of annotated discharge summaries. They asked the participants of the Third i2b2 Workshop to annotate 10 discharge summaries per person; each discharge summary was annotated by two annotators from two different teams, and a third annotator from a third team resolved disagreements.


METHODS
In order to evaluate the reliability of the annotations thus produced, the authors measured community inter-annotator agreement and compared it with the inter-annotator agreement of expert annotators when both the community and the expert annotators generated ground truth based on pooled system outputs. For this purpose, the pool consisted of the three most densely populated automatic annotations of each record. The authors also compared the community inter-annotator agreement with expert inter-annotator agreement when the experts annotated raw records without using the pool. Finally, they measured the quality of the community ground truth by comparing it with the expert ground truth.


CONCLUSIONS
The authors found that the community annotators achieved comparable inter-annotator agreement to expert annotators, regardless of whether the experts annotated from the pool. Furthermore, the ground truth generated by the community obtained F-measures above 0.90 against the ground truth of the experts, indicating the value of the community as a source of high-quality ground truth even on intricate and domain-specific annotation tasks.
Uzuner
Ozlem
O
Department of Information Studies, University at Albany, State University of New York, Albany, NY, USA. ouzuner@albany.edu
Solti
Imre
I
Xia
Fei
F
Cadag
Eithon
E
D057286:Electronic Health Records; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*; D009323:Natural Language Processing; D010351:Patient Discharge; D004364:Pharmaceutical Preparations
D004364:Pharmaceutical Preparations
J Am Med Inform Assoc. 2009 Jul-Aug;16(4):561-70
19390096
Proc AMIA Symp. 1998;:855-9
9929340
J Am Med Inform Assoc. 2005 May-Jun;12(3):296-8
15684123
J Am Med Inform Assoc. 2008 Jan-Feb;15(1):14-24
17947624
BMC Bioinformatics. 2005;6 Suppl 1:S3
15960837
J Am Med Inform Assoc. 2007 Sep-Oct;14(5):550-63
17600094
J Am Med Inform Assoc. 2010 Sep-Oct;17(5):514-8
20819854