Evaluation of training with an annotation schema for manual annotation of clinical conditions from emergency department reports.
OBJECTIVE
Determine whether agreement among annotators improves after being trained to use an annotation schema that specifies: what types of clinical conditions to annotate, the linguistic form of the annotations, and which modifiers to include.


METHODS
Three physicians and 3 lay people individually annotated all clinical conditions in 23 emergency department reports. For annotations made using a Baseline Schema and annotations made after training on a detailed annotation schema, we compared: (1) variability of annotation length and number and (2) annotator agreement, using the F-measure.


RESULTS
Physicians showed higher agreement and lower variability after training on the detailed annotation schema than when applying the Baseline Schema. Lay people agreed with physicians almost as well as other physicians did but showed a slower learning curve.


CONCLUSIONS
Training annotators on the annotation schema we developed increased agreement among annotators and should be useful in generating reference standard sets for natural language processing studies. The methodology we used to evaluate the schema could be applied to other types of annotation or classification tasks in biomedical informatics.
Chapman
Wendy W
WW
Department of Biomedical Informatics, University of Pittsburgh, 200 Meyran Avenue, M-183 VALE, Pittsburgh, PA 15260, United States. chapman@cbmi.pitt.edu
Dowling
John N
JN
Hripcsak
George
G
D001185:Artificial Intelligence; D004282:Documentation / Q000379:methods / Q000592:standards*; D004636:Emergency Service, Hospital / Q000458:organization & administration*; D016519:Hospitals, Religious; D006801:Humans; D016347:Medical Records Systems, Computerized; D009323:Natural Language Processing; D010414:Pennsylvania; D013663:Teaching