title: Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements.
abstract: OBJECTIVE
  To present a series of experiments: (1) to evaluate the impact of pre-annotation on the speed of manual annotation of clinical trial announcements; and (2) to test for potential bias, if pre-annotation is utilized.
  
  
  METHODS
  To build the gold standard, 1400 clinical trial announcements from the clinicaltrials.gov website were randomly selected and double annotated for diagnoses, signs, symptoms, Unified Medical Language System (UMLS) Concept Unique Identifiers, and SNOMED CT codes. We used two dictionary-based methods to pre-annotate the text. We evaluated the annotation time and potential bias through F-measures and ANOVA tests and implemented Bonferroni correction.
  
  
  RESULTS
  Time savings ranged from 13.85% to 21.5% per entity. Inter-annotator agreement (IAA) ranged from 93.4% to 95.5%. There was no statistically significant difference for IAA and annotator performance in pre-annotations.
  
  
  CONCLUSIONS
  On every experiment pair, the annotator with the pre-annotated text needed less time to annotate than the annotator with non-labeled text. The time savings were statistically significant. Moreover, the pre-annotation did not reduce the IAA or annotator performance. Dictionary-based pre-annotation is a feasible and practical method to reduce the cost of annotation of clinical named entity recognition in the eligibility sections of clinical trial announcements without introducing bias in the annotation process.
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA.
authors.forename: Todd
authors.initials: T
authors.lastname: Lingren
authors.forename: Louise
authors.initials: L
authors.lastname: Deleger
authors.forename: Katalin
authors.initials: K
authors.lastname: Molnar
authors.forename: Haijun
authors.initials: H
authors.lastname: Zhai
authors.forename: Jareen
authors.initials: J
authors.lastname: Meinzen-Derr
authors.forename: Megan
authors.initials: M
authors.lastname: Kaiser
authors.forename: Laura
authors.initials: L
authors.lastname: Stoutenborough
authors.forename: Qi
authors.initials: Q
authors.lastname: Li
authors.forename: Imre
authors.initials: I
authors.lastname: Solti
mesh_terms: D000704:Analysis of Variance; D002986:Clinical Trials as Topic; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*; D009323:Natural Language Processing; D013996:Time and Motion Studies
keywords: Information Extraction; Natural Language Processing; Pre-annotation; clinical trial announcements; named entity recognition; umls