title: Building and evaluating annotated corpora for medical NLP systems.
abstract: We report efforts from a pilot annotation project in which we annotated sixty clinical notes with disorder mentions. Four retrieval experts annotated the same clinical notes so that four-way inter-annotator agreement could be calculated. We find the inter-annotator agreement results encouraging to scale up the project using the same guidelines and annotation schema.
authors.affiliation: Mayo Clinic, Rochester, MN, USA.
authors.forename: Philip V
authors.initials: PV
authors.lastname: Ogren
authors.forename: Guergana
authors.initials: G
authors.lastname: Savova
authors.forename: James D
authors.initials: JD
authors.lastname: Buntrock
authors.forename: Christopher G
authors.initials: CG
authors.lastname: Chute
mesh_terms: D006801:Humans; D008499:Medical Records; D009323:Natural Language Processing; D010865:Pilot Projects; D017432:Unified Medical Language System