title: Evaluating the state of the art in disorder recognition and normalization of the clinical narrative.
abstract: OBJECTIVE
  The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners.
  
  
  METHODS
  We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text--199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance.
  
  
  RESULTS
  For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59.
  
  
  CONCLUSIONS
  Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources.
  
  
  CONCLUSIONS
  The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.
authors.affiliation: Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA.
authors.forename: Sameer
authors.initials: S
authors.lastname: Pradhan
authors.affiliation: Columbia University, New York, New York, USA.
authors.forename: Noémie
authors.initials: N
authors.lastname: Elhadad
authors.affiliation: University of Utah, Salt Lake City, Utah, USA.
authors.forename: Brett R
authors.initials: BR
authors.lastname: South
authors.affiliation: The University of Melbourne, Australia.
authors.forename: David
authors.initials: D
authors.lastname: Martinez
authors.affiliation: University of Utah, Salt Lake City, Utah, USA.
authors.forename: Lee
authors.initials: L
authors.lastname: Christensen
authors.affiliation: Columbia University, New York, New York, USA.
authors.forename: Amy
authors.initials: A
authors.lastname: Vogel
authors.affiliation: NICTA, The Australian National University, and University of Canberra, Canberra, Australian Capital Territory, Australia.
authors.forename: Hanna
authors.initials: H
authors.lastname: Suominen
authors.affiliation: University of Utah, Salt Lake City, Utah, USA.
authors.forename: Wendy W
authors.initials: WW
authors.lastname: Chapman
authors.affiliation: Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA.
authors.forename: Guergana
authors.initials: G
authors.lastname: Savova
mesh_terms: D064229:Biological Ontologies; D066264:Datasets as Topic; D004194:Disease; D057286:Electronic Health Records; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods; D009323:Natural Language Processing; D039061:Systematized Nomenclature of Medicine; D017432:Unified Medical Language System; D018875:Vocabulary, Controlled
keywords: Clinical Notes; Disorder Identifciation; Information Extraction; Named Entity Recognition; Natural Language Processing; Word Sense Disambiguation