title: A multilingual gold-standard corpus for biomedical concept recognition: the Mantra GSC.
abstract: OBJECTIVE
  To create a multilingual gold-standard corpus for biomedical concept recognition.
  
  
  METHODS
  We selected text units from different parallel corpora (Medline abstract titles, drug labels, biomedical patent claims) in English, French, German, Spanish, and Dutch. Three annotators per language independently annotated the biomedical concepts, based on a subset of the Unified Medical Language System and covering a wide range of semantic groups. To reduce the annotation workload, automatically generated preannotations were provided. Individual annotations were automatically harmonized and then adjudicated, and cross-language consistency checks were carried out to arrive at the final annotations.
  
  
  RESULTS
  The number of final annotations was 5530. Inter-annotator agreement scores indicate good agreement (median F-score 0.79), and are similar to those between individual annotators and the gold standard. The automatically generated harmonized annotation set for each language performed equally well as the best annotator for that language.
  
  
  CONCLUSIONS
  The use of automatic preannotations, harmonized annotations, and parallel corpora helped to keep the manual annotation efforts manageable. The inter-annotator agreement scores provide a reference standard for gauging the performance of automatic annotation techniques.
  
  
  CONCLUSIONS
  To our knowledge, this is the first gold-standard corpus for biomedical concept recognition in languages other than English. Other distinguishing features are the wide variety of semantic groups that are being covered, and the diversity of text genres that were annotated.
authors.affiliation: Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands j.kors@erasmusmc.nl.
authors.forename: Jan A
authors.initials: JA
authors.lastname: Kors
authors.affiliation: Institute of Computational Linguistics, University of Zurich, Zurich, Switzerland.
authors.forename: Simon
authors.initials: S
authors.lastname: Clematide
authors.affiliation: Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands.
authors.forename: Saber A
authors.initials: SA
authors.lastname: Akhondi
authors.affiliation: Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands.
authors.forename: Erik M
authors.initials: EM
authors.lastname: van Mulligen
authors.affiliation: Institute of Computational Linguistics, University of Zurich, Zurich, Switzerland.
authors.forename: Dietrich
authors.initials: D
authors.lastname: Rebholz-Schuhmann
mesh_terms: D016247:Information Storage and Retrieval / Q000379:methods*; D019303:Multilingualism; D009323:Natural Language Processing; D012660:Semantics; D009626:Terminology as Topic; D017432:Unified Medical Language System
keywords: concept identification; gold-standard corpus; inter-annotator agreement; multilinguality; semantic enrichment