title: Large-scale biomedical concept recognition: an evaluation of current automatic annotators and their parameters.
abstract: BACKGROUND
  Ontological concepts are useful for many different biomedical tasks. Concepts are difficult to recognize in text due to a disconnect between what is captured in an ontology and how the concepts are expressed in text. There are many recognizers for specific ontologies, but a general approach for concept recognition is an open problem.
  
  
  RESULTS
  Three dictionary-based systems (MetaMap, NCBO Annotator, and ConceptMapper) are evaluated on eight biomedical ontologies in the Colorado Richly Annotated Full-Text (CRAFT) Corpus. Over 1,000 parameter combinations are examined, and best-performing parameters for each system-ontology pair are presented.
  
  
  CONCLUSIONS
  Baselines for concept recognition by three systems on eight biomedical ontologies are established (F-measures range from 0.14-0.83). Out of the three systems we tested, ConceptMapper is generally the best-performing system; it produces the highest F-measure of seven out of eight ontologies. Default parameters are not ideal for most systems on most ontologies; by changing parameters F-measure can be increased by up to 0.4. Not only are best performing parameters presented, but suggestions for choosing the best parameters based on ontology characteristics are presented.
authors.affiliation: Computational Bioscience Program, U, of Colorado School of Medicine, Aurora, CO 80045, USA. christopher.funk@ucdenver.edu.
authors.forename: Christopher
authors.initials: C
authors.lastname: Funk
authors.forename: William
authors.initials: W
authors.lastname: Baumgartner
authors.forename: Benjamin
authors.initials: B
authors.lastname: Garcia
authors.forename: Christophe
authors.initials: C
authors.lastname: Roeder
authors.forename: Michael
authors.initials: M
authors.lastname: Bada
authors.forename: K Bretonnel
authors.initials: KB
authors.lastname: Cohen
authors.forename: Lawrence E
authors.initials: LE
authors.lastname: Hunter
authors.forename: Karin
authors.initials: K
authors.lastname: Verspoor
mesh_terms: D064229:Biological Ontologies; D057225:Data Mining / Q000379:methods*; D016208:Databases, Factual; D015203:Reproducibility of Results