title: Automated concept-level information extraction to reduce the need for custom software and rules development.
abstract: OBJECTIVE
  Despite at least 40 years of promising empirical performance, very few clinical natural language processing (NLP) or information extraction systems currently contribute to medical science or care. The authors address this gap by reducing the need for custom software and rules development with a graphical user interface-driven, highly generalizable approach to concept-level retrieval.
  
  
  METHODS
  A 'learn by example' approach combines features derived from open-source NLP pipelines with open-source machine learning classifiers to automatically and iteratively evaluate top-performing configurations. The Fourth i2b2/VA Shared Task Challenge's concept extraction task provided the data sets and metrics used to evaluate performance.
  
  
  RESULTS
  Top F-measure scores for each of the tasks were medical problems (0.83), treatments (0.82), and tests (0.83). Recall lagged precision in all experiments. Precision was near or above 0.90 in all tasks. Discussion With no customization for the tasks and less than 5&emsp14;min of end-user time to configure and launch each experiment, the average F-measure was 0.83, one point behind the mean F-measure of the 22 entrants in the competition. Strong precision scores indicate the potential of applying the approach for more specific clinical information extraction tasks. There was not one best configuration, supporting an iterative approach to model creation.
  
  
  CONCLUSIONS
  Acceptable levels of performance can be achieved using fully automated and generalizable approaches to concept-level information extraction. The described implementation and related documentation is available for download.
authors.affiliation: Massachusetts Veterans Epidemiology Research and Information Center Cooperative Studies Coordinating Center, VA Boston Healthcare System, Jamaica Plain, Massachusetts 02130, USA. leonard.davolio@va.gov
authors.forename: Leonard W
authors.initials: LW
authors.lastname: D'Avolio
authors.forename: Thien M
authors.initials: TM
authors.lastname: Nguyen
authors.forename: Sergey
authors.initials: S
authors.lastname: Goryachev
authors.forename: Louis D
authors.initials: LD
authors.lastname: Fiore
mesh_terms: D000465:Algorithms; D057225:Data Mining / Q000145:classification; D020000:Decision Support Systems, Clinical / Q000145:classification; D057286:Electronic Health Records / Q000145:classification; D006801:Humans; D009323:Natural Language Processing; D014584:User-Computer Interface