title: Active learning: a step towards automating medical concept extraction.
abstract: OBJECTIVE
  This paper presents an automatic, active learning-based system for the extraction of medical concepts from clinical free-text reports. Specifically, (1) the contribution of active learning in reducing the annotation effort and (2) the robustness of incremental active learning framework across different selection criteria and data sets are determined.
  
  
  METHODS
  The comparative performance of an active learning framework and a fully supervised approach were investigated to study how active learning reduces the annotation effort while achieving the same effectiveness as a supervised approach. Conditional random fields as the supervised method, and least confidence and information density as 2 selection criteria for active learning framework were used. The effect of incremental learning vs standard learning on the robustness of the models within the active learning framework with different selection criteria was also investigated. The following 2 clinical data sets were used for evaluation: the Informatics for Integrating Biology and the Bedside/Veteran Affairs (i2b2/VA) 2010 natural language processing challenge and the Shared Annotated Resources/Conference and Labs of the Evaluation Forum (ShARe/CLEF) 2013 eHealth Evaluation Lab.
  
  
  RESULTS
  The annotation effort saved by active learning to achieve the same effectiveness as supervised learning is up to 77%, 57%, and 46% of the total number of sequences, tokens, and concepts, respectively. Compared with the random sampling baseline, the saving is at least doubled.
  
  
  CONCLUSIONS
  Incremental active learning is a promising approach for building effective and robust medical concept extraction models while significantly reducing the burden of manual annotation.
authors.affiliation: Science and Engineering Faculty, Queensland University of Technology, Brisbane 4000, Queensland, Australia. The Australian e-Health Research Centre, CSIRO, Brisbane 4029, Queensland, Australia m1.kholghi@qut.edu.au.
authors.forename: Mahnoosh
authors.initials: M
authors.lastname: Kholghi
authors.affiliation: Science and Engineering Faculty, Queensland University of Technology, Brisbane 4000, Queensland, Australia.
authors.forename: Laurianne
authors.initials: L
authors.lastname: Sitbon
authors.affiliation: Science and Engineering Faculty, Queensland University of Technology, Brisbane 4000, Queensland, Australia.
authors.forename: Guido
authors.initials: G
authors.lastname: Zuccon
authors.affiliation: The Australian e-Health Research Centre, CSIRO, Brisbane 4029, Queensland, Australia.
authors.forename: Anthony
authors.initials: A
authors.lastname: Nguyen
mesh_terms: D000465:Algorithms; D057286:Electronic Health Records; D016247:Information Storage and Retrieval / Q000379:methods*; D000069550:Machine Learning; D018794:Problem-Based Learning; D012660:Semantics; D018875:Vocabulary, Controlled
keywords: active learning; clinical free text; conditional random fields; medical concept extraction; robustness analysis