title: A study of active learning methods for named entity recognition in clinical text.
abstract: OBJECTIVE
  Named entity recognition (NER), a sequential labeling task, is one of the fundamental tasks for building clinical natural language processing (NLP) systems. Machine learning (ML) based approaches can achieve good performance, but they often require large amounts of annotated samples, which are expensive to build due to the requirement of domain experts in annotation. Active learning (AL), a sample selection approach integrated with supervised ML, aims to minimize the annotation cost while maximizing the performance of ML-based models. In this study, our goal was to develop and evaluate both existing and new AL methods for a clinical NER task to identify concepts of medical problems, treatments, and lab tests from the clinical notes.
  
  
  METHODS
  Using the annotated NER corpus from the 2010 i2b2/VA NLP challenge that contained 349 clinical documents with 20,423 unique sentences, we simulated AL experiments using a number of existing and novel algorithms in three different categories including uncertainty-based, diversity-based, and baseline sampling strategies. They were compared with the passive learning that uses random sampling. Learning curves that plot performance of the NER model against the estimated annotation cost (based on number of sentences or words in the training set) were generated to evaluate different active learning and the passive learning methods and the area under the learning curve (ALC) score was computed.
  
  
  RESULTS
  Based on the learning curves of F-measure vs. number of sentences, uncertainty sampling algorithms outperformed all other methods in ALC. Most diversity-based methods also performed better than random sampling in ALC. To achieve an F-measure of 0.80, the best method based on uncertainty sampling could save 66% annotations in sentences, as compared to random sampling. For the learning curves of F-measure vs. number of words, uncertainty sampling methods again outperformed all other methods in ALC. To achieve 0.80 in F-measure, in comparison to random sampling, the best uncertainty based method saved 42% annotations in words. But the best diversity based method reduced only 7% annotation effort.
  
  
  CONCLUSIONS
  In the simulated setting, AL methods, particularly uncertainty-sampling based approaches, seemed to significantly save annotation cost for the clinical NER task. The actual benefit of active learning in clinical NER should be further evaluated in a real-time setting.
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University, School of Medicine, Nashville, TN, USA.
authors.forename: Yukun
authors.initials: Y
authors.lastname: Chen
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University, School of Medicine, Nashville, TN, USA.
authors.forename: Thomas A
authors.initials: TA
authors.lastname: Lasko
authors.affiliation: School of Information, University of Michigan, Ann Arbor, MI, USA; Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA.
authors.forename: Qiaozhu
authors.initials: Q
authors.lastname: Mei
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University, School of Medicine, Nashville, TN, USA; Department of Medicine, Vanderbilt University, School of Medicine, Nashville, TN, USA.
authors.forename: Joshua C
authors.initials: JC
authors.lastname: Denny
authors.affiliation: School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA; Department of Biomedical Informatics, Vanderbilt University, School of Medicine, Nashville, TN, USA. Electronic address: hua.xu@uth.tmc.edu.
authors.forename: Hua
authors.initials: H
authors.lastname: Xu
mesh_terms: D006801:Humans; D007858:Learning; D000069550:Machine Learning; D009323:Natural Language Processing
keywords: Active learning; Clinical named entity recognition; Clinical natural language processing; Machine learning