title: Development and evaluation of RapTAT: a machine learning system for concept mapping of phrases from medical narratives.
abstract: Rapid, automated determination of the mapping of free text phrases to pre-defined concepts could assist in the annotation of clinical notes and increase the speed of natural language processing systems. The aim of this study was to design and evaluate a token-order-specific naïve Bayes-based machine learning system (RapTAT) to predict associations between phrases and concepts. Performance was assessed using a reference standard generated from 2860 VA discharge summaries containing 567,520 phrases that had been mapped to 12,056 distinct Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) concepts by the MCVS natural language processing system. It was also assessed on the manually annotated, 2010 i2b2 challenge data. Performance was established with regard to precision, recall, and F-measure for each of the concepts within the VA documents using bootstrapping. Within that corpus, concepts identified by MCVS were broadly distributed throughout SNOMED CT, and the token-order-specific language model achieved better performance based on precision, recall, and F-measure (0.95±0.15, 0.96±0.16, and 0.95±0.16, respectively; mean±SD) than the bag-of-words based, naïve Bayes model (0.64±0.45, 0.61±0.46, and 0.60±0.45, respectively) that has previously been used for concept mapping. Precision, recall, and F-measure on the i2b2 test set were 92.9%, 85.9%, and 89.2% respectively, using the token-order-specific model. RapTAT required just 7.2ms to map all phrases within a single discharge summary, and mapping rate did not decrease as the number of processed documents increased. The high performance attained by the tool in terms of both accuracy and speed was encouraging, and the mapping rate should be sufficient to support near-real-time, interactive annotation of medical narratives. These results demonstrate the feasibility of rapidly and accurately mapping phrases to a wide range of medical concepts based on a token-order-specific naïve Bayes model and machine learning.
authors: Geriatric Research, Education and Clinical Center (GRECC), Department of Veterans Affairs Medical Center, Nashville, TN, USA; Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA; Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: glenn.t.gobbel@vanderbilt.edu.
authors: Glenn T
authors: GT
authors: Gobbel
authors: Geriatric Research, Education and Clinical Center (GRECC), Department of Veterans Affairs Medical Center, Nashville, TN, USA; Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: ruth.reeves2@va.gov.
authors: Ruth
authors: R
authors: Reeves
authors: Geriatric Research, Education and Clinical Center (GRECC), Department of Veterans Affairs Medical Center, Nashville, TN, USA; Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: shrimalini.jayaramaraja@vanderbilt.edu.
authors: Shrimalini
authors: S
authors: Jayaramaraja
authors: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: dario.giuse@vanderbilt.edu.
authors: Dario
authors: D
authors: Giuse
authors: Geriatric Research, Education and Clinical Center (GRECC), Department of Veterans Affairs Medical Center, Nashville, TN, USA; Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, USA; Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: ted.speroff@vanderbilt.edu.
authors: Theodore
authors: T
authors: Speroff
authors: Geriatric Research, Education and Clinical Center (GRECC), Department of Veterans Affairs Medical Center, Nashville, TN, USA; Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: steven.brown@va.gov.
authors: Steven H
authors: SH
authors: Brown
authors: Department of Biomedical Informatics, University at Buffalo, SUNY, Buffalo, NY, USA. Electronic address: ontolimatics@gmail.com.
authors: Peter L
authors: PL
authors: Elkin
authors: Geriatric Research, Education and Clinical Center (GRECC), Department of Veterans Affairs Medical Center, Nashville, TN, USA; Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA; Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, USA; Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, TN, USA. Electronic address: michael@matheny.info.
authors: Michael E
authors: ME
authors: Matheny
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D001331:Automation; D001499:Bayes Theorem; D016208:Databases, Factual; D057286:Electronic Health Records; D006787:Hospitals, Veterans; D006801:Humans; D015233:Models, Statistical; D009323:Natural Language Processing; D015203:Reproducibility of Results; D012984:Software; D039061:Systematized Nomenclature of Medicine; D013714:Tennessee; D009626:Terminology as Topic; D017432:Unified Medical Language System; D018875:Vocabulary, Controlled
keywords: Bayesian prediction; CSV; FN; FP; IQV; MCVS; Machine learning; Multi-threaded Clinical Vocabulary Server; NLP; Natural language processing; Opt; Perf; RapTAT; Rapid Text Annotation Tool; SNOMED-CT; SVM; Systematized Nomenclature of Medicine-Clinical Terms; Systematized nomenclature of medicine; TP; UMLS; Unified Medical Language System; comma-separated value; false negative; false positive; index of qualitative variation; natural language processing; optimism; performance; support vector machine; true positive