title: Assisted annotation of medical free text using RapTAT.
abstract: OBJECTIVE
  To determine whether assisted annotation using interactive training can reduce the time required to annotate a clinical document corpus without introducing bias.
  
  
  METHODS
  A tool, RapTAT, was designed to assist annotation by iteratively pre-annotating probable phrases of interest within a document, presenting the annotations to a reviewer for correction, and then using the corrected annotations for further machine learning-based training before pre-annotating subsequent documents. Annotators reviewed 404 clinical notes either manually or using RapTAT assistance for concepts related to quality of care during heart failure treatment. Notes were divided into 20 batches of 19-21 documents for iterative annotation and training.
  
  
  RESULTS
  The number of correct RapTAT pre-annotations increased significantly and annotation time per batch decreased by ~50% over the course of annotation. Annotation rate increased from batch to batch for assisted but not manual reviewers. Pre-annotation F-measure increased from 0.5 to 0.6 to >0.80 (relative to both assisted reviewer and reference annotations) over the first three batches and more slowly thereafter. Overall inter-annotator agreement was significantly higher between RapTAT-assisted reviewers (0.89) than between manual reviewers (0.85).
  
  
  CONCLUSIONS
  The tool reduced workload by decreasing the number of annotations needing to be added and helping reviewers to annotate at an increased rate. Agreement between the pre-annotations and reference standard, and agreement between the pre-annotations and assisted annotations, were similar throughout the annotation process, which suggests that pre-annotation did not introduce bias.
  
  
  CONCLUSIONS
  Pre-annotations generated by a tool capable of interactive training can reduce the time required to create an annotated document corpus by up to 50%.
authors.affiliation: Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Glenn T
authors.initials: GT
authors.lastname: Gobbel
authors.affiliation: IDEAS Center SLC VA Healthcare System, Salt Lake City, Utah, USA Division of Epidemiology, University of Utah School of Medicine, Salt Lake City, Utah, USA Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, Utah, USA Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Salt Lake City, Utah, USA.
authors.forename: Jennifer
authors.initials: J
authors.lastname: Garvin
authors.affiliation: Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Ruth
authors.initials: R
authors.lastname: Reeves
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Robert M
authors.initials: RM
authors.lastname: Cronin
authors.affiliation: IDEAS Center SLC VA Healthcare System, Salt Lake City, Utah, USA.
authors.forename: Julia
authors.initials: J
authors.lastname: Heavirland
authors.affiliation: IDEAS Center SLC VA Healthcare System, Salt Lake City, Utah, USA.
authors.forename: Jenifer
authors.initials: J
authors.lastname: Williams
authors.affiliation: IDEAS Center SLC VA Healthcare System, Salt Lake City, Utah, USA.
authors.forename: Allison
authors.initials: A
authors.lastname: Weaver
authors.affiliation: Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Shrimalini
authors.initials: S
authors.lastname: Jayaramaraja
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Dario
authors.initials: D
authors.lastname: Giuse
authors.affiliation: Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Theodore
authors.initials: T
authors.lastname: Speroff
authors.affiliation: Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Steven H
authors.initials: SH
authors.lastname: Brown
authors.affiliation: School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA.
authors.forename: Hua
authors.identifier: http://orcid.org/0000-0001-9730-7276
authors.initials: H
authors.lastname: Xu
authors.affiliation: Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
authors.forename: Michael E
authors.initials: ME
authors.lastname: Matheny
mesh_terms: D001185:Artificial Intelligence; D057286:Electronic Health Records; D006333:Heart Failure / Q000188:drug therapy / Q000503:physiopathology; D006801:Humans; D009323:Natural Language Processing
keywords: Clinical Informatics; Guideline Adherence; Heart Failure; Medical Informatics Computing; Natural Language Processing