title: Creation of a simple natural language processing tool to support an imaging utilization quality dashboard.
abstract: Testing for venous thromboembolism (VTE) is associated with cost and risk to patients (e.g. radiation). To assess the appropriateness of imaging utilization at the provider level, it is important to know that provider's diagnostic yield (percentage of tests positive for the diagnostic entity of interest). However, determining diagnostic yield typically requires either time-consuming, manual review of radiology reports or the use of complex and/or proprietary natural language processing software.
  
  
  
  The objectives of this study were twofold: 1) to develop and implement a simple, user-configurable, and open-source natural language processing tool to classify radiology reports with high accuracy and 2) to use the results of the tool to design a provider-specific VTE imaging dashboard, consisting of both utilization rate and diagnostic yield.
  
  
  
  Two physicians reviewed a training set of 400 lower extremity ultrasound (UTZ) and computed tomography pulmonary angiogram (CTPA) reports to understand the language used in VTE-positive and VTE-negative reports. The insights from this review informed the arguments to the five modifiable parameters of the NLP tool. A validation set of 2,000 studies was then independently classified by the reviewers and by the tool; the classifications were compared and the performance of the tool was calculated.
  
  
  
  The tool was highly accurate in classifying the presence and absence of VTE for both the UTZ (sensitivity 95.7%; 95% CI 91.5-99.8, specificity 100%; 95% CI 100-100) and CTPA reports (sensitivity 97.1%; 95% CI 94.3-99.9, specificity 98.6%; 95% CI 97.8-99.4). The diagnostic yield was then calculated at the individual provider level and the imaging dashboard was created.
  
  
  
  We have created a novel NLP tool designed for users without a background in computer programming, which has been used to classify venous thromboembolism reports with a high degree of accuracy. The tool is open-source and available for download at http://iturrate.com/simpleNLP. Results obtained using this tool can be applied to enhance quality by presenting information about utilization and yield to providers via an imaging dashboard.
authors.affiliation: New York University School of Medicine, Ronald O. Perelman Department of Emergency Medicine, New York, NY, United States. Electronic address: Jordan.Swartz@nyumc.org.
authors.forename: Jordan
authors.initials: J
authors.lastname: Swartz
authors.affiliation: New York University School of Medicine, Ronald O. Perelman Department of Emergency Medicine, New York, NY, United States.
authors.forename: Christian
authors.initials: C
authors.lastname: Koziatek
authors.affiliation: Department of Emergency Medicine, Mount Sinai West Hospital, Mount Sinai St Luke's Hospital, New York, NY, United States.
authors.forename: Jason
authors.initials: J
authors.lastname: Theobald
authors.affiliation: New York University School of Medicine, Ronald O. Perelman Department of Emergency Medicine, New York, NY, United States.
authors.forename: Silas
authors.initials: S
authors.lastname: Smith
authors.affiliation: New York University School of Medicine, Department of Internal Medicine, New York, NY, United States.
authors.forename: Eduardo
authors.initials: E
authors.lastname: Iturrate
mesh_terms: D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000592:standards*; D009323:Natural Language Processing; D014057:Tomography, X-Ray Computed / Q000379:methods*; D054556:Venous Thromboembolism / Q000000981:diagnostic imaging*
keywords: Automated text classification; Benchmarking; Deep venous thrombosis; Natural language processing; Practice variation; Pulmonary embolism