title: Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: validation study.
abstract: OBJECTIVE
  To validate the accuracy of Lexicon Mediated Entropy Reduction (LEXIMER), a new information theory-based computer algorithm developed by the authors for independent analysis and classification of unstructured radiology reports based on the presence of clinically important findings (F(T), where (T) represents "true") and recommendations for subsequent action (R(T)).
  
  
  METHODS
  The study was approved by the Human Research Committee of the institutional review board. Consecutive de-identified radiology reports (n = 1059) comprising results of barium studies (n = 99), computed tomography (n = 107), mammography (n = 90), magnetic resonance imaging (n = 108), nuclear medicine (n = 99), positron emission tomography (n = 106), radiography (n = 212), ultrasonography (n = 131), and vascular procedures (n = 107) were independently analyzed by two radiologists and then with LEXIMER to categorize the reports into F(T) and F(T)0 (containing or not containing clinically important findings) categories and R(T) and R(T)0 (containing or not containing recommendations for subsequent action) categories. Accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for placing reports into F(T) and F(T)0 and R(T) and R(T)0 categories were assessed by using appropriate statistical tests.
  
  
  RESULTS
  There was strong interobserver concordance between the two radiologists for placing radiology reports into F(T) and R(T) categories (kappa = 0.9, P < .01). For the LEXIMER program, accuracy, sensitivity, specificity, and positive and negative predictive values, respectively, were 97.5% (95% confidence interval [CI]: 96.6%, 98.5%), 98.9% (95% CI: 97.9%, 99.6%), 94.9% (95% CI: 93.1%, 96.0%), 97.5% (95% CI: 96.6%, 98.0%), and 97.7% (95% CI: 95.8%, 98.8%) for placing radiology reports into F(T) and F(T)0 categories and 99.6% (95% CI: 99.2%, 99.9%), 98.2% (95% CI: 95.0%, 99.6%), 99.9% (95% CI: 99.4%, 99.99%), 99.4% (95% CI: 96.3%, 99.9%), and 99.7% (95% CI: 98.9%, 99.9%) for placing reports into R(T) and R(T)0 categories.
  
  
  CONCLUSIONS
  LEXIMER is an accurate automated engine for evaluating the percentage positivity of clinically important findings and rates of recommendation for subsequent action in unstructured radiology reports.
authors.affiliation: Division of Computing and Information Services, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 100 Charles River Plaza, Suite 471, Cambridge St, Boston, MA 02114, USA. kdreyer@partners.org
authors.forename: Keith J
authors.initials: KJ
authors.lastname: Dreyer
authors.forename: Mannudeep K
authors.initials: MK
authors.lastname: Kalra
authors.forename: Michael M
authors.initials: MM
authors.lastname: Maher
authors.forename: Autumn M
authors.initials: AM
authors.lastname: Hurier
authors.forename: Benjamin A
authors.initials: BA
authors.lastname: Asfaw
authors.forename: Thomas
authors.initials: T
authors.lastname: Schultz
authors.forename: Elkan F
authors.initials: EF
authors.lastname: Halpern
authors.forename: James H
authors.initials: JH
authors.lastname: Thrall
mesh_terms: D000465:Algorithms; D003658:Decision Making, Computer-Assisted; D005188:False Negative Reactions; D005189:False Positive Reactions; D006801:Humans; D008279:Magnetic Resonance Imaging; D008327:Mammography; D015588:Observer Variation; D011873:Radiology Information Systems; D011877:Radionuclide Imaging; D012680:Sensitivity and Specificity; D015899:Tomography, Emission-Computed, Single-Photon; D014057:Tomography, X-Ray Computed; D014463:Ultrasonography