Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: validation study.
OBJECTIVE
To validate the accuracy of Lexicon Mediated Entropy Reduction (LEXIMER), a new information theory-based computer algorithm developed by the authors for independent analysis and classification of unstructured radiology reports based on the presence of clinically important findings (F(T), where (T) represents "true") and recommendations for subsequent action (R(T)).


METHODS
The study was approved by the Human Research Committee of the institutional review board. Consecutive de-identified radiology reports (n = 1059) comprising results of barium studies (n = 99), computed tomography (n = 107), mammography (n = 90), magnetic resonance imaging (n = 108), nuclear medicine (n = 99), positron emission tomography (n = 106), radiography (n = 212), ultrasonography (n = 131), and vascular procedures (n = 107) were independently analyzed by two radiologists and then with LEXIMER to categorize the reports into F(T) and F(T)0 (containing or not containing clinically important findings) categories and R(T) and R(T)0 (containing or not containing recommendations for subsequent action) categories. Accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for placing reports into F(T) and F(T)0 and R(T) and R(T)0 categories were assessed by using appropriate statistical tests.


RESULTS
There was strong interobserver concordance between the two radiologists for placing radiology reports into F(T) and R(T) categories (kappa = 0.9, P < .01). For the LEXIMER program, accuracy, sensitivity, specificity, and positive and negative predictive values, respectively, were 97.5% (95% confidence interval [CI]: 96.6%, 98.5%), 98.9% (95% CI: 97.9%, 99.6%), 94.9% (95% CI: 93.1%, 96.0%), 97.5% (95% CI: 96.6%, 98.0%), and 97.7% (95% CI: 95.8%, 98.8%) for placing radiology reports into F(T) and F(T)0 categories and 99.6% (95% CI: 99.2%, 99.9%), 98.2% (95% CI: 95.0%, 99.6%), 99.9% (95% CI: 99.4%, 99.99%), 99.4% (95% CI: 96.3%, 99.9%), and 99.7% (95% CI: 98.9%, 99.9%) for placing reports into R(T) and R(T)0 categories.


CONCLUSIONS
LEXIMER is an accurate automated engine for evaluating the percentage positivity of clinically important findings and rates of recommendation for subsequent action in unstructured radiology reports.
Dreyer
Keith J
KJ
Division of Computing and Information Services, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 100 Charles River Plaza, Suite 471, Cambridge St, Boston, MA 02114, USA. kdreyer@partners.org
Kalra
Mannudeep K
MK
Maher
Michael M
MM
Hurier
Autumn M
AM
Asfaw
Benjamin A
BA
Schultz
Thomas
T
Halpern
Elkan F
EF
Thrall
James H
JH
D000465:Algorithms; D003658:Decision Making, Computer-Assisted; D005188:False Negative Reactions; D005189:False Positive Reactions; D006801:Humans; D008279:Magnetic Resonance Imaging; D008327:Mammography; D015588:Observer Variation; D011873:Radiology Information Systems; D011877:Radionuclide Imaging; D012680:Sensitivity and Specificity; D015899:Tomography, Emission-Computed, Single-Photon; D014057:Tomography, X-Ray Computed; D014463:Ultrasonography