title: Assessing Inaccuracies in Automated Information Extraction of Breast Imaging Findings.
abstract: We previously identified breast imaging findings from radiology reports using an expert-based information extraction algorithm as part of the National Cancer Institute's Population-based Research Optimizing Screening through Personalized Regimens (PROSPR) initiative. We validate this algorithm and assess inaccuracies in a different institutional setting. Mammography, ultrasound (US), and breast magnetic resonance imaging (MRI) reports of patients at an academic health system between 4/2013 and 6/2013 were included for analysis. Accuracy of automatically extracting imaging findings using an algorithm developed at a different institution compared to manual gold standard review is reported. Extraction errors are further categorized based on manual review. Precision and recall for extracting BI-RADS categories remain between 0.9 and 1.0, except for MRI (0.7). F measures for extracting other findings are 0.9 for non-mass enhancement (in MRI) and 0.8-0.9 for cysts (in MRI and US). Extracting breast imaging findings resulted in lowest accuracy for findings of calcification (range 0.4-0.6 in mammography) and asymmetric density (0.5-0.7 in mammography). Majority of errors for extracting imaging findings were due to qualifier-based errors, descriptors which indicate absence of findings, missed by automated extraction (e.g., "benign" calcifications). Our information extraction algorithm provides an effective approach to extracting some breast imaging findings for populating a breast screening registry. However, errors in information extraction when utilizing methods in new settings demonstrate that further work is necessary to extract information content from unstructured multi-institutional radiology reports.
authors.affiliation: Department of Radiology, Brigham and Women's Hospital, 75 Francis Street, Boston, MA, 02115, USA. rlacson@partners.org.|Harvard Medical School, Boston, MA, USA. rlacson@partners.org.
authors.forename: Ronilda
authors.initials: R
authors.lastname: Lacson
authors.affiliation: Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Lebanon, NH, USA.
authors.forename: Martha E
authors.initials: ME
authors.lastname: Goodrich
authors.affiliation: Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA, USA.
authors.forename: Kimberly
authors.initials: K
authors.lastname: Harris
authors.affiliation: Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA, USA.
authors.forename: Phyllis
authors.initials: P
authors.lastname: Brawarsky
authors.affiliation: Harvard Medical School, Boston, MA, USA.|Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA, USA.
authors.forename: Jennifer S
authors.initials: JS
authors.lastname: Haas
mesh_terms: D000465:Algorithms; D001940:Breast / Q000000981:diagnostic imaging*; D001943:Breast Neoplasms; D005260:Female; D006801:Humans; D016247:Information Storage and Retrieval; D008279:Magnetic Resonance Imaging; D008327:Mammography; D012015:Reference Standards; D016217:Ultrasonography, Mammary
keywords: Breast neoplasm; Information storage and retrieval; Magnetic resonance imaging; Mammography; Radiology reporting; Ultrasonography