title: Validation of natural language processing to extract breast cancer pathology procedures and results.
abstract: BACKGROUND
  Pathology reports typically require manual review to abstract research data. We developed a natural language processing (NLP) system to automatically interpret free-text breast pathology reports with limited assistance from manual abstraction.
  
  
  METHODS
  We used an iterative approach of machine learning algorithms and constructed groups of related findings to identify breast-related procedures and results from free-text pathology reports. We evaluated the NLP system using an all-or-nothing approach to determine which reports could be processed entirely using NLP and which reports needed manual review beyond NLP. We divided 3234 reports for development (2910, 90%), and evaluation (324, 10%) purposes using manually reviewed pathology data as our gold standard.
  
  
  RESULTS
  NLP correctly coded 12.7% of the evaluation set, flagged 49.1% of reports for manual review, incorrectly coded 30.8%, and correctly omitted 7.4% from the evaluation set due to irrelevancy (i.e. not breast-related). Common procedures and results were identified correctly (e.g. invasive ductal with 95.5% precision and 94.0% sensitivity), but entire reports were flagged for manual review because of rare findings and substantial variation in pathology report text.
  
  
  CONCLUSIONS
  The NLP system we developed did not perform sufficiently for abstracting entire breast pathology reports. The all-or-nothing approach resulted in too broad of a scope of work and limited our flexibility to identify breast pathology procedures and results. Our NLP system was also limited by the lack of the gold standard data on rare findings and wide variation in pathology text. Focusing on individual, common elements and improving pathology text report standardization may improve performance.
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: Arika E
authors.initials: AE
authors.lastname: Wieneke
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: Erin J A
authors.initials: EJ
authors.lastname: Bowles
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: David
authors.initials: D
authors.lastname: Cronkite
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: Karen J
authors.initials: KJ
authors.lastname: Wernli
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: Hongyuan
authors.initials: H
authors.lastname: Gao
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: David
authors.initials: D
authors.lastname: Carrell
authors.affiliation: Group Health Research Institute, Seattle, WA, USA.
authors.forename: Diana S M
authors.initials: DS
authors.lastname: Buist
keywords: Breast cancer; natural language processing; pathology; validation