title: A natural language processing pipeline for pairing measurements uniquely across free-text CT reports.
abstract: OBJECTIVE
  To standardize and objectivize treatment response assessment in oncology, guidelines have been proposed that are driven by radiological measurements, which are typically communicated in free-text reports defying automated processing. We study through inter-annotator agreement and natural language processing (NLP) algorithm development the task of pairing measurements that quantify the same finding across consecutive radiology reports, such that each measurement is paired with at most one other ("partial uniqueness").
  
  
  METHODS
  Ground truth is created based on 283 abdomen and 311 chest CT reports of 50 patients each. A pre-processing engine segments reports and extracts measurements. Thirteen features are developed based on volumetric similarity between measurements, semantic similarity between their respective narrative contexts and structural properties of their report positions. A Random Forest classifier (RF) integrates all features. A "mutual best match" (MBM) post-processor ensures partial uniqueness.
  
  
  RESULTS
  In an end-to-end evaluation, RF has precision 0.841, recall 0.807, F-measure 0.824 and AUC 0.971; with MBM, which performs above chance level (P<0.001), it has precision 0.899, recall 0.776, F-measure 0.833 and AUC 0.935. RF (RF+MBM) has error-free performance on 52.7% (57.4%) of report pairs.
  
  
  CONCLUSIONS
  Inter-annotator agreement of three domain specialists with the ground truth (κ>0.960) indicates that the task is well defined. Domain properties and inter-section differences are discussed to explain superior performance in abdomen. Enforcing partial uniqueness has mixed but minor effects on performance.
  
  
  CONCLUSIONS
  A combined machine learning-filtering approach is proposed for pairing measurements, which can support prospective (supporting treatment response assessment) and retrospective purposes (data mining).
authors.affiliation: Clinical Informatics, Interventional & Translational Solutions, Philips Research North America, 345 Scarborough Road, Briarcliff Manor, NY, USA. Electronic address: Merlijn.sevenster@philips.com.
authors.forename: Merlijn
authors.initials: M
authors.lastname: Sevenster
authors.affiliation: Department of Medicine, University of Chicago, Chicago, IL, USA.
authors.forename: Jeffrey
authors.initials: J
authors.lastname: Bozeman
authors.affiliation: Department of Medicine, University of Chicago, Chicago, IL, USA.
authors.forename: Andrea
authors.initials: A
authors.lastname: Cowhy
authors.affiliation: Department of Medicine, University of Chicago, Chicago, IL, USA.
authors.forename: William
authors.initials: W
authors.lastname: Trost
mesh_terms: D000465:Algorithms; D019540:Area Under Curve; D019295:Computational Biology / Q000379:methods*; D057225:Data Mining / Q000379:methods; D006801:Humans; D000069550:Machine Learning; D008495:Medical Oncology; D009323:Natural Language Processing; D012372:ROC Curve; D011857:Radiographic Image Interpretation, Computer-Assisted; D011860:Radiography, Abdominal; D011871:Radiology; D011873:Radiology Information Systems; D015203:Reproducibility of Results; D012984:Software; D014057:Tomography, X-Ray Computed
keywords: Information correlation; Natural language processing; Oncologic measurement; RECIST; Radiology report