title: Developing a natural language processing application for measuring the quality of colonoscopy procedures.
abstract: OBJECTIVE
  The quality of colonoscopy procedures for colorectal cancer screening is often inadequate and varies widely among physicians. Routine measurement of quality is limited by the costs of manual review of free-text patient charts. Our goal was to develop a natural language processing (NLP) application to measure colonoscopy quality.
  
  
  METHODS
  Using a set of quality measures published by physician specialty societies, we implemented an NLP engine that extracts 21 variables for 19 quality measures from free-text colonoscopy and pathology reports. We evaluated the performance of the NLP engine on a test set of 453 colonoscopy reports and 226 pathology reports, considering accuracy in extracting the values of the target variables from text, and the reliability of the outcomes of the quality measures as computed from the NLP-extracted information.
  
  
  RESULTS
  The average accuracy of the NLP engine over all variables was 0.89 (range: 0.62-1.0) and the average F measure over all variables was 0.74 (range: 0.49-0.89). The average agreement score, measured as Cohen's κ, between the manually established and NLP-derived outcomes of the quality measures was 0.62 (range: 0.09-0.86).
  
  
  CONCLUSIONS
  For nine of the 19 colonoscopy quality measures, the agreement score was 0.70 or above, which we consider a sufficient score for the NLP-derived outcomes of these measures to be practically useful for quality measurement.
  
  
  CONCLUSIONS
  The use of NLP for information extraction from free-text colonoscopy and pathology reports creates opportunities for large scale, routine quality measurement, which can support quality improvement in colonoscopy care.
authors.affiliation: Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania 15260, USA. heh23@pitt.edu
authors.forename: Henk
authors.initials: H
authors.lastname: Harkema
authors.forename: Wendy W
authors.initials: WW
authors.lastname: Chapman
authors.forename: Melissa
authors.initials: M
authors.lastname: Saul
authors.forename: Evan S
authors.initials: ES
authors.lastname: Dellon
authors.forename: Robert E
authors.initials: RE
authors.lastname: Schoen
authors.forename: Ateev
authors.initials: A
authors.lastname: Mehrotra
mesh_terms: D003113:Colonoscopy / Q000592:standards*; D003199:Computer Systems; D057225:Data Mining; D006801:Humans; D009323:Natural Language Processing; D011787:Quality of Health Care