title: Natural Language Processing to identify pneumonia from radiology reports.
abstract: OBJECTIVE
  This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports.
  
  
  METHODS
  We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so, we classified reports as follows: (1) consistent with pneumonia; (2) inconsistent with pneumonia; or (3) requiring manual review because of complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data.
  
  
  RESULTS
  Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX's sensitivity was 92% (95% CI 90-93%), specificity 87% (86-88%), PPV 74% (72-76%), and NPV 96% (96-97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review. For the remainder, ONYX had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%).
  
  
  CONCLUSIONS
  For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings.
authors.affiliation: Group Health Research Institute, Seattle, WA 98101, USA. dublin.s@ghc.org
authors.forename: Sascha
authors.initials: S
authors.lastname: Dublin
authors.forename: Eric
authors.initials: E
authors.lastname: Baldwin
authors.forename: Rod L
authors.initials: RL
authors.lastname: Walker
authors.forename: Lee M
authors.initials: LM
authors.lastname: Christensen
authors.forename: Peter J
authors.initials: PJ
authors.lastname: Haug
authors.forename: Michael L
authors.initials: ML
authors.lastname: Jackson
authors.forename: Jennifer C
authors.initials: JC
authors.lastname: Nelson
authors.forename: Jeffrey
authors.initials: J
authors.lastname: Ferraro
authors.forename: David
authors.initials: D
authors.lastname: Carrell
authors.forename: Wendy W
authors.initials: WW
authors.lastname: Chapman
mesh_terms: D000293:Adolescent; D000328:Adult; D000367:Age Factors; D000368:Aged; D000369:Aged, 80 and over; D002648:Child; D002675:Child, Preschool; D006801:Humans; D007223:Infant; D016015:Logistic Models; D008875:Middle Aged; D009323:Natural Language Processing; D017891:Pharmacoepidemiology; D011014:Pneumonia / Q000175:diagnosis* / Q000000981:diagnostic imaging / Q000453:epidemiology; D011237:Predictive Value of Tests; D015995:Prevalence; D011859:Radiography; D055815:Young Adult
keywords: Natural Language Processing; pharmacoepidemiology; pneumonia; sensitivity; specificity; validity
references.citation: J Biomed Inform. 2009 Oct;42(5):839-51
references.pmid: 19435614
references.citation: Am J Med Qual. 2005 Nov-Dec;20(6):319-28
references.pmid: 16280395
references.citation: Pediatr Infect Dis J. 2006 Sep;25(9):779-81
references.pmid: 16940833
references.citation: Biometrics. 1982 Sep;38(3):613-21
references.pmid: 7171691
references.citation: J Biomed Inform. 2005 Aug;38(4):314-21
references.pmid: 16084473
references.citation: Lancet. 2008 Aug 2;372(9636):398-405
references.pmid: 18675690
references.citation: Chest. 1996 Aug;110(2):343-50
references.pmid: 8697831
references.citation: Am J Infect Control. 2008 Jun;36(5):309-32
references.pmid: 18538699
references.citation: JAMA. 2011 Aug 24;306(8):848-55
references.pmid: 21862746
references.citation: JAMA. 2009 May 27;301(20):2120-8
references.pmid: 19470989
references.citation: Biometrics. 1999 Jun;55(2):652-9
references.pmid: 11318229
references.citation: AMIA Annu Symp Proc. 2008 Nov 06;:172-6
references.pmid: 18998791
references.citation: Clin Infect Dis. 2007 Mar 1;44 Suppl 2:S27-72
references.pmid: 17278083
references.citation: JAMA. 2004 Oct 27;292(16):1955-60
references.pmid: 15507580
references.citation: Arch Intern Med. 2007 May 14;167(9):950-5
references.pmid: 17502537
references.citation: J Clin Epidemiol. 2007 Aug;60(8):834-8
references.pmid: 17606180
references.citation: J Am Geriatr Soc. 2011 Oct;59(10):1899-907
references.pmid: 22091503
references.citation: J Am Med Inform Assoc. 2000 Nov-Dec;7(6):593-604
references.pmid: 11062233
references.citation: Pharmacoepidemiol Drug Saf. 2012 Jun;21(6):651-8
references.pmid: 22271492
references.citation: Vaccine. 2008 Sep 8;26(38):4947-54
references.pmid: 18662735
references.citation: Ann Intern Med. 2010 Apr 6;152(7):418-25, W139-40
references.pmid: 20368647