title: Developing and evaluating an automated appendicitis risk stratification algorithm for pediatric patients in the emergency department.
abstract: OBJECTIVE
  To evaluate a proposed natural language processing (NLP) and machine-learning based automated method to risk stratify abdominal pain patients by analyzing the content of the electronic health record (EHR).
  
  
  METHODS
  We analyzed the EHRs of a random sample of 2100 pediatric emergency department (ED) patients with abdominal pain, including all with a final diagnosis of appendicitis. We developed an automated system to extract relevant elements from ED physician notes and lab values and to automatically assign a risk category for acute appendicitis (high, equivocal, or low), based on the Pediatric Appendicitis Score. We evaluated the performance of the system against a manually created gold standard (chart reviews by ED physicians) for recall, specificity, and precision.
  
  
  RESULTS
  The system achieved an average F-measure of 0.867 (0.869 recall and 0.863 precision) for risk classification, which was comparable to physician experts. Recall/precision were 0.897/0.952 in the low-risk category, 0.855/0.886 in the high-risk category, and 0.854/0.766 in the equivocal-risk category. The information that the system required as input to achieve high F-measure was available within the first 4 h of the ED visit.
  
  
  CONCLUSIONS
  Automated appendicitis risk categorization based on EHR content, including information from clinical notes, shows comparable performance to physician chart reviewers as measured by their inter-annotator agreement and represents a promising new approach for computerized decision support to promote application of evidence-based medicine at the point of care.
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA.
authors.forename: Louise
authors.initials: L
authors.lastname: Deleger
authors.forename: Holly
authors.initials: H
authors.lastname: Brodzinski
authors.forename: Haijun
authors.initials: H
authors.lastname: Zhai
authors.forename: Qi
authors.initials: Q
authors.lastname: Li
authors.forename: Todd
authors.initials: T
authors.lastname: Lingren
authors.forename: Eric S
authors.initials: ES
authors.lastname: Kirkendall
authors.forename: Evaline
authors.initials: E
authors.lastname: Alessandrini
authors.forename: Imre
authors.initials: I
authors.lastname: Solti
mesh_terms: D015746:Abdominal Pain / Q000209:etiology*; D000465:Algorithms; D001064:Appendicitis / Q000175:diagnosis*; D001185:Artificial Intelligence; D002648:Child; D057286:Electronic Health Records; D004636:Emergency Service, Hospital; D006801:Humans; D009323:Natural Language Processing; D018570:Risk Assessment / Q000379:methods
keywords: Electronic Health Record; Information Extraction; Natural Language Processing; Pediatric Appendicitis Score; Risk Stratification