title: Using natural language processing to provide personalized learning opportunities from trainee clinical notes.
abstract: OBJECTIVE
  Assessment of medical trainee learning through pre-defined competencies is now commonplace in schools of medicine. We describe a novel electronic advisor system using natural language processing (NLP) to identify two geriatric medicine competencies from medical student clinical notes in the electronic medical record: advance directives (AD) and altered mental status (AMS).
  
  
  METHODS
  Clinical notes from third year medical students were processed using a general-purpose NLP system to identify biomedical concepts and their section context. The system analyzed these notes for relevance to AD or AMS and generated custom email alerts to students with embedded supplemental learning material customized to their notes. Recall and precision of the two advisors were evaluated by physician review. Students were given pre and post multiple choice question tests broadly covering geriatrics.
  
  
  RESULTS
  Of 102 students approached, 66 students consented and enrolled. The system sent 393 email alerts to 54 students (82%), including 270 for AD and 123 for AMS. Precision was 100% for AD and 93% for AMS. Recall was 69% for AD and 100% for AMS. Students mentioned ADs for 43 patients, with all mentions occurring after first having received an AD reminder. Students accessed educational links 34 times from the 393 email alerts. There was no difference in pre (mean 62%) and post (mean 60%) test scores.
  
  
  CONCLUSIONS
  The system effectively identified two educational opportunities using NLP applied to clinical notes and demonstrated a small change in student behavior. Use of electronic advisors such as these may provide a scalable model to assess specific competency elements and deliver educational opportunities.
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, United States; Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, United States. Electronic address: josh.denny@vanderbilt.edu.
authors.forename: Joshua C
authors.initials: JC
authors.lastname: Denny
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, United States; Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, United States.
authors.forename: Anderson
authors.initials: A
authors.lastname: Spickard
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, United States.
authors.forename: Peter J
authors.initials: PJ
authors.lastname: Speltz
authors.affiliation: The Center for Quality Aging, Vanderbilt University School of Medicine, Nashville, TN, United States; Office of Health Sciences Education, Vanderbilt University School of Medicine, Nashville, TN, United States.
authors.forename: Renee
authors.initials: R
authors.lastname: Porier
authors.affiliation: The Center for Quality Aging, Vanderbilt University School of Medicine, Nashville, TN, United States.
authors.forename: Donna E
authors.initials: DE
authors.lastname: Rosenstiel
authors.affiliation: Department of Medicine, Vanderbilt University School of Medicine, Nashville, TN, United States; The Center for Quality Aging, Vanderbilt University School of Medicine, Nashville, TN, United States; The Meharry Consortium Geriatric Education Center, Meharry Medical Center, Nashville, TN, United States; The Tennessee Valley Geriatric Research Education and Clinical Center, Tennessee Valley Healthcare System, Nashville, TN, United States.
authors.forename: James S
authors.initials: JS
authors.lastname: Powers
mesh_terms: D000046:Academic Medical Centers; D016223:Advance Directives; D000368:Aged; D000465:Algorithms; D001331:Automation; D002982:Clinical Clerkship; D002983:Clinical Competence; D004501:Education, Medical; D004521:Educational Measurement; D057286:Electronic Health Records; D005853:Geriatrics / Q000193:education*; D006787:Hospitals, Veterans; D006801:Humans; D007858:Learning; D001523:Mental Disorders / Q000175:diagnosis*; D008875:Middle Aged; D009323:Natural Language Processing; D017063:Outcome Assessment, Health Care; D015203:Reproducibility of Results; D012984:Software; D013337:Students, Medical; D013714:Tennessee; D014584:User-Computer Interface
keywords: Advanced directives; Altered mental status; Decision support; Geriatric education; Medical education; Natural language processing