title: Evaluating topic model interpretability from a primary care physician perspective.
abstract: OBJECTIVE
  Probabilistic topic models provide an unsupervised method for analyzing unstructured text. These models discover semantically coherent combinations of words (topics) that could be integrated in a clinical automatic summarization system for primary care physicians performing chart review. However, the human interpretability of topics discovered from clinical reports is unknown. Our objective is to assess the coherence of topics and their ability to represent the contents of clinical reports from a primary care physician's point of view.
  
  
  METHODS
  Three latent Dirichlet allocation models (50 topics, 100 topics, and 150 topics) were fit to a large collection of clinical reports. Topics were manually evaluated by primary care physicians and graduate students. Wilcoxon Signed-Rank Tests for Paired Samples were used to evaluate differences between different topic models, while differences in performance between students and primary care physicians (PCPs) were tested using Mann-Whitney U tests for each of the tasks.
  
  
  RESULTS
  While the 150-topic model produced the best log likelihood, participants were most accurate at identifying words that did not belong in topics learned by the 100-topic model, suggesting that 100 topics provides better relative granularity of discovered semantic themes for the data set used in this study. Models were comparable in their ability to represent the contents of documents. Primary care physicians significantly outperformed students in both tasks.
  
  
  CONCLUSIONS
  This work establishes a baseline of interpretability for topic models trained with clinical reports, and provides insights on the appropriateness of using topic models for informatics applications. Our results indicate that PCPs find discovered topics more coherent and representative of clinical reports relative to students, warranting further research into their use for automatic summarization.
authors.affiliation: Medical Imaging and Informatics Group, Department of Radiological Sciences, University of California, Los Angeles, United States. Electronic address: cwarnold@ucla.edu.
authors.forename: Corey W
authors.initials: CW
authors.lastname: Arnold
authors.affiliation: Medical Imaging and Informatics Group, Department of Radiological Sciences, University of California, Los Angeles, United States.
authors.forename: Andrea
authors.initials: A
authors.lastname: Oh
authors.affiliation: Medical Imaging and Informatics Group, Department of Radiological Sciences, University of California, Los Angeles, United States.
authors.forename: Shawn
authors.initials: S
authors.lastname: Chen
authors.affiliation: Medical Imaging and Informatics Group, Department of Radiological Sciences, University of California, Los Angeles, United States.
authors.forename: William
authors.initials: W
authors.lastname: Speier
mesh_terms: D001291:Attitude of Health Personnel; D032882:Comprehension; D057286:Electronic Health Records / Q000145:classification* / Q000458:organization & administration; D016247:Information Storage and Retrieval / Q000379:methods; D062527:Meaningful Use / Q000706:statistics & numerical data; D008498:Medical Record Linkage / Q000379:methods; D015233:Models, Statistical; D009323:Natural Language Processing; D058007:Physicians, Primary Care / Q000706:statistics & numerical data*; D012660:Semantics; D009626:Terminology as Topic; D018875:Vocabulary, Controlled
keywords: Clinical reports; Primary care; Topic modeling