title: Comparison of automatic summarisation methods for clinical free text notes.
abstract: OBJECTIVE
  A major source of information available in electronic health record (EHR) systems are the clinical free text notes documenting patient care. Managing this information is time-consuming for clinicians. Automatic text summarisation could assist clinicians in obtaining an overview of the free text information in ongoing care episodes, as well as in writing final discharge summaries. We present a study of automated text summarisation of clinical notes. It looks to identify which methods are best suited for this task and whether it is possible to automatically evaluate the quality differences of summaries produced by different methods in an efficient and reliable way.
  
  
  METHODS
  The study is based on material consisting of 66,884 care episodes from EHRs of heart patients admitted to a university hospital in Finland between 2005 and 2009. We present novel extractive text summarisation methods for summarising the free text content of care episodes. Most of these methods rely on word space models constructed using distributional semantic modelling. The summarisation effectiveness is evaluated using an experimental automatic evaluation approach incorporating well-known ROUGE measures. We also developed a manual evaluation scheme to perform a meta-evaluation on the ROUGE measures to see if they reflect the opinions of health care professionals.
  
  
  RESULTS
  The agreement between the human evaluators is good (ICC=0.74, p<0.001), demonstrating the stability of the proposed manual evaluation method. Furthermore, the correlation between the manual and automated evaluations are high (> 0.90 Spearman's rho). Three of the presented summarisation methods ('Composite', 'Case-Based' and 'Translate') significantly outperform the other methods for all ROUGE measures (p<0.05, Wilcoxon signed-rank test and Bonferroni correction).
  
  
  CONCLUSIONS
  The results indicate the feasibility of the automated summarisation of care episodes. Moreover, the high correlation between manual and automated evaluations suggests that the less labour-intensive automated evaluations can be used as a proxy for human evaluations when developing summarisation methods. This is of significant practical value for summarisation method development, because manual evaluation cannot be afforded for every variation of the summarisation methods. Instead, one can resort to automatic evaluation during the method development process.
authors.affiliation: Department of Computer and Information Science, Norwegian University of Science and Technology, Sem Saelands vei 9, 7491 Trondheim, Norway; Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Department of Nursing Science, University of Turku, Lemminkäisenkatu 1, 20520 Turku, Finland. Electronic address: hans.moen@idi.ntnu.no.
authors.forename: Hans
authors.initials: H
authors.lastname: Moen
authors.affiliation: Department of Nursing Science, University of Turku, Lemminkäisenkatu 1, 20520 Turku, Finland; Turku University Hospital, Kiinamyllynkatu 4-8, 20521 Turku, Finland. Electronic address: lmemur@utu.fi.
authors.forename: Laura-Maria
authors.initials: LM
authors.lastname: Peltonen
authors.affiliation: Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Turku Centre for Computer Science (TUCS), Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: juaheim@utu.fi.
authors.forename: Juho
authors.initials: J
authors.lastname: Heimonen
authors.affiliation: Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: ajairo@utu.fi.
authors.forename: Antti
authors.initials: A
authors.lastname: Airola
authors.affiliation: Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Turku Centre for Computer Science (TUCS), Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: aatapa@utu.fi.
authors.forename: Tapio
authors.initials: T
authors.lastname: Pahikkala
authors.affiliation: Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Turku Centre for Computer Science (TUCS), Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: tapio.salakoski@utu.fi.
authors.forename: Tapio
authors.initials: T
authors.lastname: Salakoski
authors.affiliation: Department of Nursing Science, University of Turku, Lemminkäisenkatu 1, 20520 Turku, Finland; Turku University Hospital, Kiinamyllynkatu 4-8, 20521 Turku, Finland. Electronic address: sansala@utu.fi.
authors.forename: Sanna
authors.initials: S
authors.lastname: Salanterä
mesh_terms: D001331:Automation; D057286:Electronic Health Records; D005387:Finland; D006331:Heart Diseases / Q000503:physiopathology / Q000628:therapy; D006785:Hospitals, University; D006801:Humans
keywords: Automatic text summarisation; Clinical text processing; Distributional semantics; Electronic health records; Summarisation evaluation; Word space models