title: Ease of adoption of clinical natural language processing software: An evaluation of five systems.
abstract: OBJECTIVE
  In recognition of potential barriers that may inhibit the widespread adoption of biomedical software, the 2014 i2b2 Challenge introduced a special track, Track 3 - Software Usability Assessment, in order to develop a better understanding of the adoption issues that might be associated with the state-of-the-art clinical NLP systems. This paper reports the ease of adoption assessment methods we developed for this track, and the results of evaluating five clinical NLP system submissions.
  
  
  METHODS
  A team of human evaluators performed a series of scripted adoptability test tasks with each of the participating systems. The evaluation team consisted of four "expert evaluators" with training in computer science, and eight "end user evaluators" with mixed backgrounds in medicine, nursing, pharmacy, and health informatics. We assessed how easy it is to adopt the submitted systems along the following three dimensions: communication effectiveness (i.e., how effective a system is in communicating its designed objectives to intended audience), effort required to install, and effort required to use. We used a formal software usability testing tool, TURF, to record the evaluators' interactions with the systems and 'think-aloud' data revealing their thought processes when installing and using the systems and when resolving unexpected issues.
  
  
  RESULTS
  Overall, the ease of adoption ratings that the five systems received are unsatisfactory. Installation of some of the systems proved to be rather difficult, and some systems failed to adequately communicate their designed objectives to intended adopters. Further, the average ratings provided by the end user evaluators on ease of use and ease of interpreting output are -0.35 and -0.53, respectively, indicating that this group of users generally deemed the systems extremely difficult to work with. While the ratings provided by the expert evaluators are higher, 0.6 and 0.45, respectively, these ratings are still low indicating that they also experienced considerable struggles.
  
  
  CONCLUSIONS
  The results of the Track 3 evaluation show that the adoptability of the five participating clinical NLP systems has a great margin for improvement. Remedy strategies suggested by the evaluators included (1) more detailed and operation system specific use instructions; (2) provision of more pertinent onscreen feedback for easier diagnosis of problems; (3) including screen walk-throughs in use instructions so users know what to expect and what might have gone wrong; (4) avoiding jargon and acronyms in materials intended for end users; and (5) packaging prerequisites required within software distributions so that prospective adopters of the software do not have to obtain each of the third-party components on their own.
authors.affiliation: School of Public Health Department of Health Management and Policy, University of Michigan, Ann Arbor, MI, USA; School of Information, University of Michigan, Ann Arbor, MI, USA. Electronic address: kzheng@umich.edu.
authors.forename: Kai
authors.initials: K
authors.lastname: Zheng
authors.affiliation: Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA.
authors.forename: V G Vinod
authors.initials: VGV
authors.lastname: Vydiswaran
authors.affiliation: School of Information, University of Michigan, Ann Arbor, MI, USA.
authors.forename: Yang
authors.initials: Y
authors.lastname: Liu
authors.affiliation: Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA.
authors.forename: Yue
authors.initials: Y
authors.lastname: Wang
authors.affiliation: School of Library and Information Science, Simmons College, Boston, MA, USA.
authors.forename: Amber
authors.initials: A
authors.lastname: Stubbs
authors.affiliation: Department of Information Studies, University at Albany, SUNY, Albany, NY, USA.
authors.forename: Özlem
authors.initials: Ö
authors.lastname: Uzuner
authors.affiliation: The University of Texas School of Biomedical Informatics at Houston, Houston, TX, USA.
authors.forename: Anupama E
authors.initials: AE
authors.lastname: Gururaj
authors.affiliation: The MITRE Corporation, Bedford, MA, USA.
authors.forename: Samuel
authors.initials: S
authors.lastname: Bayer
authors.affiliation: The MITRE Corporation, Bedford, MA, USA.
authors.forename: John
authors.initials: J
authors.lastname: Aberdeen
authors.affiliation: Department of Computer Science, University of Massachusetts, Lowell, MA, USA.
authors.forename: Anna
authors.initials: A
authors.lastname: Rumshisky
authors.affiliation: University of Minnesota, Minneapolis, MN, USA.
authors.forename: Serguei
authors.initials: S
authors.lastname: Pakhomov
authors.affiliation: Department of Health Sciences Research, Mayo Clinic, Rochester, MN, USA.
authors.forename: Hongfang
authors.initials: H
authors.lastname: Liu
authors.affiliation: The University of Texas School of Biomedical Informatics at Houston, Houston, TX, USA. Electronic address: hua.xu@uth.tmc.edu.
authors.forename: Hua
authors.initials: H
authors.lastname: Xu
mesh_terms: D001292:Attitude to Computers; D057225:Data Mining / Q000379:methods / Q000706:statistics & numerical data*; D057286:Electronic Health Records / Q000706:statistics & numerical data*; D006801:Humans; D008875:Middle Aged; D009323:Natural Language Processing; D010363:Pattern Recognition, Automated / Q000379:methods*; D012984:Software; D014584:User-Computer Interface
keywords: Human–computer interaction; Natural language processing [L01.224.065.580]; Software design [L01.224.900.820]; Software validation [L01.224.900.868]; Usability; User-computer interface [L01.224.900.910]