title: Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR.
abstract: OBJECTIVE
  To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys.
  
  
  METHODS
  Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35).
  
  
  RESULTS
  The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods.
  
  
  CONCLUSIONS
  Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR.
  
  
  CONCLUSIONS
  We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.
authors.affiliation: The University of Texas Health Science Center at Houston, United States. Electronic address: Muhammad.F.Walji@uth.tmc.edu.
authors.forename: Muhammad F
authors.initials: MF
authors.lastname: Walji
authors.affiliation: Harvard School of Dental Medicine, United States.
authors.forename: Elsbeth
authors.initials: E
authors.lastname: Kalenderian
authors.affiliation: The University of Texas Health Science Center at Houston, United States.
authors.forename: Mark
authors.initials: M
authors.lastname: Piotrowski
authors.affiliation: The University of Texas Health Science Center at Houston, United States.
authors.forename: Duong
authors.initials: D
authors.lastname: Tran
authors.affiliation: The University of Texas Health Science Center at Houston, United States.
authors.forename: Krishna K
authors.initials: KK
authors.lastname: Kookal
authors.affiliation: Harvard School of Dental Medicine, United States.
authors.forename: Oluwabunmi
authors.initials: O
authors.lastname: Tokede
authors.affiliation: University of California, San Francisco, United States.
authors.forename: Joel M
authors.initials: JM
authors.lastname: White
authors.affiliation: University of California, San Francisco, United States.
authors.forename: Ram
authors.initials: R
authors.lastname: Vaderhobli
authors.affiliation: Harvard Medical School, United States.
authors.forename: Rachel
authors.initials: R
authors.lastname: Ramoni
authors.affiliation: Tufts University, United States.
authors.forename: Paul C
authors.initials: PC
authors.lastname: Stark
authors.affiliation: Creighton University, United States.
authors.forename: Nicole S
authors.initials: NS
authors.lastname: Kimmes
authors.affiliation: Academic Centre for Dentistry at Amsterdam (ACTA), United States.
authors.forename: Maxim
authors.initials: M
authors.lastname: Lagerweij
authors.affiliation: The New York Academy of Medicine, United States.
authors.forename: Vimla L
authors.initials: VL
authors.lastname: Patel
mesh_terms: D003625:Data Collection; D057286:Electronic Health Records / Q000706:statistics & numerical data*; D005260:Female; D006801:Humans; D007407:Interviews as Topic; D008297:Male; D008490:Medical Informatics; D009862:Online Systems / Q000592:standards*; D012984:Software; D014584:User-Computer Interface; D014600:Utilization Review
keywords: EHR; Human factors; Methodology; Usability