title: From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.
abstract: Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of "the best available" comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires - System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model for Patient Care and Clinical Trials (IMPACT). We conclude that this mixed-methods evaluation framework enables an integrated assessment of user needs satisfaction and user-perceived usefulness and usability of a novel design. This evaluation framework effectively bridges the gap between co-evolving user needs and technology designs during iterative prototyping and is particularly useful when it is difficult for users to articulate their needs for technology support due to the lack of a baseline.
authors: Department of Biomedical Informatics, Columbia University, New York, NY, USA.
authors: Mary Regina
authors: MR
authors: Boland
authors: Department of Anesthesiology, Columbia University, New York, NY, USA.
authors: Alexander
authors: A
authors: Rusanov
authors: Department of Biomedical Informatics, Columbia University, New York, NY, USA.
authors: Yat
authors: Y
authors: So
authors: Department of Medicine, Division of Cardiology, Columbia University, New York, NY, USA.
authors: Carlos
authors: C
authors: Lopez-Jimenez
authors: The Clinical Trials Office, Columbia University, New York, NY, USA.
authors: Linda
authors: L
authors: Busacca
authors: Department of Medicine, Division of Cardiology, Columbia University, New York, NY, USA; The Irving Institute for Clinical and Translational Research, Columbia University, New York, NY, USA.
authors: Richard C
authors: RC
authors: Steinman
authors: Department of Biomedical Informatics, Columbia University, New York, NY, USA; School of Nursing, Columbia University, New York, NY, USA; The Irving Institute for Clinical and Translational Research, Columbia University, New York, NY, USA.
authors: Suzanne
authors: S
authors: Bakken
authors: Department of Medicine, Division of Cardiology, Columbia University, New York, NY, USA; The Irving Institute for Clinical and Translational Research, Columbia University, New York, NY, USA.
authors: J Thomas
authors: JT
authors: Bigger
authors: Department of Biomedical Informatics, Columbia University, New York, NY, USA; The Irving Institute for Clinical and Translational Research, Columbia University, New York, NY, USA. Electronic address: cw2384@columbia.edu.
authors: Chunhua
authors: C
authors: Weng
mesh_terms: D035843:Biomedical Research; D002986:Clinical Trials as Topic; D005069:Evaluation Studies as Topic; D006801:Humans; D008490:Medical Informatics; D020380:Needs Assessment
keywords: Clinical trials; Evaluation studies; Medical informatics; Needs assessment; Workflow