An extended protocol for usability validation of medical devices: Research design and reference model.
This paper proposes and demonstrates an extended protocol for usability validation testing of medical devices. A review of currently used methods for the usability evaluation of medical devices revealed two main shortcomings. Firstly, the lack of methods to closely trace the interaction sequences and derive performance measures. Secondly, a prevailing focus on cross-sectional validation studies, ignoring the issues of learnability and training. The U.S. Federal Drug and Food Administration's recent proposal for a validation testing protocol for medical devices is then extended to address these shortcomings: (1) a novel process measure 'normative path deviations' is introduced that is useful for both quantitative and qualitative usability studies and (2) a longitudinal, completely within-subject study design is presented that assesses learnability, training effects and allows analysis of diversity of users. A reference regression model is introduced to analyze data from this and similar studies, drawing upon generalized linear mixed-effects models and a Bayesian estimation approach. The extended protocol is implemented and demonstrated in a study comparing a novel syringe infusion pump prototype to an existing design with a sample of 25 healthcare professionals. Strong performance differences between designs were observed with a variety of usability measures, as well as varying training-on-the-job effects. We discuss our findings with regard to validation testing guidelines, reflect on the extensions and discuss the perspectives they add to the validation process.
Department of Cognitive Psychology and Ergonomics, University of Twente, Enschede, The Netherlands. Electronic address: m.schmettow@utwente.nl.
Martin
M
Schmettow
Monash University Accident Research Centre, Monash University, Clayton, Australia.
Raphaela
R
Schnittker
Department of Cognitive Psychology and Ergonomics, University of Twente, Enschede, The Netherlands; Department of Human Behaviour and Organisational Innovation, TNO Earth, Life, and Social Sciences, Soesterberg, The Netherlands.
Jan Maarten
JM
Schraagen
D000465:Algorithms; D001499:Bayes Theorem; D019985:Benchmarking; D003430:Cross-Sectional Studies; D004869:Equipment Safety; D007260:Infusion Pumps; D012986:Software Validation
Generalized linear mixed-effects models; Human Factors; Longitudinal research; Medical device design; Patient safety; Usability testing