Verbal protocols for assessing the usability of clinical decision support: the retrospective sense making protocol.
We compare the effectiveness of two types of verbal protocol, concurrent think aloud vs. retrospective sense making, for evaluating the usability of a clinical decision support tool. Thirty-five medical and nursing students participated in a usability experiment. Participants were asked to complete seven tasks using the system under evaluation. Eighteen students were allocated to the concurrent think aloud group and the remainder followed the retrospective protocol. The retrospective protocol was significantly more sensitive than the concurrent protocol in recording unique usability problems related to users' cognitive behaviour. These problems concerned the interpretation and comprehension of statistical output, search results and system messages. These findings can be explained by the retrospective protocol's greater ability to detect compound usability problems, capturing the cognitive dimensions of users' interactions with the interface in greater depth. Evaluations of clinical decision support systems should take an evidence-based approach to selecting protocols.
Centre for Health Informatics, Institute of Population Health, University of Manchester and Manchester Academic Health Science Centre, UK.
Panagiotis
P
Balatsoukas
John
J
Ainsworth
Richard
R
Williams
Emma
E
Carruthers
Colin
C
Davies
James
J
McGrath
Artur
A
Akbarov
Claudia
C
Soiland-Reyes
Saurin
S
Badiyani
Iain
I
Buchan
D000465:Algorithms; D032882:Comprehension; D003258:Consumer Behavior / Q000706:statistics & numerical data*; D020000:Decision Support Systems, Clinical / Q000706:statistics & numerical data*; D005260:Female; D006801:Humans; D008297:Male; D012057:Reinforcement, Verbal; D012189:Retrospective Studies; D012984:Software; D012986:Software Validation; D006113:United Kingdom; D014584:User-Computer Interface