Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data.
OBJECTIVE
Evaluate KNAVE-II, a knowledge-based framework for visualization, interpretation, and exploration of longitudinal clinical data, clinical concepts and patterns. KNAVE-II mediates queries to a distributed temporal-abstraction architecture (IDAN), which uses a knowledge-based problem-solving method specializing in on-the-fly computation of clinical queries.


METHODS
A two-phase, balanced cross-over study to compare efficiency and satisfaction of a group of clinicians when answering queries of variable complexity about time-oriented clinical data, typical for oncology protocols, using KNAVE-II, versus standard methods: both paper charts and a popular electronic spreadsheet (ESS) in Phase I; an ESS in Phase II. The measurements included the time required to answer and the correctness of answer for each query and each complexity category, and for all queries, assessed versus a predetermined gold standard set by a domain expert. User satisfaction was assessed by the Standard Usability Score (SUS) tool-specific questionnaire and by a "Usability of Tool Comparison" comparative questionnaire developed for this study.


RESULTS
In both evaluations, subjects answered higher-complexity queries significantly faster using KNAVE-II than when using paper charts or an ESS up to a mean of 255 s difference per query versus the ESS for hard queries (p=0.0003) in the second evaluation. Average correctness scores when using KNAVE-II versus paper charts, in the first phase, and the ESS, in the second phase, were significantly higher over all queries. In the second evaluation, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, opposed to only 57.5% (69/120) using the ESS (p<0.0001). User satisfaction with KNAVE-II was significantly superior compared to using either a paper chart or the ESS (p=0.006). Clinicians ranked KNAVE-II superior to both paper and the ESS.


CONCLUSIONS
An evaluation of the functionality and usability of KNAVE-II and its supporting knowledge-based temporal-mediation architecture has produced highly encouraging results regarding saving of physician time, enhancement of accuracy of clinical assessment, and user satisfaction.
Martins
Susana B
SB
Center for Primary Care and Outcomes Research, Stanford University School of Medicine, Stanford, CA, USA.
Shahar
Yuval
Y
Goren-Bar
Dina
D
Galperin
Maya
M
Kaizer
Herbert
H
Basso
Lawrence V
LV
McNaughton
Deborah
D
Goldstein
Mary K
MK
D001185:Artificial Intelligence; D003195:Computer Communication Networks; D003258:Consumer Behavior; D018592:Cross-Over Studies; D020000:Decision Support Systems, Clinical; D006801:Humans; D008491:Medical Informatics Applications; D016347:Medical Records Systems, Computerized; D014584:User-Computer Interface