title: Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data.
abstract: OBJECTIVE
  Evaluate KNAVE-II, a knowledge-based framework for visualization, interpretation, and exploration of longitudinal clinical data, clinical concepts and patterns. KNAVE-II mediates queries to a distributed temporal-abstraction architecture (IDAN), which uses a knowledge-based problem-solving method specializing in on-the-fly computation of clinical queries.
  
  
  METHODS
  A two-phase, balanced cross-over study to compare efficiency and satisfaction of a group of clinicians when answering queries of variable complexity about time-oriented clinical data, typical for oncology protocols, using KNAVE-II, versus standard methods: both paper charts and a popular electronic spreadsheet (ESS) in Phase I; an ESS in Phase II. The measurements included the time required to answer and the correctness of answer for each query and each complexity category, and for all queries, assessed versus a predetermined gold standard set by a domain expert. User satisfaction was assessed by the Standard Usability Score (SUS) tool-specific questionnaire and by a "Usability of Tool Comparison" comparative questionnaire developed for this study.
  
  
  RESULTS
  In both evaluations, subjects answered higher-complexity queries significantly faster using KNAVE-II than when using paper charts or an ESS up to a mean of 255 s difference per query versus the ESS for hard queries (p=0.0003) in the second evaluation. Average correctness scores when using KNAVE-II versus paper charts, in the first phase, and the ESS, in the second phase, were significantly higher over all queries. In the second evaluation, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, opposed to only 57.5% (69/120) using the ESS (p<0.0001). User satisfaction with KNAVE-II was significantly superior compared to using either a paper chart or the ESS (p=0.006). Clinicians ranked KNAVE-II superior to both paper and the ESS.
  
  
  CONCLUSIONS
  An evaluation of the functionality and usability of KNAVE-II and its supporting knowledge-based temporal-mediation architecture has produced highly encouraging results regarding saving of physician time, enhancement of accuracy of clinical assessment, and user satisfaction.
authors.affiliation: Center for Primary Care and Outcomes Research, Stanford University School of Medicine, Stanford, CA, USA.
authors.forename: Susana B
authors.initials: SB
authors.lastname: Martins
authors.forename: Yuval
authors.initials: Y
authors.lastname: Shahar
authors.forename: Dina
authors.initials: D
authors.lastname: Goren-Bar
authors.forename: Maya
authors.initials: M
authors.lastname: Galperin
authors.forename: Herbert
authors.initials: H
authors.lastname: Kaizer
authors.forename: Lawrence V
authors.initials: LV
authors.lastname: Basso
authors.forename: Deborah
authors.initials: D
authors.lastname: McNaughton
authors.forename: Mary K
authors.initials: MK
authors.lastname: Goldstein
mesh_terms: D001185:Artificial Intelligence; D003195:Computer Communication Networks; D003258:Consumer Behavior; D018592:Cross-Over Studies; D020000:Decision Support Systems, Clinical; D006801:Humans; D008491:Medical Informatics Applications; D016347:Medical Records Systems, Computerized; D014584:User-Computer Interface