title: Comparison of the evaluations of a case-based reasoning decision support tool by specialist expert reviewers with those of end users.
abstract: BACKGROUND
  Decision-support tools (DST) are typically developed by computer engineers for use by clinicians. Prototype testing DSTs may be performed relatively easily by one or two clinical experts. The costly alternative is to test each prototype on a larger number of diverse clinicians, based on the untested assumption that these evaluations would more accurately reflect those of actual end users.
  
  
  OBJECTIVE
  We hypothesized substantial or better agreement (as defined by a kappa statistic greater than 0.6) between the evaluations of a case based reasoning (CBR) DST predicting ED admission for bronchiolitis performed by the clinically diverse end users, to those of two clinical experts who evaluated the same DST output.
  
  
  METHODS
  Three outputs from a previously described DST were evaluated by the emergency physicians (EP) who originally saw the patients and by two pediatric EPs with an interest in bronchiolitis. The DST outputs were as follows: predicted disposition, an example of another previously seen patient to explain the prediction, and explanatory dialog. Each was rated using the scale Definitely Not, No, Maybe, Yes, and Absolutely. This was converted to a Likert scale for analysis. Agreement was measured using the kappa statistic.
  
  
  RESULTS
  Agreement with the DST predicted disposition was moderate between end users and the expert reviewers, but was only fair or poor for value of the explanatory case and dialog.
  
  
  CONCLUSIONS
  Agreement between expert evaluators and end users on the value of a CBR DST predicted dispositions was moderate. For the more subjective explicative components, agreement was fair, poor, or worse.
authors: Kern Medical Center, Department of Emergency Medicine. yousentwhohome@yahoo.com
authors: Paul
authors: P
authors: Walsh
authors: Donal
authors: D
authors: Doyle
authors: Kemedy K
authors: KK
authors: McQuillen
authors: Joshua
authors: J
authors: Bigler
authors: Caleb
authors: C
authors: Thompson
authors: Ed
authors: E
authors: Lin
authors: Padraig
authors: P
authors: Cunningham