Comparison of the evaluations of a case-based reasoning decision support tool by specialist expert reviewers with those of end users.
BACKGROUND
Decision-support tools (DST) are typically developed by computer engineers for use by clinicians. Prototype testing DSTs may be performed relatively easily by one or two clinical experts. The costly alternative is to test each prototype on a larger number of diverse clinicians, based on the untested assumption that these evaluations would more accurately reflect those of actual end users.


OBJECTIVE
We hypothesized substantial or better agreement (as defined by a kappa statistic greater than 0.6) between the evaluations of a case based reasoning (CBR) DST predicting ED admission for bronchiolitis performed by the clinically diverse end users, to those of two clinical experts who evaluated the same DST output.


METHODS
Three outputs from a previously described DST were evaluated by the emergency physicians (EP) who originally saw the patients and by two pediatric EPs with an interest in bronchiolitis. The DST outputs were as follows: predicted disposition, an example of another previously seen patient to explain the prediction, and explanatory dialog. Each was rated using the scale Definitely Not, No, Maybe, Yes, and Absolutely. This was converted to a Likert scale for analysis. Agreement was measured using the kappa statistic.


RESULTS
Agreement with the DST predicted disposition was moderate between end users and the expert reviewers, but was only fair or poor for value of the explanatory case and dialog.


CONCLUSIONS
Agreement between expert evaluators and end users on the value of a CBR DST predicted dispositions was moderate. For the more subjective explicative components, agreement was fair, poor, or worse.
Walsh
Paul
P
Kern Medical Center, Department of Emergency Medicine. yousentwhohome@yahoo.com
Doyle
Donal
D
McQuillen
Kemedy K
KK
Bigler
Joshua
J
Thompson
Caleb
C
Lin
Ed
E
Cunningham
Padraig
P