Validity of scores generated by a web-based multimedia simulated patient case software: a pilot study.
OBJECTIVE
The value of multimedia simulated patient cases (MSPCs) in medical education remains unclear. The authors conducted a pilot study to assess the validity of automated scores of diagnostic reasoning ability provided by DxR Clinician, a widely available Web-based MSPC software.


METHODS
In 2002-03, all 89 students enrolled in a required third-year primary care clerkship at the University of California, Davis, School of Medicine were assigned to complete four MSPCs. The authors determined the degree of correlation between the Clinical Reasoning Score (CRS) and Level of Diagnostic Performance (LDP) generated by the MSPC software and subscale scores from a validated measure of diagnostic reasoning sophistication, the Diagnostic Thinking Inventory (DTI).


RESULTS
Of 356 completed case events, instructor override of automated scoring was required in 206 (58%) to obtain an accurate LDP and CRS. Mean DTI subscale scores improved significantly from the beginning to the end of the year (p <.0001, Wilcoxon signed rank test). However, there were no significant correlations between CRS or LDP scores on any of the four cases and either of the two DTI subscale scores.


CONCLUSIONS
Automated diagnostic reasoning scores generated by one widely available MSPC software appear to lack criterion validity. The validity of automated diagnostic reasoning scores generated by MSPCs should be established before such cases can be confidently employed as educational tools.
Jerant
Anthony F
AF
Department of Family and Community Medicine, University of California Davis School of Medicine, 4860 Y Street, Suite 2300, Sacramento, CA 95817, USA. afjerant@ucdavis.edu
Azari
Rahman
R
D002982:Clinical Clerkship / Q000379:methods*; D003194:Computer-Assisted Instruction; D003936:Diagnosis, Computer-Assisted; D004504:Education, Medical, Undergraduate / Q000379:methods* / Q000592:standards; D004521:Educational Measurement; D005260:Female; D006801:Humans; D020407:Internet; D008297:Male; D019212:Multimedia; D016544:Patient Simulation; D010865:Pilot Projects; D012680:Sensitivity and Specificity; D012984:Software