Evaluation of an intelligent tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance.
OBJECTIVE
Determine effects of computer-based tutoring on diagnostic performance gains, meta-cognition, and acceptance using two different problem representations. Describe impact of tutoring on spectrum of diagnostic skills required for task performance. Identify key features of student-tutor interaction contributing to learning gains.


METHODS
Prospective, between-subjects study, controlled for participant level of training. Resident physicians in two academic pathology programs spent four hours using one of two interfaces which differed mainly in external problem representation. The case-focused representation provided an open-learning environment in which students were free to explore evidence-hypothesis relationships within a case, but could not visualize the entire diagnostic space. The knowledge-focused representation provided an interactive representation of the entire diagnostic space, which more tightly constrained student actions.


METHODS
Metrics included results of pretest, post-test and retention-test for multiple choice and case diagnosis tests, ratios of performance to student reported certainty, results of participant survey, learning curves, and interaction behaviors during tutoring.


RESULTS
Students had highly significant learning gains after one tutoring session. Learning was retained at one week. There were no differences between the two interfaces in learning gains on post-test or retention test. Only students in the knowledge-focused interface exhibited significant metacognitive gains from pretest to post-test and pretest to retention test. Students rated the knowledge-focused interface significantly higher than the case-focused interface.


CONCLUSIONS
Cognitive tutoring is associated with improved diagnostic performance in a complex medical domain. The effect is retained at one-week post-training. Knowledge-focused external problem representation shows an advantage over case-focused representation for metacognitive effects and user acceptance.
Crowley
Rebecca S
RS
Department of Biomedical Informatics, University of Pittsburgh School of Medicine, UPMC Shadyside Cancer Pavilion, Room 307, 5230 Centre Avenue, Pittsburgh, PA 15232, USA. crowleyrs@upmc.edu
Legowski
Elizabeth
E
Medvedeva
Olga
O
Tseytlin
Eugene
E
Roh
Ellen
E
Jukic
Drazen
D
D001292:Attitude to Computers; D003071:Cognition; D003194:Computer-Assisted Instruction; D003625:Data Collection; D003933:Diagnosis; D006801:Humans; D010336:Pathology / Q000193:education*; D018794:Problem-Based Learning; D013334:Students; D014584:User-Computer Interface