title: Evaluation of an intelligent tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance.
abstract: OBJECTIVE
  Determine effects of computer-based tutoring on diagnostic performance gains, meta-cognition, and acceptance using two different problem representations. Describe impact of tutoring on spectrum of diagnostic skills required for task performance. Identify key features of student-tutor interaction contributing to learning gains.
  
  
  METHODS
  Prospective, between-subjects study, controlled for participant level of training. Resident physicians in two academic pathology programs spent four hours using one of two interfaces which differed mainly in external problem representation. The case-focused representation provided an open-learning environment in which students were free to explore evidence-hypothesis relationships within a case, but could not visualize the entire diagnostic space. The knowledge-focused representation provided an interactive representation of the entire diagnostic space, which more tightly constrained student actions.
  
  
  METHODS
  Metrics included results of pretest, post-test and retention-test for multiple choice and case diagnosis tests, ratios of performance to student reported certainty, results of participant survey, learning curves, and interaction behaviors during tutoring.
  
  
  RESULTS
  Students had highly significant learning gains after one tutoring session. Learning was retained at one week. There were no differences between the two interfaces in learning gains on post-test or retention test. Only students in the knowledge-focused interface exhibited significant metacognitive gains from pretest to post-test and pretest to retention test. Students rated the knowledge-focused interface significantly higher than the case-focused interface.
  
  
  CONCLUSIONS
  Cognitive tutoring is associated with improved diagnostic performance in a complex medical domain. The effect is retained at one-week post-training. Knowledge-focused external problem representation shows an advantage over case-focused representation for metacognitive effects and user acceptance.
authors.affiliation: Department of Biomedical Informatics, University of Pittsburgh School of Medicine, UPMC Shadyside Cancer Pavilion, Room 307, 5230 Centre Avenue, Pittsburgh, PA 15232, USA. crowleyrs@upmc.edu
authors.forename: Rebecca S
authors.initials: RS
authors.lastname: Crowley
authors.forename: Elizabeth
authors.initials: E
authors.lastname: Legowski
authors.forename: Olga
authors.initials: O
authors.lastname: Medvedeva
authors.forename: Eugene
authors.initials: E
authors.lastname: Tseytlin
authors.forename: Ellen
authors.initials: E
authors.lastname: Roh
authors.forename: Drazen
authors.initials: D
authors.lastname: Jukic
mesh_terms: D001292:Attitude to Computers; D003071:Cognition; D003194:Computer-Assisted Instruction; D003625:Data Collection; D003933:Diagnosis; D006801:Humans; D010336:Pathology / Q000193:education*; D018794:Problem-Based Learning; D013334:Students; D014584:User-Computer Interface