Collecting Validity Evidence for the Assessment of Mastery Learning in Simulation-Based Ultrasound Training.
OBJECTIVE
To collect validity evidence for the assessment of mastery learning on a virtual reality transabdominal ultrasound simulator.


METHODS
We assessed the validity evidence using Messick's framework for validity. The study included 20 novices and 9 ultrasound experts who all completed 10 obstetric training modules on a transabdominal ultrasound simulator that provided automated measures of performance for each completed module (i. e., simulator metrics). Differences in the performance of the two groups were used to identify simulator metrics with validity evidence for the assessment of mastery learning. The novices continued to practice until they had attained mastery learning level.


RESULTS
One-third of the simulator metrics discriminated between the two groups. The median simulator scores from a maximum of 40 metrics were 17.5 percent (range 0 - 45.0 percent) for novices and 90.0 percent (range 85.0 - 97.5) for experts, p < 0.001. Internal consistency was high, with a Cronbach's alpha value of 0.98. The test/retest reliability gave an intra-class correlation coefficient (ICC) of 0.62 for novices who reached the mastery learning level twice. Novices reached the mastery learning level within a median of 4 attempts (range 3 - 8) corresponding to a median of 252 minutes of simulator training (range 211 - 394 minutes).


CONCLUSIONS
This study found that validity evidence for the assessment of mastery learning in simulation-based ultrasound training can be demonstrated and that ultrasound novices can attain mastery learning levels with less than 5 hours of training. Only one-third of the standard simulator metrics discriminated between different levels of competence.
Juliane Marie Centre, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.
L
L
Dyre
Department of Gynaecology and Obstetrics, University of Copenhagen, Nordsjaelland Hospital, Hilleroed, Denmark.
L N
LN
Nørgaard
Juliane Marie Centre, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.
A
A
Tabor
Juliane Marie Centre, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.
M E
ME
Madsen
Juliane Marie Centre, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.
J L
JL
Sørensen
Faculty of Health, University of Aarhus, Denmark.
C
C
Ringsted
Juliane Marie Centre, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.
M
M
Tolsgaard
D000005:Abdomen / Q000000981:diagnostic imaging*; D000328:Adult; D002983:Clinical Competence; D003479:Curriculum; D003718:Denmark; D004501:Education, Medical; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D014463:Ultrasonography; D016216:Ultrasonography, Prenatal; D014584:User-Computer Interface