The reliability of summative judgements based on objective structured clinical examination cases distributed across the clinical year.
BACKGROUND
Objective structured clinical examinations (OSCEs) can be used for formative and summative evaluation. We sought to determine the generalisability of students' summary scores aggregated from formative OSCE cases distributed across 5 clerkships during Year 3 of medical school.


METHODS
Five major clerkships held OSCEs with 2-4 cases each during their rotations. All cases used 15-minute student-standardised patient encounters and performance was assessed using clinical and communication skills checklists. As not all students completed every clerkship or OSCE case, the generalisability (G) study was an unbalanced student x (case : clerkship) design. After completion of the G study, a decision (D) study was undertaken and phi (phi) values for different cut-points were calculated.


RESULTS
The data for this report were collected over 2 academic years involving 262 Year 3 students. The G study found that 9.7% of the score variance originated from the student, 3.1% from the student-clerkship interaction, and 87.2% from the student-case nested within clerkship effect. Using the variance components from the G study, the D study suggested that if students completed 3 OSCE cases in each of the 5 different clerkships, the reliability of the aggregated scores would be 0.63. The phi, calculated at a cut-point 1 standard deviation below the mean, would be approximately 0.85.


CONCLUSIONS
Aggregating case scores from low stakes OSCEs within clerkships results in a score set that allows for very reliable decisions about which students are performing poorly. Medical schools can use OSCE case scores collected over a clinical year for summative evaluation.
Bergus
George R
GR
Department of Family Medicine, Carver College of Medicine, University of Iowa, Iowa City, Iowa 52245, USA. george-bergus@uiowa.edu
Kreiter
Clarence D
CD
D002982:Clinical Clerkship / Q000592:standards*; D002983:Clinical Competence / Q000592:standards*; D006801:Humans; D007484:Iowa; D007600:Judgment; D012680:Sensitivity and Specificity; D013337:Students, Medical