Locating, characterizing and minimizing sources of error for a paper case-based structured oral examination in a multi-campus clerkship.
OBJECTIVE
To determine consistency of assessment in a new paper case-based structured oral examination in a multi-community pediatrics clerkship, and to identify correctable problems in the administration of examination and assessment process.


METHODS
Nine paper case-based oral examinations were audio-taped. From audio-tapes five community coordinators scored examiner behaviors and graded student performance. Correlations among examiner behaviors scores were examined. Graphs identified grading patterns of evaluators. The effect of exam-giving on evaluators was assessed by t-test. Reliability of grades was calculated and the effect of reducing assessment problems was modeled.


RESULTS
Exam-givers differed most in their "teaching-guiding" behavior, and this negatively correlated with student grades. Exam reliability was lowered mainly by evaluator differences in leniency and grading pattern; less important was absence of standardization in cases.


CONCLUSIONS
While grade reliability was low in early use of the paper case-based oral examination, modeling of plausible effects of training and monitoring for greater uniformity in administration of the examination and assigning scores suggests that more adequate reliabilities can be attained.
Kumar
A
A
Department of Pediatrics and Human Development, C-203 East Fee Hall, Michigan State University, East Lansing, MI 48824-1316, USA. kumara@msu.edu
Bridgham
R
R
Potts
M
M
Gushurst
C
C
Hamp
M
M
Passal
D
D
D002982:Clinical Clerkship; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D006040:Goals; D006801:Humans; D010372:Pediatrics; D015203:Reproducibility of Results; D013337:Students, Medical; D014481:United States