title: Locating, characterizing and minimizing sources of error for a paper case-based structured oral examination in a multi-campus clerkship.
abstract: OBJECTIVE
  To determine consistency of assessment in a new paper case-based structured oral examination in a multi-community pediatrics clerkship, and to identify correctable problems in the administration of examination and assessment process.
  
  
  METHODS
  Nine paper case-based oral examinations were audio-taped. From audio-tapes five community coordinators scored examiner behaviors and graded student performance. Correlations among examiner behaviors scores were examined. Graphs identified grading patterns of evaluators. The effect of exam-giving on evaluators was assessed by t-test. Reliability of grades was calculated and the effect of reducing assessment problems was modeled.
  
  
  RESULTS
  Exam-givers differed most in their "teaching-guiding" behavior, and this negatively correlated with student grades. Exam reliability was lowered mainly by evaluator differences in leniency and grading pattern; less important was absence of standardization in cases.
  
  
  CONCLUSIONS
  While grade reliability was low in early use of the paper case-based oral examination, modeling of plausible effects of training and monitoring for greater uniformity in administration of the examination and assigning scores suggests that more adequate reliabilities can be attained.
authors.affiliation: Department of Pediatrics and Human Development, C-203 East Fee Hall, Michigan State University, East Lansing, MI 48824-1316, USA. kumara@msu.edu
authors.forename: A
authors.initials: A
authors.lastname: Kumar
authors.forename: R
authors.initials: R
authors.lastname: Bridgham
authors.forename: M
authors.initials: M
authors.lastname: Potts
authors.forename: C
authors.initials: C
authors.lastname: Gushurst
authors.forename: M
authors.initials: M
authors.lastname: Hamp
authors.forename: D
authors.initials: D
authors.lastname: Passal
mesh_terms: D002982:Clinical Clerkship; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D006040:Goals; D006801:Humans; D010372:Pediatrics; D015203:Reproducibility of Results; D013337:Students, Medical; D014481:United States