title: Two models of raters in a structured oral examination: does it make a difference?
abstract: Oral examinations have become more standardized over recent years. Traditionally a small number of raters were used for this type of examination. Past studies suggested that more raters should improve reliability. We compared the results of a multi-station structured oral examination using two different rater models, those based in a station, (station-specific raters), and those who follow a candidate throughout the entire examination, (candidate-specific raters).Two station-specific and two candidate-specific raters simultaneously evaluated internal medicine residents' performance at each station. No significant differences were found in examination scores. Reliability was higher for the candidate-specific raters. Inter-rater reliability, internal consistency and a study of station inter-correlations suggested that a halo effect may be present for candidates examined by candidate-specific raters. This study suggests that although the model of candidate-specific raters was more reliable than the model of station-specific raters for the overall examination, the presence of a halo effect may influence individual examination outcomes.
authors: Division of General Internal Medicine, Department of Medicine, University of Ottawa, Ottawa, ON, Canada. ctouchie@ottawahospital.on.ca.|The Ottawa Hospital, General Campus, 501 Smyth Road, LM-14, Ottawa, ON, K1H 8L6, Canada. ctouchie@ottawahospital.on.ca.
authors: Claire
authors: C
authors: Touchie
authors: Division of Rheumatology, Department of Medicine, University of Ottawa, Ottawa, ON, Canada.
authors: Susan
authors: S
authors: Humphrey-Murto
authors: Division of Respirology, Department of Medicine, University of Calgary, Calgary, AB, Canada.
authors: Martha
authors: M
authors: Ainslie
authors: Division of General Internal Medicine, Department of Medicine, University of Western Ontario, London, ON, Canada.
authors: Kathryn
authors: K
authors: Myers
authors: Medical Council of Canada, Ottawa, ON, Canada.
authors: Timothy J
authors: TJ
authors: Wood
mesh_terms: D002983:Clinical Competence; D004501:Education, Medical / Q000379:methods*; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results