A surgery oral examination: interrater agreement and the influence of rater characteristics.
BACKGROUND
Poor interrater reliability is a common objection to the use of oral examinations.


METHODS
In 1990 the authors measured the agreement of 140 U.S. and Canadian surgical raters and the influences, if any, of age, years in practice, and experience as an examiner on individual oral examination scores. Eight actor examinees memorized transcripts of actual oral examinations and were videotaped using a single examiner. Examinee verbal style, dress, content of answers, and gender were purposefully adjusted. A repeated-measures analysis of variance was used for data analysis.


RESULTS
Three aspects of examinee performance influenced scores (verbal style, dress, and content of answers). No rater characteristic significantly affected scores. Raters showed high agreement (86%) when rating "good" performances but less agreement (67%) when rating "poor" performances.


CONCLUSIONS
The oral examination scores were not influenced by rater selection. The raters ranked good performances more consistently than poor performances. Therefore, more than one examiner appears necessary to confirm a poor performance during an examination.
Burchard
K W
KW
Department of Surgery, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire 03756-0001, USA.
Rowland-Morin
P A
PA
Coe
N P
NP
Garb
J L
JL
D000367:Age Factors; D000704:Analysis of Variance; D002170:Canada; D003020:Clothing; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D013502:General Surgery / Q000193:education*; D006801:Humans; D007398:Interpersonal Relations; D008297:Male; D008875:Middle Aged; D015588:Observer Variation; D011364:Professional Practice; D015203:Reproducibility of Results; D014481:United States; D014705:Verbal Behavior; D014743:Videotape Recording