Achieving acceptable reliability in oral examinations: an analysis of the Royal College of General Practitioners membership examination's oral component.
BACKGROUND
The membership examination of the Royal College of General Practitioners (RCGP) uses structured oral examinations to assess candidates' decision making skills and professional values.


OBJECTIVE
To estimate three indices of reliability for these oral examinations.


METHODS
In summer 1998, a revised system was introduced for the oral examinations. Candidates took two 20-minute (five topic) oral examinations with two examiner pairs. Areas for oral topics had been identified. Examiners set their own topics in three competency areas (communication, professional values and personal development) and four contexts (patient, teamwork, personal, society). They worked in two pairs (a quartet) to preplan questions on 10 topics. The results were analysed in detail. Generalisability theory was used to estimate three indices of reliability: (A) intercase (B) pass/fail decision and (C) standard error of measurement (SEM). For each index, a benchmark requirement was preset at (A) 0.8 (B) 0.9 and (C) 0.5.


RESULTS
There were 896 candidates in total. Of these, 87 candidates (9.7%) failed. Total score variance was attributed to: 41% candidates, 32% oral content, 27% examiners and general error. Reliability coefficients were: (A) intercase 0.65; (B) pass/fail 0.85. The SEM was 0.52 (i.e. precise enough to distinguish within one unit on the rating scale). Extending testing time to four 20-minute oral examinations, each with two examiners, or five orals, each with one examiner, would improve intercase and pass/fail reliabilities to 0.78 and 0.94, respectively.


CONCLUSIONS
Structured oral examinations can achieve reliabilities appropriate to high stakes examinations if sufficient resources are available.
Wass
Val
V
Department of General Practice and Primary Care, Guy's, King's and St Thomas' School of Medicine, London, UK. valerie.wass@kcl.ac.uk
Wakeford
Richard
R
Neighbour
Roger
R
Van der Vleuten
Cees
C
D002983:Clinical Competence / Q000592:standards; D003657:Decision Making; D004504:Education, Medical, Undergraduate / Q000379:methods / Q000592:standards*; D004521:Educational Measurement; D004739:England; D006801:Humans; D015203:Reproducibility of Results; D013060:Speech