title: Achieving acceptable reliability in oral examinations: an analysis of the Royal College of General Practitioners membership examination's oral component.
abstract: BACKGROUND
  The membership examination of the Royal College of General Practitioners (RCGP) uses structured oral examinations to assess candidates' decision making skills and professional values.
  
  
  OBJECTIVE
  To estimate three indices of reliability for these oral examinations.
  
  
  METHODS
  In summer 1998, a revised system was introduced for the oral examinations. Candidates took two 20-minute (five topic) oral examinations with two examiner pairs. Areas for oral topics had been identified. Examiners set their own topics in three competency areas (communication, professional values and personal development) and four contexts (patient, teamwork, personal, society). They worked in two pairs (a quartet) to preplan questions on 10 topics. The results were analysed in detail. Generalisability theory was used to estimate three indices of reliability: (A) intercase (B) pass/fail decision and (C) standard error of measurement (SEM). For each index, a benchmark requirement was preset at (A) 0.8 (B) 0.9 and (C) 0.5.
  
  
  RESULTS
  There were 896 candidates in total. Of these, 87 candidates (9.7%) failed. Total score variance was attributed to: 41% candidates, 32% oral content, 27% examiners and general error. Reliability coefficients were: (A) intercase 0.65; (B) pass/fail 0.85. The SEM was 0.52 (i.e. precise enough to distinguish within one unit on the rating scale). Extending testing time to four 20-minute oral examinations, each with two examiners, or five orals, each with one examiner, would improve intercase and pass/fail reliabilities to 0.78 and 0.94, respectively.
  
  
  CONCLUSIONS
  Structured oral examinations can achieve reliabilities appropriate to high stakes examinations if sufficient resources are available.
authors.affiliation: Department of General Practice and Primary Care, Guy's, King's and St Thomas' School of Medicine, London, UK. valerie.wass@kcl.ac.uk
authors.forename: Val
authors.initials: V
authors.lastname: Wass
authors.forename: Richard
authors.initials: R
authors.lastname: Wakeford
authors.forename: Roger
authors.initials: R
authors.lastname: Neighbour
authors.forename: Cees
authors.initials: C
authors.lastname: Van der Vleuten
mesh_terms: D002983:Clinical Competence / Q000592:standards; D003657:Decision Making; D004504:Education, Medical, Undergraduate / Q000379:methods / Q000592:standards*; D004521:Educational Measurement; D004739:England; D006801:Humans; D015203:Reproducibility of Results; D013060:Speech