title: The FRCR 2B oral examination: is it reliable?
abstract: OBJECTIVE
  To investigate the reliability of the oral component of the Fellowship of the Royal College of Radiologists (FRCR) 2B examination.
  
  
  METHODS
  Anonymized candidate test scores were analysed from nine consecutive sittings of the FRCR 2B oral examination covering the period from spring 2006 to spring 2010. Interobserver reliability was assessed using weighted Kappa coefficient, intraclass correlation coefficient, and a modified Bland-Altman plot.
  
  
  RESULTS
  During the study period, 2235 candidates sat the FRCR 2B examination. Eighty-five point one percent of candidates obtained paired oral assessment scores within one mark of each other. This figure rises to 95.7% for paired scores within 1.5 marks of each other. Mean difference in scores was 0.67 (95% CI: 0.65-0.70). Agreement rises at the extremities of mean score. Reliability coefficients for the FRCR 2B oral examination were calculated as 0.27 (weighted Kappa) and 0.44 (intraclass correlation coefficient).
  
  
  CONCLUSIONS
  The calculated reliability coefficients indicate fair to, at best, moderate interobserver reliability in the FRCR 2B oral examination. These findings are disappointing but are comparable with other oral assessment reliability studies. There is scope for improvement, although further work to measure the combined reliability of all the components of the FRCR 2B examination is desirable. Measures that could potentially increase reliability must be carefully considered against any negative impact on test validity, acceptability, and cost.
authors.affiliation: Department of Radiology, Royal Victoria Infirmary, Newcastle upon Tyne, UK. andrewyeung@hotmail.com
authors.forename: A
authors.initials: A
authors.lastname: Yeung
authors.forename: T C
authors.initials: TC
authors.lastname: Booth
authors.forename: T J
authors.initials: TJ
authors.lastname: Larkin
authors.forename: P
authors.initials: P
authors.lastname: McCoubrie
authors.forename: L
authors.initials: L
authors.lastname: McKnight
mesh_terms: D002983:Clinical Competence / Q000706:statistics & numerical data*; D004503:Education, Medical, Graduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D005257:Fellowships and Scholarships; D006801:Humans; D015588:Observer Variation; D011871:Radiology / Q000193:education*; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012955:Societies, Medical; D011795:Surveys and Questionnaires; D006113:United Kingdom; D014705:Verbal Behavior