title: Does Changing Examiner Stations During UK Postgraduate Surgery Objective Structured Clinical Examinations Influence Examination Reliability and Candidates' Scores?
abstract: OBJECTIVE
  Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates' scores compared with examiners who examined the same scenario all day.
  
  
  METHODS
  An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach's alpha, and the comparative distribution of total candidates' scores for each day was evaluated using t-tests of unit-weighted z scores.
  
  
  RESULTS
  A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching.
  
  
  CONCLUSIONS
  This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond.
authors.affiliation: Intercollegiate Committee for Basic Surgical Examinations, The Royal College of Surgeons of England, London, United Kingdom. Electronic address: Peter.brennan@porthosp.nhs.uk.
authors.forename: Peter A
authors.initials: PA
authors.lastname: Brennan
authors.affiliation: Department of Quality Enhancement, The Royal College of Surgeons in Ireland, Dublin, Ireland.
authors.forename: David T
authors.initials: DT
authors.lastname: Croke
authors.affiliation: Dean, Brighton Medical School, Brighton, United Kingdom.
authors.forename: Malcolm
authors.initials: M
authors.lastname: Reed
authors.affiliation: Intercollegiate Committee for Basic Surgical Examinations, The Royal College of Surgeons of England, London, United Kingdom.
authors.forename: Lee
authors.initials: L
authors.lastname: Smith
authors.affiliation: Intercollegiate Committee for Basic Surgical Examinations, The Royal College of Surgeons of England, London, United Kingdom.
authors.forename: Euan
authors.initials: E
authors.lastname: Munro
authors.affiliation: Intercollegiate Committee for Basic Surgical Examinations, The Royal College of Surgeons of England, London, United Kingdom.
authors.forename: John
authors.initials: J
authors.lastname: Foulkes
authors.affiliation: Department of Quality Enhancement, The Royal College of Surgeons in Ireland, Dublin, Ireland.
authors.forename: Richard
authors.initials: R
authors.lastname: Arnett
mesh_terms: D002983:Clinical Competence; D004503:Education, Medical, Graduate; D004521:Educational Measurement / Q000379:methods*; D013502:General Surgery / Q000193:education*; D006801:Humans; D015203:Reproducibility of Results; D006113:United Kingdom
keywords: Interpersonal and Communication Skills; Medical Knowledge; Practice-Based Learning and Improvement; Professionalism; Systems-Based Practice; examiner; objective structured clinical examination; reliability; scenario; surgery