title: A pilot study of marking accuracy and mental workload as measures of OSCE examiner performance.
abstract: BACKGROUND
  The Objective Structured Clinical Examination (OSCE) is now a standard assessment format and while examiner training is seen as essential to assure quality, there appear to be no widely accepted measures of examiner performance.
  
  
  METHODS
  The objective of this study was to determine whether the routine training provided to examiners improved their accuracy and reduced their mental workload. Accuracy was defined as the difference between the rating of each examiner and that of an expert group expressed as the mean error per item. At the same time the mental workload of each examiner was measured using a previously validated secondary task methodology.
  
  
  RESULTS
  Training was not associated with an improvement in accuracy (p = 0.547) and that there was no detectable effect on mental workload. However, accuracy was improved after exposure to the same scenario (p < 0.001) and accuracy was greater when marking an excellent compared to a borderline performance.
  
  
  CONCLUSIONS
  This study suggests that the method of training OSCE examiners studied is not effective in improving their performance, but that average item accuracy and mental workload appear to be valid methods of assessing examiner performance.
authors.affiliation: Abertawe Bro Morgannwg University Local Health Board, Swansea, UK. aidanbyrne27@gmail.com.|Department of Anaesthesia, Morriston Hospital, Swansea, SA6 6NL, UK. aidanbyrne27@gmail.com.
authors.forename: Aidan
authors.identifier: http://orcid.org/0000-0001-9289-1995
authors.initials: A
authors.lastname: Byrne
authors.affiliation: Abertawe Bro Morgannwg University Local Health Board, Swansea, UK.
authors.forename: Tereza
authors.initials: T
authors.lastname: Soskova
authors.affiliation: Abertawe Bro Morgannwg University Local Health Board, Swansea, UK.
authors.forename: Jayne
authors.initials: J
authors.lastname: Dawkins
authors.affiliation: Cardiff University, School of Medicine, Cardiff, UK.
authors.forename: Lee
authors.initials: L
authors.lastname: Coombes
mesh_terms: D004521:Educational Measurement / Q000592:standards*; D006801:Humans; D010865:Pilot Projects; D011361:Professional Competence / Q000592:standards*; D011930:Reaction Time; D015203:Reproducibility of Results; D012107:Research Design; D013337:Students, Medical; D013647:Task Performance and Analysis; D016526:Workload / Q000523:psychology*
keywords: Examiner accuracy; Examiner training; Mental workload; OSCE