title: Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations.
abstract: BACKGROUND
  Bias of clinical examiners against some types of candidate, based on characteristics such as sex or ethnicity, would represent a threat to the validity of an examination, since sex or ethnicity are 'construct-irrelevant' characteristics. In this paper we report a novel method for assessing sex and ethnic bias in over 2000 examiners who had taken part in the PACES and nPACES (new PACES) examinations of the MRCP(UK).
  
  
  METHODS
  PACES and nPACES are clinical skills examinations that have two examiners at each station who mark candidates independently. Differences between examiners cannot be due to differences in performance of a candidate because that is the same for the two examiners, and hence may result from bias or unreliability on the part of the examiners. By comparing each examiner against a 'basket' of all of their co-examiners, it is possible to identify examiners whose behaviour is anomalous. The method assessed hawkishness-doveishness, sex bias, ethnic bias and, as a control condition to assess the statistical method, 'even-number bias' (i.e. treating candidates with odd and even exam numbers differently). Significance levels were Bonferroni corrected because of the large number of examiners being considered.
  
  
  RESULTS
  The results of 26 diets of PACES and six diets of nPACES were examined statistically to assess the extent of hawkishness, as well as sex bias and ethnicity bias in individual examiners. The control (odd-number) condition suggested that about 5% of examiners were significant at an (uncorrected) 5% level, and that the method therefore worked as expected. As in a previous study (BMC Medical Education, 2006, 6:42), some examiners were hawkish or doveish relative to their peers. No examiners showed significant sex bias, and only a single examiner showed evidence consistent with ethnic bias. A re-analysis of the data considering only one examiner per station, as would be the case for many clinical examinations, showed that analysis with a single examiner runs a serious risk of false positive identifications probably due to differences in case-mix and content-specificity.
  
  
  CONCLUSIONS
  In examinations where there are two independent examiners at a station, our method can assess the extent of bias against candidates with particular characteristics. The method would be far less sensitive in examinations with only a single examiner per station as examiner variance would be confounded with candidate performance variance. The method however works well when there is more than one examiner at a station and in the case of the current MRCP(UK) clinical examination, nPACES, found possible sex bias in no examiners and possible ethnic bias in only one.
authors.affiliation: Academic Centre for Medical Education, Division of Medical Education, University College London, Gower Street, London WC1E 6BT, UK. i.mcmanus@ucl.ac.uk
authors.forename: I C
authors.initials: IC
authors.lastname: McManus
authors.forename: Andrew T
authors.initials: AT
authors.lastname: Elder
authors.forename: Jane
authors.initials: J
authors.lastname: Dacre
mesh_terms: D002568:Certification / Q000379:methods / Q000592:standards; D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D005006:Ethnicity; D005260:Female; D006801:Humans; D008297:Male; D015588:Observer Variation; D063505:Racism; D063507:Sexism; D006113:United Kingdom
references.citation: Clin Med (Lond). 2011 Jun;11(3):231-4
references.pmid: 21902073
references.citation: Ann Acad Med Singap. 2011 Mar;40(3):119-25
references.pmid: 21603729
references.citation: BMJ. 2005 May 7;330(7499):1064-6
references.pmid: 15879391
references.citation: BMJ. 2011 Mar 08;342:d901
references.pmid: 21385802
references.citation: BMC Med Educ. 2006 Aug 18;6:42
references.pmid: 16919156
references.citation: J Appl Meas. 2003;4(4):386-422
references.pmid: 14523257
references.citation: Clin Teach. 2013 Feb;10(1):27-31
references.pmid: 23294740
references.citation: Med Educ. 2011 Dec;45(12):1181-9
references.pmid: 21988659
references.citation: Clin Med (Lond). 2003 Sep-Oct;3(5):452-9
references.pmid: 14601946
references.citation: BMC Med. 2007 May 03;5:8
references.pmid: 17477862
references.citation: BMC Med Educ. 2008 Apr 16;8:21
references.pmid: 18416818