title: Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview?
abstract: BACKGROUND
  There are significant levels of variation in candidate multiple mini-interview (MMI) scores caused by interviewer-related factors. Multi-facet Rasch modelling (MFRM) has the capability to both identify these sources of error and partially adjust for them within a measurement model that may be fairer to the candidate.
  
  
  METHODS
  Using facets software, a variance components analysis estimated sources of measurement error that were comparable with those produced by generalisability theory. Fair average scores for the effects of the stringency/leniency of interviewers and question difficulty were calculated and adjusted rankings of candidates were modelled.
  
  
  RESULTS
  The decisions of 207 interviewers had an acceptable fit to the MFRM model. For one candidate assessed by one interviewer on one MMI question, 19.1% of the variance reflected candidate ability, 8.9% reflected interviewer stringency/leniency, 5.1% reflected interviewer question-specific stringency/leniency and 2.6% reflected question difficulty. If adjustments were made to candidates' raw scores for interviewer stringency/leniency and question difficulty, 11.5% of candidates would see a significant change in their ranking for selection into the programme. Greater interviewer leniency was associated with the number of candidates interviewed.
  
  
  CONCLUSIONS
  Interviewers differ in their degree of stringency/leniency and this appears to be a stable characteristic. The MFRM provides a recommendable way of giving a candidate score which adjusts for the stringency/leniency of whichever interviewers the candidate sees and the difficulty of the questions the candidate is asked.
authors.affiliation: Sydney Medical School-Northern, University of Sydney, Sydney, New South Wales, Australia. christopher.roberts@sydney.edu.au
authors.forename: Chris
authors.initials: C
authors.lastname: Roberts
authors.forename: Imogene
authors.initials: I
authors.lastname: Rothnie
authors.forename: Nathan
authors.initials: N
authors.lastname: Zoanetti
authors.forename: Jim
authors.initials: J
authors.lastname: Crossley
mesh_terms: D002983:Clinical Competence; D003142:Communication; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D005180:Faculty, Medical; D006801:Humans; D007407:Interviews as Topic; D015588:Observer Variation; D011594:Psychometrics / Q000379:methods; D012570:School Admission Criteria