title: Modifying Hofstee standard setting for assessments that vary in difficulty, and to determine boundaries for different levels of achievement.
abstract: BACKGROUND
  Fixed mark grade boundaries for non-linear assessment scales fail to account for variations in assessment difficulty. Where assessment difficulty varies more than ability of successive cohorts or the quality of the teaching, anchoring grade boundaries to median cohort performance should provide an effective method for setting standards.
  
  
  METHODS
  This study investigated the use of a modified Hofstee (MH) method for setting unsatisfactory/satisfactory and satisfactory/excellent grade boundaries for multiple choice question-style assessments, adjusted using the cohort median to obviate the effect of subjective judgements and provision of grade quotas.
  
  
  RESULTS
  Outcomes for the MH method were compared with formula scoring/correction for guessing (FS/CFG) for 11 assessments, indicating that there were no significant differences between MH and FS/CFG in either the effective unsatisfactory/satisfactory grade boundary or the proportion of unsatisfactory graded candidates (p > 0.05). However the boundary for excellent performance was significantly higher for MH (p < 0.01), and the proportion of candidates returned as excellent was significantly lower (p < 0.01). MH also generated performance profiles and pass marks that were not significantly different from those given by the Ebel method of criterion-referenced standard setting.
  
  
  CONCLUSIONS
  This supports MH as an objective model for calculating variable grade boundaries, adjusted for test difficulty. Furthermore, it easily creates boundaries for unsatisfactory/satisfactory and satisfactory/excellent performance that are protected against grade inflation. It could be implemented as a stand-alone method of standard setting, or as part of the post-examination analysis of results for assessments for which pre-examination criterion-referenced standard setting is employed.
authors.affiliation: Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Peninsula Schools of Medicine and Dentistry, Plymouth University, Devon, PL4 8AA, UK.
authors.forename: Steven A
authors.initials: SA
authors.lastname: Burr
authors.affiliation: School of Medicine, University of Nottingham, Queen's Medical Centre, Nottingham, NG7 2UH, UK.
authors.forename: John
authors.initials: J
authors.lastname: Whittle
authors.affiliation: School of Life Sciences, University of Nottingham, Queen's Medical Centre, Nottingham, NG7 2UH, UK.
authors.forename: Lucy C
authors.initials: LC
authors.lastname: Fairclough
authors.affiliation: Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Peninsula Schools of Medicine and Dentistry, Plymouth University, Devon, PL4 8AA, UK.|Current address: Institute of Medical Education, School of Medicine, University of Cardiff, Cardiff, CF14 4YS, UK.
authors.forename: Lee
authors.initials: L
authors.lastname: Coombes
authors.affiliation: School of Life Sciences, University of Nottingham, Queen's Medical Centre, Nottingham, NG7 2UH, UK. ian.todd@nottingham.ac.uk.
authors.forename: Ian
authors.initials: I
authors.lastname: Todd
mesh_terms: D004504:Education, Medical, Undergraduate / Q000592:standards* / Q000706:statistics & numerical data; D004521:Educational Measurement / Q000379:methods / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D017145:Models, Educational; D013337:Students, Medical; D006113:United Kingdom
references.citation: Lancet. 2001 Mar 24;357(9260):945-9
references.pmid: 11289364
references.citation: Med Educ. 2002 Sep;36(9):860-7
references.pmid: 12354249
references.citation: Acad Med. 2005 Oct;80(10 Suppl):S63-6
references.pmid: 16199461
references.citation: Am J Surg. 2007 Feb;193(2):252-7
references.pmid: 17236857
references.citation: Med Educ. 2008 Jan;42(1):82-8
references.pmid: 18181848
references.citation: Med Teach. 2014 Apr;36(4):358-9
references.pmid: 24548113
references.citation: Med Teach. 2010;32(2):154-60
references.pmid: 20163232
references.citation: Med Teach. 2010;32(2):174-6
references.pmid: 20163238
references.citation: Med Teach. 2014 Feb;36(2):181-2
references.pmid: 24156828
references.citation: Med Teach. 2014 Feb;36(2):97-110
references.pmid: 24256050
references.citation: Med Teach. 2008;30(9-10):836-45
references.pmid: 19117221