title: Insights into the Angoff method: results from a simulation study.
abstract: BACKGROUND
  In standard setting techniques involving panels of judges, the attributes of judges may affect the cut-scores. This simulation study modelled the effect of the number of judges and test items, as well as the impact of judges' attributes such as accuracy, stringency and influence on others on the precision of the cut-scores.
  
  
  METHODS
  Forty nine combinations of Angoff panels (N = 5, 10, 15, 20, 30, 50, and 80) and test items (n = 5, 10, 15, 20, 30, 50, and 80) were simulated. Each combination was simulated 100 times (in total 4,900 simulations). The simulation was of judges attributes: stringency, accuracy and leadership. Impact of judges attributes, number of judges, number of test items and Angoff's second (compared to the first) round on the precision of a panel's cut-score was measured by the deviation of the panel's cut-score from the cut-score's true value.
  
  
  RESULTS
  Findings from 4900 simulated panels supported Angoff being both reliable and valid. Unless the number of test items is small, panels of around 15 judges with mixed levels of expertise provide the most precise estimates. Furthermore, if test data were not presented, a second round of decision-making, as used in the modified Angoff, adds little to precision. A panel which has only experts or only non-experts yields a cut-score which is less precise than a cut-score yielded by a mixed-expertise panel, suggesting that optimal composition of an Angoff panel should include a range of judges with diverse expertise and stringency.
  
  
  CONCLUSIONS
  Simulations aim to improve our understanding of the models assessed but they do not describe natural phenomena as they do not use observed data. While the simulations undertaken in this study help clarify how to set cut-scores defensibly, it is essential to confirm these theories in practice.
authors.affiliation: University of New South Wales, Sydney, Australia. b.shulruf@unsw.edu.au.
authors.forename: Boaz
authors.initials: B
authors.lastname: Shulruf
authors.affiliation: Otago University Christchurch, Christchurch, New Zealand.
authors.forename: Tim
authors.initials: T
authors.lastname: Wilkinson
authors.affiliation: University of Auckland, Auckland, New Zealand.
authors.forename: Jennifer
authors.initials: J
authors.lastname: Weller
authors.affiliation: University of New South Wales, Sydney, Australia.
authors.forename: Philip
authors.initials: P
authors.lastname: Jones
authors.affiliation: University of Auckland, Auckland, New Zealand.
authors.forename: Phillippa
authors.initials: P
authors.lastname: Poole
mesh_terms: D003657:Decision Making; D004501:Education, Medical; D004521:Educational Measurement; D006801:Humans; D007600:Judgment; D007857:Leadership; D015588:Observer Variation; D018794:Problem-Based Learning; D015203:Reproducibility of Results