title: Setting standards for performance tests: a pilot study of a three-level Angoff method.
abstract: BACKGROUND
  The Yes/No Angoff method is a standard-setting variant in which judges estimate whether a borderline student would or would not accomplish each item. However, when there is a 50% chance of success, systematically severe or lenient ratings may skew the cut score. This study piloted a three-level Angoff in which items were assigned to "Yes," "No," or "Maybe" categories.
  
  
  METHODS
  Five judges provided three-level Angoff ratings for seven checklists used in a clinical skills exam for fourth-year students. Systematic severity and leniency were simulated by converting all "Maybe" ratings to "Yes" or "No" ratings, respectively.
  
  
  RESULTS
  Forty-one percent of 121 items had at least one "Maybe" rating. Ten percent of all ratings were in the "Maybe" category. One judge accounted for 83% of all "Maybe" ratings; three did not use it at all. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted.
  
  
  CONCLUSIONS
  The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate.
authors.affiliation: UIC Department of Medical Education, 963 CME, 808 S. Wood, MC 591, Chicago, IL 60612, USA. rachely@uic.edu
authors.forename: Rachel
authors.initials: R
authors.lastname: Yudkowsky
authors.forename: Steven M
authors.initials: SM
authors.lastname: Downing
authors.forename: Mihaela
authors.initials: M
authors.lastname: Popescu
mesh_terms: D002983:Clinical Competence; D004504:Education, Medical, Undergraduate; D006801:Humans; D007600:Judgment; D008487:Medical History Taking; D015588:Observer Variation; D016544:Patient Simulation; D010808:Physical Examination; D010865:Pilot Projects; D012015:Reference Standards; D015203:Reproducibility of Results