Setting standards for performance tests: a pilot study of a three-level Angoff method.
BACKGROUND
The Yes/No Angoff method is a standard-setting variant in which judges estimate whether a borderline student would or would not accomplish each item. However, when there is a 50% chance of success, systematically severe or lenient ratings may skew the cut score. This study piloted a three-level Angoff in which items were assigned to "Yes," "No," or "Maybe" categories.


METHODS
Five judges provided three-level Angoff ratings for seven checklists used in a clinical skills exam for fourth-year students. Systematic severity and leniency were simulated by converting all "Maybe" ratings to "Yes" or "No" ratings, respectively.


RESULTS
Forty-one percent of 121 items had at least one "Maybe" rating. Ten percent of all ratings were in the "Maybe" category. One judge accounted for 83% of all "Maybe" ratings; three did not use it at all. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted.


CONCLUSIONS
The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate.
Yudkowsky
Rachel
R
UIC Department of Medical Education, 963 CME, 808 S. Wood, MC 591, Chicago, IL 60612, USA. rachely@uic.edu
Downing
Steven M
SM
Popescu
Mihaela
M
D002983:Clinical Competence; D004504:Education, Medical, Undergraduate; D006801:Humans; D007600:Judgment; D008487:Medical History Taking; D015588:Observer Variation; D016544:Patient Simulation; D010808:Physical Examination; D010865:Pilot Projects; D012015:Reference Standards; D015203:Reproducibility of Results