Standard setting: comparison of two methods.
BACKGROUND
The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard-setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method.


METHODS
The norm-reference method of standard-setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method.


RESULTS
The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% - 87%). The modified Angoff method had an inter-rater reliability of 0.81-0.82 and a test-retest reliability of 0.59-0.74.


CONCLUSIONS
There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability.
George
Sanju
S
Queen Elizabeth Psychiatric Hospital, Mindelsohn Way, Edgbaston, Birmingham, UK. sanju.george@talk21.com
Haque
M Sayeed
MS
Oyebode
Femi
F
D002983:Clinical Competence / Q000706:statistics & numerical data*; D032921:Consensus; D004504:Education, Medical, Undergraduate; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D007600:Judgment; D015588:Observer Variation; D011336:Probability; D011570:Psychiatry / Q000193:education*; D012015:Reference Standards; D013337:Students, Medical; D006113:United Kingdom