Standards and reliability in evaluation: when rules of thumb don't apply.
The purpose of this paper is to identify situations in which two rules of thumb in evaluation do not apply. The first rule is that all standards should be absolute. When selection decisions are being made or when classroom tests are given, however, relative standards may be better. The second rule of thumb is that every test should have a reliability of .80 or better. Depending on the circumstances, though, the standard error of measurement, the consistency of pass/fail classifications, and the domain-referenced reliability coefficients may be better indicators of reproducibility.
Norcini
J J
JJ
Institute for Clinical Evaluation, American Board of Internal Medicine, Philadelphia, Pennsylvania 19106-3699, USA. jnorcini@abim.org
D004501:Education, Medical; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D011594:Psychometrics / Q000379:methods*; D015203:Reproducibility of Results; D013044:Specialty Boards