Validity threats: overcoming interference with proposed interpretations of assessment data.
BACKGROUND
Factors that interfere with the ability to interpret assessment scores or ratings in the proposed manner threaten validity. To be interpreted in a meaningful manner, all assessments in medical education require sound, scientific evidence of validity.


OBJECTIVE
The purpose of this essay is to discuss 2 major threats to validity: construct under-representation (CU) and construct-irrelevant variance (CIV). Examples of each type of threat for written, performance and clinical performance examinations are provided.


CONCLUSIONS
The CU threat to validity refers to undersampling the content domain. Using too few items, cases or clinical performance observations to adequately generalise to the domain represents CU. Variables that systematically (rather than randomly) interfere with the ability to meaningfully interpret scores or ratings represent CIV. Issues such as flawed test items written at inappropriate reading levels or statistically biased questions represent CIV in written tests. For performance examinations, such as standardised patient examinations, flawed cases or cases that are too difficult for student ability contribute CIV to the assessment. For clinical performance data, systematic rater error, such as halo or central tendency error, represents CIV. The term face validity is rejected as representative of any type of legitimate validity evidence, although the fact that the appearance of the assessment may be an important characteristic other than validity is acknowledged.


CONCLUSIONS
There are multiple threats to validity in all types of assessment in medical education. Methods to eliminate or control validity threats are suggested.
University of Illinois at Chicago, College of Medicine, Department of Medical Education, Chicago, Illinois 60612-7309, USA. sdowning@uic.edu
Steven M
SM
Downing
Thomas M
TM
Haladyna
D015982:Bias; D003627:Data Interpretation, Statistical; D004504:Education, Medical, Undergraduate / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D006801:Humans; D015203:Reproducibility of Results