The dependability of medical students' performance ratings as documented on in-training evaluations.
OBJECTIVE
To demonstrate an approach to obtain an unbiased estimate of the dependability of students' performance ratings during training, when the data-collection design includes nesting of student in rater, unbalanced nest sizes, and dependent observations.


METHODS
In 2003, two variance components analyses of in-training evaluation (ITE) report data were conducted using urGENOVA software. In the first analysis, the dependability for the nested and unbalanced data-collection design was calculated. In the second analysis, an approach using multiple generalizability studies was used to obtain an unbiased estimate of the student variance component, resulting in an unbiased estimate of dependability.


RESULTS
Results suggested that there is bias in estimates of the dependability of students' performance on ITEs that are attributable to the data-collection design. When the bias was corrected, the results indicated that the dependability of ratings of student performance was almost zero.


CONCLUSIONS
The combination of the multiple generalizability studies method and the use of specialized software provides an unbiased estimate of the dependability of ratings of student performance on ITE scores for data-collection designs that include nesting of student in rater, unbalanced nest sizes, and dependent observations.
van Barneveld
Christina
C
Faculty of Education, Lakehead University, 955 Oliver Road, Thunder Bay, ON, Canada, P7B 5E1. cvanbarn@lakeheadu.ca
D000704:Analysis of Variance; D015982:Bias; D002982:Clinical Clerkship; D004521:Educational Measurement / Q000706:statistics & numerical data*; D005163:Factor Analysis, Statistical; D006801:Humans; D007388:Internal Medicine / Q000193:education; D015203:Reproducibility of Results