Medical students' ratings of faculty teaching in a multi-instructor setting: an examination of monotonic response patterns.
Realizing that the psychometric properties of a measure may be highly variable is especially relevant in a multi-instructor context, since an implicit assumption is that student ratings are equally reliable and valid for all faculty ratees. As a possible indicator of nonattending (i.e. invalid) responses, the authors examined the effects of monotonic response patterns on the reliabilities of students' ratings of faculty teaching - including how an alternative presentation format may reduce the prevalence of this behavior. Second-year medical and dental students (n = 130) enrolled in a required basic science course during the 1998-99 academic year were randomly assigned to one of two groups - each of which evaluated the teaching of 6 different faculty across 6 distinct dimensions (i.e. overall quality, organization, preparation, stimulation, respectfulness, and helpfulness). Using a 'split ballot' design, two versions of the conceptually equivalent faculty evaluation form were distributed at random to students in each group. Form A contained the 'traditional' items-within-faculty format, while Form B listed faculty-within-item.The number of monotonic forms (i.e. the identical rating of all 6 items) varied measurably across faculty ratees, as did the respective effects on scale reliabilities. Alpha was especially inflated where a sizeable proportion of monotonic patterns were located on response categories that were either very high (> +1.28 z(m) deviations) or very low (< -1.28 z(m) deviations) compared to the group mean. Lastly, the prevalence of monotonic response patterns was significantly (p = < or = 0.01) less when a faculty-within-item format is used (Form B). These findings suggest that monotonic response patterns differentially impact the reliabilities and, hence, the validity of students' ratings of individual faculty in a multi-instructor context.
Stratton
Terry D
TD
Office of Academic Affairs, Division of Testing and Evaluation, University of Kentucky College of Medicine, UKMC Room MN104, Lexington, KY, USA.
Witzke
Donald B
DB
Jacob
Robert J
RJ
Sauer
Marlene J
MJ
Murphy-Spencer
Amy
A
D001291:Attitude of Health Personnel; D004497:Education, Dental / Q000592:standards; D004501:Education, Medical / Q000592:standards; D005180:Faculty, Medical / Q000592:standards*; D006801:Humans; D011361:Professional Competence / Q000706:statistics & numerical data*; D011594:Psychometrics; D015203:Reproducibility of Results; D015144:Southeastern United States; D013337:Students, Medical / Q000523:psychology*; D011795:Surveys and Questionnaires; D013663:Teaching / Q000592:standards*