Test order in teacher-rated behavior assessments: Is counterbalancing necessary?
Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22% of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance. (PsycINFO Database Record
Department of Educational Psychology.
Janice
0000-0003-4815-5733
J
Kooken
School of Education, University of California, Davis.
Megan E
ME
Welsh
Department of Educational Psychology.
D Betsy
DB
McCoach
Department of Educational Psychology, University of Minnesota.
Faith G
FG
Miller
Department of Educational Psychology.
Sandra M
SM
Chafouleas
Department of Educational, School, and Counseling Psychology, University of Missouri.
T Chris
TC
Riley-Tillman
Department of Counseling, School, and Educational Psychology, University of Buffalo.
Gregory
G
Fabiano
D000293:Adolescent; D002648:Child; D002652:Child Behavior; D004644:Emotions; D006801:Humans; D008297:Male; D001523:Mental Disorders; D000066553:Problem Behavior; D000070777:School Teachers; D066107:Social Skills; D013334:Students / Q000523:psychology*