title: Evaluating response shift in training evaluation: comparing the retrospective pretest with an adapted measurement invariance approach in a classroom management training program.
abstract: BACKGROUND
  Response shift (RS) can threaten the internal validity of pre-post designs. As RS may indicate a redefinition of the target construct, its occurrence in training evaluation is rather likely. The most common approach to deal with RS is to implement a retrospective pretest (then-test) instead of the traditional pre-test. In health psychology, an adapted measurement invariance approach (MIad) was developed as an alternative technique to study RS. Results produced by identifying RS with the two approaches were rarely studied simultaneously or within an experimental framework.
  
  
  OBJECTIVE
  To study RS in two different treatment conditions and compare results produced by both techniques in identifying various types of RS. We further studied validity aspects of the then-test.
  
  
  METHODS
  We evaluated RS by applying the then-test procedure (TP) and the measurement invariance apporach MIad within an experimental design: Participants either attended a short-term or a long-term classroom management training program.
  
  
  METHODS
  Participants were 146 student teachers in their first year of master's study.
  
  
  METHODS
  Pre (before training), post, and then self-ratings (after training) on classroom management knowledge were administered.
  
  
  RESULTS
  Results indicated that the two approaches do not yield the same results. The MIad identified more and also group-specific RS as opposed to the findings of the TP, which found less and only little evidence for group-specific RS.
  
  
  CONCLUSIONS
  Further research is needed to study the usability and validity of the respective approaches. In particular, the usability of the then-test seems to be challenged.
authors.affiliation: Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany v.piwowar@fu-berlin.de.
authors.forename: Valentina
authors.initials: V
authors.lastname: Piwowar
authors.affiliation: Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany.
authors.forename: Felicitas
authors.initials: F
authors.lastname: Thiel
mesh_terms: D004500:Education, Graduate; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D015397:Program Evaluation; D012107:Research Design; D013663:Teaching
keywords: classroom management; pre–post design; response shift; retrospective pretest; training evaluation