title: A Simulation Study of Threats to Validity in Quasi-Experimental Designs: Interrelationship between Design, Measurement, and Analysis.
abstract: The Campbellian tradition provides a conceptual framework to assess threats to validity. On the other hand, different models of causal analysis have been developed to control estimation biases in different research designs. However, the link between design features, measurement issues, and concrete impact estimation analyses is weak. In order to provide an empirical solution to this problem, we use Structural Equation Modeling (SEM) as a first approximation to operationalize the analytical implications of threats to validity in quasi-experimental designs. Based on the analogies established between the Classical Test Theory (CTT) and causal analysis, we describe an empirical study based on SEM in which range restriction and statistical power have been simulated in two different models: (1) A multistate model in the control condition (pre-test); and (2) A single-trait-multistate model in the control condition (post-test), adding a new mediator latent exogenous (independent) variable that represents a threat to validity. Results show, empirically, how the differences between both the models could be partially or totally attributed to these threats. Therefore, SEM provides a useful tool to analyze the influence of potential threats to validity.
authors.affiliation: Metodología de las Ciencias del Comportamiento, Universidad Nacional de Educación a Distancia Madrid, Spain.
authors.forename: Fco P
authors.initials: FP
authors.lastname: Holgado-Tello
authors.affiliation: HUM-649, Innoevalua, Psicología Experimental, Universidad de SevillaSevilla, Spain; Universidad Autónoma de ChileSantiago, Chile.
authors.forename: Salvador
authors.initials: S
authors.lastname: Chacón-Moscoso
authors.affiliation: HUM-649, Innoevalua, Psicología Experimental, Universidad de Sevilla Sevilla, Spain.
authors.forename: Susana
authors.initials: S
authors.lastname: Sanduvete-Chaves
authors.affiliation: HUM-649, Innoevalua, Psicología Experimental, Universidad de Sevilla Sevilla, Spain.
authors.forename: José A
authors.initials: JA
authors.lastname: Pérez-Gil
keywords: Classical Test Theory; Structural Equation Modeling; causal analysis; quasi-experimental designs; threats to validity