title: An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials.
abstract: BACKGROUND
  Prior research has investigated design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Less is known about parameters important for assessing differential impacts.
  
  
  OBJECTIVE
  This article develops a statistical framework for designing CRTs to assess differences in impact among student subgroups and presents initial estimates of critical parameters.
  
  
  METHODS
  Effect sizes and minimum detectable effect sizes for average and differential impacts are calculated before and after conditioning on effects of covariates using results from several CRTs. Relative sensitivities to detect average and differential impacts are also examined.
  
  
  METHODS
  Student outcomes from six CRTs are analyzed.
  
  
  METHODS
  Achievement in math, science, reading, and writing.
  
  
  RESULTS
  The ratio of between-cluster variation in the slope of the moderator divided by total variance-the "moderator gap variance ratio"-is important for designing studies to detect differences in impact between student subgroups. This quantity is the analogue of the intraclass correlation coefficient. Typical values were .02 for gender and .04 for socioeconomic status. For studies considered, in many cases estimates of differential impact were larger than of average impact, and after conditioning on effects of covariates, similar power was achieved for detecting average and differential impacts of the same size.
  
  
  CONCLUSIONS
  Measuring differential impacts is important for addressing questions of equity, generalizability, and guiding interpretation of subgroup impact findings. Adequate power for doing this is in some cases reachable with CRTs designed to measure average impacts. Continuing collection of parameters for assessing differential impacts is the next step.
authors.affiliation: Empirical Education Inc., Palo Alto, CA, USA ajaciw@empiricaleducation.com ajaciw@hotmail.com.
authors.forename: Andrew P
authors.initials: AP
authors.lastname: Jaciw
authors.affiliation: Empirical Education Inc., Palo Alto, CA, USA.
authors.forename: Li
authors.initials: L
authors.lastname: Lin
authors.affiliation: Empirical Education Inc., Palo Alto, CA, USA.
authors.forename: Boya
authors.initials: B
authors.lastname: Ma
keywords: design and evaluation of programs and policies; education