Assessing methods for generalizing experimental impact estimates to target populations.
Randomized experiments are considered the gold standard for causal inference, as they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments, as the experimental participants may be unrepresentative of the target population of interest. This paper examines whether generalization may be assisted by statistical methods that adjust for observed differences between the experimental participants and members of a target population. The methods examined include approaches that reweight the experimental data so that participants more closely resemble the target population and methods that utilize models of the outcome. Two simulation studies and one empirical analysis investigate and compare the methods' performance. One simulation uses purely simulated data while the other utilizes data from an evaluation of a school-based dropout prevention program. Our simulations suggest that machine learning methods outperform regression-based methods when the required structural (ignorability) assumptions are satisfied. When these assumptions are violated, all of the methods examined perform poorly. Our empirical analysis uses data from a multi-site experiment to assess how well results from a given site predict impacts in other sites. Using a variety of extrapolation methods, predicted effects for each site are compared to actual benchmarks. Flexible modeling approaches perform best, although linear regression is not far behind. Taken together, these results suggest that flexible modeling techniques can aid generalization while underscoring the fact that even state-of-the-art statistical techniques still rely on strong assumptions.
Department of Political Science, Florida State University.
Holger L
HL
Kern
Departments of Mental Health, Biostatistics, and Health, Policy, and Management, Bloomberg School of Public Health, Johns Hopkins University.
Elizabeth A
EA
Stuart
Professor of Applied Statistics and Data Sciences, Steinhardt School of Culture, Education, and Human Development, New York University.
Jennifer
J
Hill
Department of Political Science, Columbia University.
Donald P
DP
Green
Psychol Methods. 2010 Sep;15(3):234-49
20822250
JAMA. 1994 Oct 26;272(16):1257-62
7933370
J R Stat Soc Ser A Stat Soc. 2001 Apr 1;174(2):369-386
24926156
Neuroepidemiology. 2009;32(3):229-39
19176974
J Policy Anal Manage. 2013 Winter;32(1):107-121
25152557
Stat Med. 2010 Feb 10;29(3):337-46
19960510
Stat Sci. 2010 Feb 1;25(1):1-21
20871802
Lifetime Data Anal. 2009 Mar;15(1):120-46
18958616
PLoS One. 2011 Mar 31;6(3):e18174
21483818
Am J Epidemiol. 2010 Jul 1;172(1):107-15
20547574