title: Implications of Small Samples for Generalization: Adjustments and Rules of Thumb.
abstract: Policy makers and researchers are frequently interested in understanding how effective a particular intervention may be for a specific population. One approach is to assess the degree of similarity between the sample in an experiment and the population. Another approach is to combine information from the experiment and the population to estimate the population average treatment effect (PATE).
  
  
  
  Several methods for assessing the similarity between a sample and population currently exist as well as methods estimating the PATE. In this article, we investigate properties of six of these methods and statistics in the small sample sizes common in education research (i.e., 10-70 sites), evaluating the utility of rules of thumb developed from observational studies in the generalization case.
  
  
  
  In small random samples, large differences between the sample and population can arise simply by chance and many of the statistics commonly used in generalization are a function of both sample size and the number of covariates being compared. The rules of thumb developed in observational studies (which are commonly applied in generalization) are much too conservative given the small sample sizes found in generalization.
  
  
  
  This article implies that sharp inferences to large populations from small experiments are difficult even with probability sampling. Features of random samples should be kept in mind when evaluating the extent to which results from experiments conducted on nonrandom samples might generalize.
authors.affiliation: 1 Department of Human Development, Teachers College, Columbia University, New York, NY, USA.
authors.forename: Elizabeth
authors.initials: E
authors.lastname: Tipton
authors.affiliation: 2 University of Chicago - Urban Labs, Chicago, IL, USA.
authors.forename: Kelly
authors.initials: K
authors.lastname: Hallberg
authors.affiliation: 3 Institute for Policy Research, Northwestern University, Evanston, IL, USA.
authors.forename: Larry V
authors.initials: LV
authors.lastname: Hedges
authors.affiliation: 4 Quantitative Methods Division, Graduate School of Education, University of Pennsylvania, Philadelphia, PA USA.
authors.forename: Wendy
authors.initials: W
authors.lastname: Chan
mesh_terms: D000076363:Academic Success; D000293:Adolescent; D002648:Child; D002675:Child, Preschool; D005069:Evaluation Studies as Topic; D006801:Humans; D007196:Indiana; D015233:Models, Statistical; D011050:Policy Making; D057216:Propensity Score; D016032:Randomized Controlled Trials as Topic; D018401:Sample Size
keywords: content area; education; methodological development