title: Avoiding randomization failure in program evaluation, with application to the Medicare Health Support program.
abstract: We highlight common problems in the application of random treatment assignment in large-scale program evaluation. Random assignment is the defining feature of modern experimental design, yet errors in design, implementation, and analysis often result in real-world applications not benefiting from its advantages. The errors discussed here cover the control of variability, levels of randomization, size of treatment arms, and power to detect causal effects, as well as the many problems that commonly lead to post-treatment bias. We illustrate these issues by identifying numerous serious errors in the Medicare Health Support evaluation and offering recommendations to improve the design and analysis of this and other large-scale randomized experiments.
authors.affiliation: Institute for Quantitative Social Science, Harvard University, Cambridge, Massachusetts 02138, USA. king@harvard.edu
authors.forename: Gary
authors.initials: G
authors.lastname: King
authors.forename: Richard
authors.initials: R
authors.lastname: Nielsen
authors.forename: Carter
authors.initials: C
authors.lastname: Coberley
authors.forename: James E
authors.initials: JE
authors.lastname: Pope
authors.forename: Aaron
authors.initials: A
authors.lastname: Wells
mesh_terms: D006078:Government Programs / Q000191:economics* / Q000706:statistics & numerical data; D006801:Humans; D006278:Medicare / Q000191:economics* / Q000706:statistics & numerical data; D010865:Pilot Projects; D015397:Program Evaluation / Q000191:economics* / Q000706:statistics & numerical data; D016032:Randomized Controlled Trials as Topic / Q000191:economics* / Q000706:statistics & numerical data; D012107:Research Design; D015983:Selection Bias; D014481:United States