title: Do estimated impacts on earnings depend on the source of the data used to measure them? Evidence from previous social experiments.
abstract: BACKGROUND
  Impact evaluations draw their data from two sources, namely, surveys conducted for the evaluation or administrative data collected for other purposes. Both types of data have been used in impact evaluations of social programs.
  
  
  OBJECTIVE
  This study analyzes the causes of differences in impact estimates when survey data and administrative data are used to evaluate earnings impacts in social experiments and discusses the differences observed in eight evaluations of social experiments that used both survey and administrative data.
  
  
  RESULTS
  There are important trade-offs between the two data sources. Administrative data are less expensive but may not cover all income and may not cover the time period desired, while surveys can be designed to avoid these problems. We note that errors can be due to nonresponse or reporting, and errors can be balanced between the treatment and the control groups or unbalanced. We find that earnings are usually higher in survey data than in administrative data due to differences in coverage and likely overreporting of overtime hours and pay in survey data. Evaluations using survey data usually find greater impacts, sometimes much greater.
  
  
  CONCLUSIONS
  The much lower cost of administrative data make their use attractive, but they are still subject to underreporting and other problems. We recommend further evaluations using both types of data with investigative audits to better understand the sources and magnitudes of errors in both survey and administrative data so that appropriate corrections to the data can be made.
authors.affiliation: George Washington University, Washington, DC, USA barnow@gwu.edu.
authors.forename: Burt S
authors.initials: BS
authors.lastname: Barnow
authors.affiliation: University of Maryland, Baltimore County, Baltimore, MD, USA.
authors.forename: David
authors.initials: D
authors.lastname: Greenberg
mesh_terms: D016208:Databases, Factual; D006801:Humans; D007182:Income / Q000706:statistics & numerical data*; D015233:Models, Statistical; D011237:Predictive Value of Tests; D012924:Social Conditions / Q000191:economics*; D013223:Statistics as Topic; D011795:Surveys and Questionnaires; D006113:United Kingdom; D014481:United States
keywords: administrative data; impact evaluations; social experiments; survey data