title: Validating vignette and conjoint survey experiments against real-world behavior.
abstract: Survey experiments, like vignette and conjoint analyses, are widely used in the social sciences to elicit stated preferences and study how humans make multidimensional choices. However, there is a paucity of research on the external validity of these methods that examines whether the determinants that explain hypothetical choices made by survey respondents match the determinants that explain what subjects actually do when making similar choices in real-world situations. This study compares results from conjoint and vignette analyses on which immigrant attributes generate support for naturalization with closely corresponding behavioral data from a natural experiment in Switzerland, where some municipalities used referendums to decide on the citizenship applications of foreign residents. Using a representative sample from the same population and the official descriptions of applicant characteristics that voters received before each referendum as a behavioral benchmark, we find that the effects of the applicant attributes estimated from the survey experiments perform remarkably well in recovering the effects of the same attributes in the behavioral benchmark. We also find important differences in the relative performances of the different designs. Overall, the paired conjoint design, where respondents evaluate two immigrants side by side, comes closest to the behavioral benchmark; on average, its estimates are within 2% percentage points of the effects in the behavioral benchmark.
authors: Department of Political Science and Graduate School of Business, Stanford University, Stanford, CA 94305-6044;
authors: Jens
authors: J
authors: Hainmueller
authors: Department of Methodology, London School of Economics, London WC2A 2AE, United Kingdom; Institute of Political Science, University of Zurich, 8050 Zurich, Switzerland; and.
authors: Dominik
authors: D
authors: Hangartner
authors: Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139 teppei@mit.edu.
authors: Teppei
authors: T
authors: Yamamoto
mesh_terms: D002755:Choice Behavior; D003625:Data Collection; D004641:Emigration and Immigration; D006801:Humans; D015203:Reproducibility of Results; D013557:Switzerland
keywords: conjoint; public opinion; stated preferences; survey methodology; vignette
references: Psychol Bull. 2007 Sep;133(5):859-83
references: 17723033
references: Public Opin Q. 2001 Spring;65(1):22-44
references: 11264053