title: STATISTICS. The reusable holdout: Preserving validity in adaptive data analysis.
abstract: Misapplication of statistical data analysis is a common cause of spurious discoveries in scientific research. Existing approaches to ensuring the validity of inferences drawn from data assume a fixed procedure to be performed, selected before the data are examined. In common practice, however, data analysis is an intrinsically adaptive process, with new analyses generated on the basis of data exploration, as well as the results of previous analyses on the same data. We demonstrate a new approach for addressing the challenges of adaptivity based on insights from privacy-preserving data analysis. As an application, we show how to safely reuse a holdout data set many times to validate the results of adaptively chosen analyses.
authors.affiliation: Microsoft Research, Mountain View, CA 94043, USA. dwork@microsoft.com vitaly@post.harvard.edu m@mrtz.org toni@cs.toronto.edu omer.reingold@gmail.com aaroth@cis.upenn.edu.
authors.forename: Cynthia
authors.initials: C
authors.lastname: Dwork
authors.affiliation: IBM Almaden Research Center, San Jose, CA 95120, USA. dwork@microsoft.com vitaly@post.harvard.edu m@mrtz.org toni@cs.toronto.edu omer.reingold@gmail.com aaroth@cis.upenn.edu.
authors.forename: Vitaly
authors.initials: V
authors.lastname: Feldman
authors.affiliation: Google Research, Mountain View, CA 94043, USA. dwork@microsoft.com vitaly@post.harvard.edu m@mrtz.org toni@cs.toronto.edu omer.reingold@gmail.com aaroth@cis.upenn.edu.
authors.forename: Moritz
authors.initials: M
authors.lastname: Hardt
authors.affiliation: Department of Computer Science, University of Toronto, Toronto, Ontario M5S 3G4, Canada. dwork@microsoft.com vitaly@post.harvard.edu m@mrtz.org toni@cs.toronto.edu omer.reingold@gmail.com aaroth@cis.upenn.edu.
authors.forename: Toniann
authors.initials: T
authors.lastname: Pitassi
authors.affiliation: Samsung Research America, Mountain View, CA 94043, USA. dwork@microsoft.com vitaly@post.harvard.edu m@mrtz.org toni@cs.toronto.edu omer.reingold@gmail.com aaroth@cis.upenn.edu.
authors.forename: Omer
authors.initials: O
authors.lastname: Reingold
authors.affiliation: Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA. dwork@microsoft.com vitaly@post.harvard.edu m@mrtz.org toni@cs.toronto.edu omer.reingold@gmail.com aaroth@cis.upenn.edu.
authors.forename: Aaron
authors.initials: A
authors.lastname: Roth