title: Mining big data to extract patterns and predict real-life outcomes.
abstract: This article aims to introduce the reader to essential tools that can be used to obtain insights and build predictive models using large data sets. Recent user proliferation in the digital environment has led to the emergence of large samples containing a wealth of traces of human behaviors, communication, and social interactions. Such samples offer the opportunity to greatly improve our understanding of individuals, groups, and societies, but their analysis presents unique methodological challenges. In this tutorial, we discuss potential sources of such data and explain how to efficiently store them. Then, we introduce two methods that are often employed to extract patterns and reduce the dimensionality of large data sets: singular value decomposition and latent Dirichlet allocation. Finally, we demonstrate how to use dimensions or clusters extracted from data to build predictive models in a cross-validated way. The text is accompanied by examples of R code and a sample data set, allowing the reader to practice the methods discussed here. A companion website (http://dataminingtutorial.com) provides additional learning resources. (PsycINFO Database Record
authors.affiliation: Graduate School of Business, Stanford University.
authors.forename: Michal
authors.identifier: 0000-0003-2936-4775
authors.initials: M
authors.lastname: Kosinski
authors.affiliation: Department of Computer Science, Stanford University.
authors.forename: Yilun
authors.initials: Y
authors.lastname: Wang
authors.affiliation: Department of Computer Science, Stanford University.
authors.forename: Himabindu
authors.initials: H
authors.lastname: Lakkaraju
authors.affiliation: Department of Computer Science, Stanford University.
authors.forename: Jure
authors.initials: J
authors.lastname: Leskovec
mesh_terms: D057225:Data Mining / Q000379:methods*; D066264:Datasets as Topic; D006801:Humans; D020407:Internet; D008962:Models, Theoretical