Statistical methods and computing for big data.
Big data are data on a massive scale in terms of volume, intensity, and complexity that exceed the capacity of standard analytic tools. They present opportunities as well as challenges to statisticians. The role of computational statisticians in scientific discovery from big data analyses has been under-recognized even by peer statisticians. This article summarizes recent methodological and software developments in statistics that address the big data challenges. Methodologies are grouped into three classes: subsampling-based, divide and conquer, and online updating for stream data. As a new contribution, the online updating approach is extended to variable selection with commonly used criteria, and their performances are assessed in a simulation study with stream data. Software packages are summarized with focuses on the open source R and R packages, covering recent tools that help break the barriers of computer memory and computing power. Some of the tools are illustrated in a case study with a logistic regression for the chance of airline delay.
215 Glenbrook Rd., Storrs, 06269, USA.
Chun
C
Wang
215 Glenbrook Rd., Storrs, 06269, USA.
Ming-Hui
MH
Chen
215 Glenbrook Rd., Storrs, 06269, USA.
Elizabeth
E
Schifano
215 Glenbrook Rd., Storrs, 06269, USA.
Jing
J
Wu
215 Glenbrook Rd., Storrs, 06269, USA.
Jun
J
Yan
Bootstrap; Divide and conquer; External memory algorithm; High performance computing; Online update; Sampling; Software
J R Stat Soc Series B Stat Methodol. 2008;70(5):849-911
19603084
J Comput Graph Stat. 2010 Jun 1;19(2):419-438
20877443
IEEE Trans Inf Theory. 2011 Aug;57(8):5467-5484
22287795
Mol Biol Evol. 2014 Jul;31(7):1929-36
24739305
PLoS One. 2014 Sep 26;9(9):e108425
25259608
Natl Sci Rev. 2014 Jun;1(2):293-314
25419469
Proc Natl Acad Sci U S A. 2014 Dec 9;111(49):17408-13
25422442
Technometrics. 2016;58(3):393-403
28018007
Technometrics. 2016;58(3):604-318
29033469