title: Statistical challenges of high-dimensional data.
abstract: Modern applications of statistical theory and methods can involve extremely large datasets, often with huge numbers of measurements on each of a comparatively small number of experimental units. New methodology and accompanying theory have emerged in response: the goal of this Theme Issue is to illustrate a number of these recent developments. This overview article introduces the difficulties that arise with high-dimensional data in the context of the very familiar linear statistical model: we give a taste of what can nevertheless be achieved when the parameter vector of interest is sparse, that is, contains many zero elements. We describe other ways of identifying low-dimensional subspaces of the data space that contain all useful information. The topic of classification is then reviewed along with the problem of identifying, from within a very large set, the variables that help to classify observations. Brief mention is made of the visualization of high-dimensional data and ways to handle computational problems in Bayesian analysis are described. At appropriate points, reference is made to the other papers in the issue.
authors.affiliation: Department of Statistics, Stanford University, Stanford, CA 94305, USA.
authors.forename: Iain M
authors.initials: IM
authors.lastname: Johnstone
authors.forename: D Michael
authors.initials: DM
authors.lastname: Titterington
mesh_terms: D001499:Bayes Theorem; D016014:Linear Models; D013223:Statistics as Topic / Q000379:methods*