title: Valid population inference for information-based imaging: From the second-level t-test to prevalence inference.
abstract: In multivariate pattern analysis of neuroimaging data, 'second-level' inference is often performed by entering classification accuracies into a t-test vs chance level across subjects. We argue that while the random-effects analysis implemented by the t-test does provide population inference if applied to activation differences, it fails to do so in the case of classification accuracy or other 'information-like' measures, because the true value of such measures can never be below chance level. This constraint changes the meaning of the population-level null hypothesis being tested, which becomes equivalent to the global null hypothesis that there is no effect in any subject in the population. Consequently, rejecting it only allows to infer that there are some subjects in which there is an information effect, but not that it generalizes, rendering it effectively equivalent to fixed-effects analysis. This statement is supported by theoretical arguments as well as simulations. We review possible alternative approaches to population inference for information-based imaging, converging on the idea that it should not target the mean, but the prevalence of the effect in the population. One method to do so, 'permutation-based information prevalence inference using the minimum statistic', is described in detail and applied to empirical data.
authors.affiliation: Bernstein Center for Computational Neuroscience, Berlin Center of Advanced Neuroimaging, Department of Neurology, and Excellence Cluster NeuroCure, Charité - Universitätsmedizin Berlin, Germany. Electronic address: carsten.allefeld@bccn-berlin.de.
authors.forename: Carsten
authors.initials: C
authors.lastname: Allefeld
authors.affiliation: Bernstein Center for Computational Neuroscience, Berlin Center of Advanced Neuroimaging, Department of Neurology, and Excellence Cluster NeuroCure, Charité - Universitätsmedizin Berlin, Germany.
authors.forename: Kai
authors.initials: K
authors.lastname: Görgen
authors.affiliation: Bernstein Center for Computational Neuroscience, Berlin Center of Advanced Neuroimaging, Department of Neurology, and Excellence Cluster NeuroCure, Charité - Universitätsmedizin Berlin, Germany; Berlin School of Mind and Brain and Department of Psychology, Humboldt-Universität zu Berlin, Germany. Electronic address: haynes@bccn-berlin.de.
authors.forename: John-Dylan
authors.initials: JD
authors.lastname: Haynes
mesh_terms: D001921:Brain / Q000502:physiology*; D001931:Brain Mapping / Q000379:methods*; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D015233:Models, Statistical; D015999:Multivariate Analysis; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity
keywords: Effect prevalence; Information-based imaging; Multivariate pattern analysis; Population inference; t-Test