title: Classification based hypothesis testing in neuroscience: Below-chance level classification rates and overlooked statistical properties of linear parametric classifiers.
abstract: Multivariate pattern analysis (MVPA) has recently become a popular tool for data analysis. Often, classification accuracy as quantified by correct classification rate (CCR) is used to illustrate the size of the effect under investigation. However, we show that in low sample size (LSS), low effect size (LES) data, which is typical in neuroscience, the distribution of CCRs from cross-validation of linear MVPA is asymmetric and can show classification rates considerably below what would be expected from chance classification. Conversely, the mode of the distribution in these cases is above expected chance levels, leading to a spuriously high number of above chance CCRs. This unexpected distribution has strong implications when using MVPA for hypothesis testing. Our analyses warrant the conclusion that CCRs do not well reflect the size of the effect under investigation. Moreover, the skewness of the null-distribution precludes the use of many standard parametric tests to assess significance of CCRs. We propose that MVPA results should be reported in terms of P values, which are estimated using randomization tests. Also, our results show that cross-validation procedures using a low number of folds, e.g. twofold, are generally more sensitive, even though the average CCRs are often considerably lower than those obtained using a higher number of folds. Hum Brain Mapp 37:1842-1855, 2016. © 2016 Wiley Periodicals, Inc.
authors.affiliation: Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany.|Bernstein Center for Computational Neuroscience, Ludwig-Maximilians-Universität München, Planegg-Martinsried, Germany.|IMPRS for Cognitive and Systems Neuroscience, University of Tübingen, Tübingen, Germany.
authors.forename: Hamidreza
authors.initials: H
authors.lastname: Jamalabadi
authors.affiliation: Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany.|Bernstein Center for Computational Neuroscience, Ludwig-Maximilians-Universität München, Planegg-Martinsried, Germany.|IMPRS for Cognitive and Systems Neuroscience, University of Tübingen, Tübingen, Germany.
authors.forename: Sarah
authors.initials: S
authors.lastname: Alizadeh
authors.affiliation: Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany.|Bernstein Center for Computational Neuroscience, Ludwig-Maximilians-Universität München, Planegg-Martinsried, Germany.|Department of Psychology, Ludwig-Maximilians-Universität München, München, Germany.
authors.forename: Monika
authors.initials: M
authors.lastname: Schönauer
authors.affiliation: Bernstein Center for Computational Neuroscience, Ludwig-Maximilians-Universität München, Planegg-Martinsried, Germany.|Department of Biology II, Ludwig-Maximilians-Universität München, Planegg-Martinsried, Germany.
authors.forename: Christian
authors.initials: C
authors.lastname: Leibold
authors.affiliation: Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany.|Bernstein Center for Computational Neuroscience, Ludwig-Maximilians-Universität München, Planegg-Martinsried, Germany.|Department of Psychology, Ludwig-Maximilians-Universität München, München, Germany.
authors.forename: Steffen
authors.initials: S
authors.lastname: Gais
mesh_terms: D001921:Brain / Q000033:anatomy & histology*; D001931:Brain Mapping; D004569:Electroencephalography; D001330:Electronic Data Processing; D005071:Evoked Potentials / Q000502:physiology; D006801:Humans; D008959:Models, Neurological; D009488:Neurosciences / Q000145:classification; D011336:Probability
keywords: below chance classification rate; cross-validation; hypothesis testing; multivariate pattern classification