title: Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines.
abstract: Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within- and across-subject predictions, on multiple datasets -anatomical and functional MRI and MEG- and simulations. Theory and experiments outline that the popular "leave-one-out" strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non-sparse decoders.
authors.affiliation: Parietal project-team, INRIA Saclay-ile de France, France; CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France.
authors.forename: Gaël
authors.initials: G
authors.lastname: Varoquaux
authors.affiliation: Rotman Research Institute, Baycrest Health Sciences, Toronto, ON, Canada M6A 2E1; Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada M5S 1A1.
authors.forename: Pradeep Reddy
authors.initials: PR
authors.lastname: Raamana
authors.affiliation: CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France; Cognitive Neuroimaging Unit, INSERM, Université Paris-Sud and Université Paris-Saclay, 91191 Gif-sur-Yvette, France; Neuropsychology & Neuroimaging team INSERM UMRS 975, Brain and Spine Institute (ICM), Paris.
authors.forename: Denis A
authors.initials: DA
authors.lastname: Engemann
authors.affiliation: Parietal project-team, INRIA Saclay-ile de France, France; CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France.
authors.forename: Andrés
authors.initials: A
authors.lastname: Hoyos-Idrobo
authors.affiliation: Parietal project-team, INRIA Saclay-ile de France, France; CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France.
authors.forename: Yannick
authors.initials: Y
authors.lastname: Schwartz
authors.affiliation: Parietal project-team, INRIA Saclay-ile de France, France; CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France.
authors.forename: Bertrand
authors.initials: B
authors.lastname: Thirion
mesh_terms: D001927:Brain Diseases / Q000000981:diagnostic imaging*; D006801:Humans; D059906:Neuroimaging / Q000379:methods* / Q000592:standards*
keywords: Bagging; Cross-validation; Decoding; FMRI; MVPA; Model selection; Sparse