title: Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.
abstract: Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the cross-validation was further illustrated on real-data from a brain-computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson's disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium ; Coma Science Group, Neurology Department, University Hospital of Liège, Liège, Belgium.
authors.forename: Quentin
authors.initials: Q
authors.lastname: Noirhomme
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium ; Coma Science Group, Neurology Department, University Hospital of Liège, Liège, Belgium.
authors.forename: Damien
authors.initials: D
authors.lastname: Lesenfants
authors.affiliation: Complexus Group, Computer Science Department, Universidad Central de Colombia, Bogotá, Colombia.
authors.forename: Francisco
authors.initials: F
authors.lastname: Gomez
authors.affiliation: Department of Physics & Astronomy, Brain and Mind Institute, University of Western Ontario, London, ON, Canada.
authors.forename: Andrea
authors.initials: A
authors.lastname: Soddu
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium ; Laboratory of Behavioral and Cognitive Neurology, Stanford University, Palo Alto, USA.
authors.forename: Jessica
authors.initials: J
authors.lastname: Schrouff
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium.
authors.forename: Gaëtan
authors.initials: G
authors.lastname: Garraux
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium.
authors.forename: André
authors.initials: A
authors.lastname: Luxen
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium ; Department of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium.
authors.forename: Christophe
authors.initials: C
authors.lastname: Phillips
authors.affiliation: Cyclotron Research Centre, University of Liège, Liège, Belgium ; Coma Science Group, Neurology Department, University Hospital of Liège, Liège, Belgium.
authors.forename: Steven
authors.initials: S
authors.lastname: Laureys
mesh_terms: D000328:Adult; D000368:Aged; D015982:Bias; D001921:Brain / Q000473:pathology*; D001930:Brain Injuries / Q000175:diagnosis*; D062207:Brain-Computer Interfaces; D003198:Computer Simulation; D006801:Humans; D008875:Middle Aged; D015233:Models, Statistical; D055815:Young Adult
keywords: binomial; classification; cross-validation; permutation test