title: Reliability of dissimilarity measures for multi-voxel pattern analysis.
abstract: Representational similarity analysis of activation patterns has become an increasingly important tool for studying brain representations. The dissimilarity between two patterns is commonly quantified by the correlation distance or the accuracy of a linear classifier. However, there are many different ways to measure pattern dissimilarity and little is known about their relative reliability. Here, we compare the reliability of three classes of dissimilarity measure: classification accuracy, Euclidean/Mahalanobis distance, and Pearson correlation distance. Using simulations and four real functional magnetic resonance imaging (fMRI) datasets, we demonstrate that continuous dissimilarity measures are substantially more reliable than the classification accuracy. The difference in reliability can be explained by two characteristics of classifiers: discretization and susceptibility of the discriminant function to shifts of the pattern ensemble between imaging runs. Reliability can be further improved through multivariate noise normalization for all measures. Finally, unlike conventional distance measures, crossvalidated distances provide unbiased estimates of pattern dissimilarity on a ratio scale, thus providing an interpretable zero point. Overall, our results indicate that the crossvalidated Mahalanobis distance is preferable to both the classification accuracy and the correlation distance for characterizing representational geometries.
authors.affiliation: MRC Cognition and Brain Sciences Unit, 15 Chaucer Road, CB2 7EF, Cambridge, United Kingdom; Institute of Cognitive Neuroscience, University College London, Alexandra House, 17 Queen Square, London WC1N 3AR, United Kingdom. Electronic address: alexander.walther@mrc-cbu.cam.ac.uk.
authors.forename: Alexander
authors.initials: A
authors.lastname: Walther
authors.affiliation: Department of Experimental Psychology, University of Oxford, South Parks Road, OX1 3UD, Oxford, United Kingdom. Electronic address: hamed.nili@psy.ox.ac.uk.
authors.forename: Hamed
authors.initials: H
authors.lastname: Nili
authors.affiliation: Institute of Cognitive Neuroscience, University College London, Alexandra House, 17 Queen Square, London WC1N 3AR, United Kingdom. Electronic address: n.ejaz@ucl.ac.uk.
authors.forename: Naveed
authors.initials: N
authors.lastname: Ejaz
authors.affiliation: MRC Cognition and Brain Sciences Unit, 15 Chaucer Road, CB2 7EF, Cambridge, United Kingdom. Electronic address: arjen.alink@mrc-cbu.cam.ac.uk.
authors.forename: Arjen
authors.initials: A
authors.lastname: Alink
authors.affiliation: MRC Cognition and Brain Sciences Unit, 15 Chaucer Road, CB2 7EF, Cambridge, United Kingdom. Electronic address: nikolaus.kriegeskorte@mrc-cbu.cam.ac.uk.
authors.forename: Nikolaus
authors.initials: N
authors.lastname: Kriegeskorte
authors.affiliation: Institute of Cognitive Neuroscience, University College London, Alexandra House, 17 Queen Square, London WC1N 3AR, United Kingdom. Electronic address: j.diedrichsen@ucl.ac.uk.
authors.forename: Jörn
authors.initials: J
authors.lastname: Diedrichsen
mesh_terms: D000465:Algorithms; D001921:Brain / Q000502:physiology*; D001931:Brain Mapping / Q000379:methods*; D003627:Data Interpretation, Statistical; D005260:Female; D006801:Humans; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D010363:Pattern Recognition, Automated / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013382:Subtraction Technique
keywords: Classification; Crossvalidation; Decoding; Linear discriminant; Machine learning; Multi-voxel pattern analysis; Noise normalization; Representational similarity analysis; fMRI