title: Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study.
abstract: Both embodied and symbolic accounts of conceptual organization would predict partial sharing and partial differentiation between the neural activations seen for concepts activated via different stimulus modalities. But cross-participant and cross-session variability in BOLD activity patterns makes analyses of such patterns with MVPA methods challenging. Here, we examine the effect of cross-modal and individual variation on the machine learning analysis of fMRI data recorded during a word property generation task. We present the same set of living and non-living concepts (land-mammals, or work tools) to a cohort of Japanese participants in two sessions: the first using auditory presentation of spoken words; the second using visual presentation of words written in Japanese characters. Classification accuracies confirmed that these semantic categories could be detected in single trials, with within-session predictive accuracies of 80-90%. However cross-session prediction (learning from auditory-task data to classify data from the written-word-task, or vice versa) suffered from a performance penalty, achieving 65-75% (still individually significant at p « 0.05). We carried out several follow-on analyses to investigate the reason for this shortfall, concluding that distributional differences in neither time nor space alone could account for it. Rather, combined spatio-temporal patterns of activity need to be identified for successful cross-session learning, and this suggests that feature selection strategies could be modified to take advantage of this.
authors.affiliation: Akama Laboratory, Graduate School of Decision Science and Technology, Tokyo Institute of Technology Tokyo, Japan.
authors.forename: Hiroyuki
authors.initials: H
authors.lastname: Akama
authors.forename: Brian
authors.initials: B
authors.lastname: Murphy
authors.forename: Li
authors.initials: L
authors.lastname: Na
authors.forename: Yumiko
authors.initials: Y
authors.lastname: Shimizu
authors.forename: Massimo
authors.initials: M
authors.lastname: Poesio