title: A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes.
abstract: Recent multi-voxel pattern classification (MVPC) studies have shown that in early visual cortex patterns of brain activity generated during mental imagery are similar to patterns of activity generated during perception. This finding implies that low-level visual features (e.g., space, spatial frequency, and orientation) are encoded during mental imagery. However, the specific hypothesis that low-level visual features are encoded during mental imagery is difficult to directly test using MVPC. The difficulty is especially acute when considering the representation of complex, multi-object scenes that can evoke multiple sources of variation that are distinct from low-level visual features. Therefore, we used a voxel-wise modeling and decoding approach to directly test the hypothesis that low-level visual features are encoded in activity generated during mental imagery of complex scenes. Using fMRI measurements of cortical activity evoked by viewing photographs, we constructed voxel-wise encoding models of tuning to low-level visual features. We also measured activity as subjects imagined previously memorized works of art. We then used the encoding models to determine if putative low-level visual features encoded in this activity could pick out the imagined artwork from among thousands of other randomly selected images. We show that mental images can be accurately identified in this way; moreover, mental image identification accuracy depends upon the degree of tuning to low-level visual features in the voxels selected for decoding. These results directly confirm the hypothesis that low-level visual features are encoded during mental imagery of complex scenes. Our work also points to novel forms of brain-machine interaction: we provide a proof-of-concept demonstration of an internet image search guided by mental imagery.
authors.affiliation: Department of Neurosciences, Medical University of South Carolina, SC, USA. Electronic address: tnaselar@musc.edu.
authors.forename: Thomas
authors.initials: T
authors.lastname: Naselaris
authors.affiliation: Department of Psychology, University of Minnesota, MN, USA; Center for Magnetic Resonance Research, University of Minnesota, MN, USA.
authors.forename: Cheryl A
authors.initials: CA
authors.lastname: Olman
authors.affiliation: Vision Science Group, University of California, Berkeley, CA, USA.
authors.forename: Dustin E
authors.initials: DE
authors.lastname: Stansbury
authors.affiliation: Center for Magnetic Resonance Research, University of Minnesota, MN, USA.
authors.forename: Kamil
authors.initials: K
authors.lastname: Ugurbil
authors.affiliation: Vision Science Group, University of California, Berkeley, CA, USA; Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA; Department of Psychology, University of California, Berkeley, CA, USA.
authors.forename: Jack L
authors.initials: JL
authors.lastname: Gallant
mesh_terms: D000328:Adult; D001931:Brain Mapping / Q000379:methods*; D006801:Humans; D007092:Imagination / Q000502:physiology*; D008279:Magnetic Resonance Imaging; D010364:Pattern Recognition, Visual / Q000502:physiology*; D014793:Visual Cortex / Q000502:physiology*
keywords: Decoding; Mental imagery; Perception; Vision; Voxel-wise encoding models; fMRI