title: Decoding the Semantic Content of Natural Movies from Human Brain Activity.
abstract: One crucial test for any quantitative model of the brain is to show that the model can be used to accurately decode information from evoked brain activity. Several recent neuroimaging studies have decoded the structure or semantic content of static visual images from human brain activity. Here we present a decoding algorithm that makes it possible to decode detailed information about the object and action categories present in natural movies from human brain activity signals measured by functional MRI. Decoding is accomplished using a hierarchical logistic regression (HLR) model that is based on labels that were manually assigned from the WordNet semantic taxonomy. This model makes it possible to simultaneously decode information about both specific and general categories, while respecting the relationships between them. Our results show that we can decode the presence of many object and action categories from averaged blood-oxygen level-dependent (BOLD) responses with a high degree of accuracy (area under the ROC curve > 0.9). Furthermore, we used this framework to test whether semantic relationships defined in the WordNet taxonomy are represented the same way in the human brain. This analysis showed that hierarchical relationships between general categories and atypical examples, such asorganismandplant, did not seem to be reflected in representations measured by BOLD fMRI.
authors: Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
authors: Alexander G
authors: AG
authors: Huth
authors: Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
authors: Tyler
authors: T
authors: Lee
authors: Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
authors: Shinji
authors: S
authors: Nishimoto
authors: Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
authors: Natalia Y
authors: NY
authors: Bilenko
authors: Bioengineering Graduate Group, University of California Berkeley, Berkeley, CA, USA.
authors: An T
authors: AT
authors: Vu
authors: Helen Wills Neuroscience Institute, University of CaliforniaBerkeley, Berkeley, CA, USA; Bioengineering Graduate Group, University of CaliforniaBerkeley, Berkeley, CA, USA; Department of Psychology, University of CaliforniaBerkeley, Berkeley, CA, USA.
authors: Jack L
authors: JL
authors: Gallant
keywords: WordNet; decoding; fMRI; natural images; structured output