title: Performance-optimized hierarchical models predict neural responses in higher visual cortex.
abstract: The ventral visual stream underlies key human visual object recognition abilities. However, neural encoding in the higher areas of the ventral stream remains poorly understood. Here, we describe a modeling approach that yields a quantitatively accurate model of inferior temporal (IT) cortex, the highest ventral cortical area. Using high-throughput computational techniques, we discovered that, within a class of biologically plausible hierarchical neural network models, there is a strong correlation between a model's categorization performance and its ability to predict individual IT neural unit response data. To pursue this idea, we then identified a high-performing neural network that matches human performance on a range of recognition tasks. Critically, even though we did not constrain this model to match neural data, its top output layer turns out to be highly predictive of IT spiking responses to complex naturalistic images at both the single site and population levels. Moreover, the model's intermediate layers are highly predictive of neural responses in the V4 cortex, a midlevel visual area that provides the dominant cortical input to IT. These results show that performance optimization--applied in a biologically appropriate model class--can be used to build quantitative predictive models of neural processing.
authors.affiliation: Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139; and.
authors.forename: Daniel L K
authors.initials: DL
authors.lastname: Yamins
authors.affiliation: Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139; andHarvard-MIT Division of Health Sciences and Technology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA 02139.
authors.forename: Ha
authors.initials: H
authors.lastname: Hong
authors.affiliation: Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139; and.
authors.forename: Charles F
authors.initials: CF
authors.lastname: Cadieu
authors.affiliation: Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139; and.
authors.forename: Ethan A
authors.initials: EA
authors.lastname: Solomon
authors.affiliation: Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139; and.
authors.forename: Darren
authors.initials: D
authors.lastname: Seibert
authors.affiliation: Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139; and dicarlo@mit.edu.
authors.forename: James J
authors.initials: JJ
authors.lastname: DiCarlo
mesh_terms: D000465:Algorithms; D000818:Animals; D006801:Humans; D008253:Macaca mulatta / Q000502:physiology*; D008959:Models, Neurological; D009415:Nerve Net / Q000502:physiology; D016571:Neural Networks, Computer; D010775:Photic Stimulation / Q000379:methods; D011597:Psychomotor Performance / Q000502:physiology; D021641:Recognition, Psychology / Q000502:physiology; D014793:Visual Cortex / Q000502:physiology*; D014795:Visual Pathways / Q000502:physiology; D014796:Visual Perception / Q000502:physiology
keywords: array electrophysiology; computational neuroscience; computer vision