title: Applying artificial vision models to human scene understanding.
abstract: How do we understand the complex patterns of neural responses that underlie scene understanding? Studies of the network of brain regions held to be scene-selective-the parahippocampal/lingual region (PPA), the retrosplenial complex (RSC), and the occipital place area (TOS)-have typically focused on single visual dimensions (e.g., size), rather than the high-dimensional feature space in which scenes are likely to be neurally represented. Here we leverage well-specified artificial vision systems to explicate a more complex understanding of how scenes are encoded in this functional network. We correlated similarity matrices within three different scene-spaces arising from: (1) BOLD activity in scene-selective brain regions; (2) behavioral measured judgments of visually-perceived scene similarity; and (3) several different computer vision models. These correlations revealed: (1) models that relied on mid- and high-level scene attributes showed the highest correlations with the patterns of neural activity within the scene-selective network; (2) NEIL and SUN-the models that best accounted for the patterns obtained from PPA and TOS-were different from the GIST model that best accounted for the pattern obtained from RSC; (3) The best performing models outperformed behaviorally-measured judgments of scene similarity in accounting for neural data. One computer vision method-NEIL ("Never-Ending-Image-Learner"), which incorporates visual features learned as statistical regularities across web-scale numbers of scenes-showed significant correlations with neural activity in all three scene-selective regions and was one of the two models best able to account for variance in the PPA and TOS. We suggest that these results are a promising first step in explicating more fine-grained models of neural scene understanding, including developing a clearer picture of the division of labor among the components of the functional scene-selective brain network.
authors.affiliation: Center for the Neural Basis of Cognition, Carnegie Mellon University Pittsburgh, PA, USA ; Department of Psychology, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Elissa M
authors.initials: EM
authors.lastname: Aminoff
authors.affiliation: Center for the Neural Basis of Cognition, Carnegie Mellon University Pittsburgh, PA, USA ; Department of Machine Learning, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Mariya
authors.initials: M
authors.lastname: Toneva
authors.affiliation: Robotics Institute, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Abhinav
authors.initials: A
authors.lastname: Shrivastava
authors.affiliation: Robotics Institute, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Xinlei
authors.initials: X
authors.lastname: Chen
authors.affiliation: Robotics Institute, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Ishan
authors.initials: I
authors.lastname: Misra
authors.affiliation: Robotics Institute, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Abhinav
authors.initials: A
authors.lastname: Gupta
authors.affiliation: Center for the Neural Basis of Cognition, Carnegie Mellon University Pittsburgh, PA, USA ; Department of Psychology, Carnegie Mellon University Pittsburgh, PA, USA.
authors.forename: Michael J
authors.initials: MJ
authors.lastname: Tarr
keywords: computer vision; parahippocampal place area; retrosplenial cortex; scene processing; transverse occipital sulcus