title: Humans and Deep Networks Largely Agree on Which Kinds of Variation Make Object Recognition Harder.
abstract: View-invariant object recognition is a challenging problem that has attracted much attention among the psychology, neuroscience, and computer vision communities. Humans are notoriously good at it, even if some variations are presumably more difficult to handle than others (e.g., 3D rotations). Humans are thought to solve the problem through hierarchical processing along the ventral stream, which progressively extracts more and more invariant visual features. This feed-forward architecture has inspired a new generation of bio-inspired computer vision systems called deep convolutional neural networks (DCNN), which are currently the best models for object recognition in natural images. Here, for the first time, we systematically compared human feed-forward vision and DCNNs at view-invariant object recognition task using the same set of images and controlling the kinds of transformation (position, scale, rotation in plane, and rotation in depth) as well as their magnitude, which we call "variation level." We used four object categories: car, ship, motorcycle, and animal. In total, 89 human subjects participated in 10 experiments in which they had to discriminate between two or four categories after rapid presentation with backward masking. We also tested two recent DCNNs (proposed respectively by Hinton's group and Zisserman's group) on the same tasks. We found that humans and DCNNs largely agreed on the relative difficulties of each kind of variation: rotation in depth is by far the hardest transformation to handle, followed by scale, then rotation in plane, and finally position (much easier). This suggests that DCNNs would be reasonable models of human feed-forward vision. In addition, our results show that the variation levels in rotation in depth and scale strongly modulate both humans' and DCNNs' recognition performances. We thus argue that these variations should be controlled in the image datasets used in vision research.
authors.affiliation: Department of Computer Science, School of Mathematics, Statistics, and Computer Science, University of TehranTehran, Iran; CerCo UMR 5549, Centre National de la Recherche Scientifique - Université de ToulouseToulouse, France.
authors.forename: Saeed R
authors.initials: SR
authors.lastname: Kheradpisheh
authors.affiliation: Department of Physiology, Monash UniversityClayton, VIC, Australia; Neuroscience Program, Biomedicine Discovery Institute, Monash UniversityClayton, VIC, Australia.
authors.forename: Masoud
authors.initials: M
authors.lastname: Ghodrati
authors.affiliation: Department of Computer Science, School of Mathematics, Statistics, and Computer Science, University of Tehran Tehran, Iran.
authors.forename: Mohammad
authors.initials: M
authors.lastname: Ganjtabesh
authors.affiliation: CerCo UMR 5549, Centre National de la Recherche Scientifique - Université de ToulouseToulouse, France; Institut National de la Santé et de la Recherche Médicale, U968Paris, France; Sorbonne Universités, UPMC Univ Paris 06, UMR-S 968, Institut de la VisionParis, France; Centre National de la Recherche Scientifique, UMR-7210Paris, France.
authors.forename: Timothée
authors.initials: T
authors.lastname: Masquelier
keywords: 2D and 3D object variations; deep networks; feed-forward vision; rapid invariant object recognition; ventral stream models