Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence.
The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the statistics of real-world visual recognition. We showed that the DNN captured the stages of human visual processing in both time and space from early visual areas towards the dorsal and ventral streams. Further investigation of crucial DNN parameters revealed that while model architecture was important, training on real-world categorization was necessary to enforce spatio-temporal hierarchical relationships with the brain. Together our results provide an algorithmically informed view on the spatio-temporal dynamics of visual object recognition in the human visual brain.
Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.|Department of Education and Psychology, Free University Berlin, Berlin, Germany.
Radoslaw Martin
RM
Cichy
Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
Aditya
A
Khosla
McGovern Institute for Brain Research, MIT, Cambridge, MA, USA.
Dimitrios
D
Pantazis
Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
Antonio
A
Torralba
Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
Aude
A
Oliva
D000328:Adult; D001921:Brain / Q000502:physiology*; D001931:Brain Mapping; D005260:Female; D006801:Humans; D008279:Magnetic Resonance Imaging; D015225:Magnetoencephalography; D008297:Male; D016571:Neural Networks, Computer; D013647:Task Performance and Analysis; D013997:Time Factors; D014793:Visual Cortex; D014795:Visual Pathways / Q000502:physiology; D014796:Visual Perception / Q000502:physiology*