title: Crossmodal integration enhances neural representation of task-relevant features in audiovisual face perception.
abstract: Previous studies have shown that audiovisual integration improves identification performance and enhances neural activity in heteromodal brain areas, for example, the posterior superior temporal sulcus/middle temporal gyrus (pSTS/MTG). Furthermore, it has also been demonstrated that attention plays an important role in crossmodal integration. In this study, we considered crossmodal integration in audiovisual facial perception and explored its effect on the neural representation of features. The audiovisual stimuli in the experiment consisted of facial movie clips that could be classified into 2 gender categories (male vs. female) or 2 emotion categories (crying vs. laughing). The visual/auditory-only stimuli were created from these movie clips by removing the auditory/visual contents. The subjects needed to make a judgment about the gender/emotion category for each movie clip in the audiovisual, visual-only, or auditory-only stimulus condition as functional magnetic resonance imaging (fMRI) signals were recorded. The neural representation of the gender/emotion feature was assessed using the decoding accuracy and the brain pattern-related reproducibility indices, obtained by a multivariate pattern analysis method from the fMRI data. In comparison to the visual-only and auditory-only stimulus conditions, we found that audiovisual integration enhanced the neural representation of task-relevant features and that feature-selective attention might play a role of modulation in the audiovisual integration.
authors: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou 510640, China.
authors: Yuanqing
authors: Y
authors: Li
authors: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou 510640, China.
authors: Jinyi
authors: J
authors: Long
authors: Department of Radiology, Guangdong General Hospital, Guangzhou 510080, China.
authors: Biao
authors: B
authors: Huang
authors: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou 510640, China.
authors: Tianyou
authors: T
authors: Yu
authors: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou 510640, China.
authors: Wei
authors: W
authors: Wu
authors: Department of MR, Foshan Hospital of Traditional Chinese Medicine, Foshan 528000, China.
authors: Yongjian
authors: Y
authors: Liu
authors: Department of Radiology, Guangdong General Hospital, Guangzhou 510080, China.
authors: Changhong
authors: C
authors: Liang
authors: Department of Psychology, Tsinghua University, Beijing 100084, China.
authors: Pei
authors: P
authors: Sun
mesh_terms: D000161:Acoustic Stimulation; D000328:Adult; D001288:Attention / Q000502:physiology; D001307:Auditory Perception / Q000502:physiology*; D004644:Emotions; D005145:Face; D006801:Humans; D007600:Judgment / Q000502:physiology; D008279:Magnetic Resonance Imaging; D008297:Male; D008875:Middle Aged; D009483:Neuropsychological Tests; D046709:Pattern Recognition, Physiological / Q000502:physiology*; D010775:Photic Stimulation; D012727:Sex Characteristics; D014741:Video Recording; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult
keywords: audiovisual face perception; brain pattern; crossmodal integration; decoding; feature-selective attention; reproducibility