title: Reproducibility and discriminability of brain patterns of semantic categories enhanced by congruent audiovisual stimuli.
abstract: One of the central questions in cognitive neuroscience is the precise neural representation, or brain pattern, associated with a semantic category. In this study, we explored the influence of audiovisual stimuli on the brain patterns of concepts or semantic categories through a functional magnetic resonance imaging (fMRI) experiment. We used a pattern search method to extract brain patterns corresponding to two semantic categories: "old people" and "young people." These brain patterns were elicited by semantically congruent audiovisual, semantically incongruent audiovisual, unimodal visual, and unimodal auditory stimuli belonging to the two semantic categories. We calculated the reproducibility index, which measures the similarity of the patterns within the same category. We also decoded the semantic categories from these brain patterns. The decoding accuracy reflects the discriminability of the brain patterns between two categories. The results showed that both the reproducibility index of brain patterns and the decoding accuracy were significantly higher for semantically congruent audiovisual stimuli than for unimodal visual and unimodal auditory stimuli, while the semantically incongruent stimuli did not elicit brain patterns with significantly higher reproducibility index or decoding accuracy. Thus, the semantically congruent audiovisual stimuli enhanced the within-class reproducibility of brain patterns and the between-class discriminability of brain patterns, and facilitate neural representations of semantic categories or concepts. Furthermore, we analyzed the brain activity in superior temporal sulcus and middle temporal gyrus (STS/MTG). The strength of the fMRI signal and the reproducibility index were enhanced by the semantically congruent audiovisual stimuli. Our results support the use of the reproducibility index as a potential tool to supplement the fMRI signal amplitude for evaluating multimodal integration.
authors.affiliation: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China. rozenl@u.washington.edu
authors.forename: Yuanqing
authors.initials: Y
authors.lastname: Li
authors.forename: Guangyi
authors.initials: G
authors.lastname: Wang
authors.forename: Jinyi
authors.initials: J
authors.lastname: Long
authors.forename: Zhuliang
authors.initials: Z
authors.lastname: Yu
authors.forename: Biao
authors.initials: B
authors.lastname: Huang
authors.forename: Xiaojian
authors.initials: X
authors.lastname: Li
authors.forename: Tianyou
authors.initials: T
authors.lastname: Yu
authors.forename: Changhong
authors.initials: C
authors.lastname: Liang
authors.forename: Zheng
authors.initials: Z
authors.lastname: Li
authors.forename: Pei
authors.initials: P
authors.lastname: Sun
mesh_terms: D000161:Acoustic Stimulation; D000328:Adult; D000375:Aging / Q000502:physiology; D001921:Brain / Q000502:physiology*; D006801:Humans; D008279:Magnetic Resonance Imaging; D008297:Male; D046709:Pattern Recognition, Physiological; D010775:Photic Stimulation; D015203:Reproducibility of Results; D012660:Semantics; D013702:Temporal Lobe / Q000502:physiology