title: The modulatory effect of semantic familiarity on the audiovisual integration of face-name pairs.
abstract: To recognize individuals, the brain often integrates audiovisual information from familiar or unfamiliar faces, voices, and auditory names. To date, the effects of the semantic familiarity of stimuli on audiovisual integration remain unknown. In this functional magnetic resonance imaging (fMRI) study, we used familiar/unfamiliar facial images, auditory names, and audiovisual face-name pairs as stimuli to determine the influence of semantic familiarity on audiovisual integration. First, we performed a general linear model analysis using fMRI data and found that audiovisual integration occurred for familiar congruent and unfamiliar face-name pairs but not for familiar incongruent pairs. Second, we decoded the familiarity categories of the stimuli (familiar vs. unfamiliar) from the fMRI data and calculated the reproducibility indices of the brain patterns that corresponded to familiar and unfamiliar stimuli. The decoding accuracy rate was significantly higher for familiar congruent versus unfamiliar face-name pairs (83.2%) than for familiar versus unfamiliar faces (63.9%) and for familiar versus unfamiliar names (60.4%). This increase in decoding accuracy was not observed for familiar incongruent versus unfamiliar pairs. Furthermore, compared with the brain patterns associated with facial images or auditory names, the reproducibility index was significantly improved for the brain patterns of familiar congruent face-name pairs but not those of familiar incongruent or unfamiliar pairs. Our results indicate the modulatory effect that semantic familiarity has on audiovisual integration. Specifically, neural representations were enhanced for familiar congruent face-name pairs compared with visual-only faces and auditory-only names, whereas this enhancement effect was not observed for familiar incongruent or unfamiliar pairs. Hum Brain Mapp 37:4333-4348, 2016. © 2016 Wiley Periodicals, Inc.
authors.affiliation: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.|Guangzhou Key Laboratory of Brain Computer Interaction and Applications, Guangzhou, China.
authors.forename: Yuanqing
authors.initials: Y
authors.lastname: Li
authors.affiliation: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.|Guangzhou Key Laboratory of Brain Computer Interaction and Applications, Guangzhou, China.
authors.forename: Fangyi
authors.initials: F
authors.lastname: Wang
authors.affiliation: Department of Radiology, Guangdong General Hospital, Guangzhou, China.
authors.forename: Biao
authors.initials: B
authors.lastname: Huang
authors.affiliation: Department of Radiology, Guangdong General Hospital, Guangzhou, China.
authors.forename: Wanqun
authors.initials: W
authors.lastname: Yang
authors.affiliation: Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.|Guangzhou Key Laboratory of Brain Computer Interaction and Applications, Guangzhou, China.
authors.forename: Tianyou
authors.initials: T
authors.lastname: Yu
authors.affiliation: Department of Experimental Psychology, Ghent University, Ghent, Belgium.
authors.forename: Durk
authors.initials: D
authors.lastname: Talsma
mesh_terms: D000328:Adult; D001921:Brain / Q000000981:diagnostic imaging / Q000502:physiology*; D001931:Brain Mapping; D000066499:Facial Recognition / Q000502:physiology*; D005260:Female; D006801:Humans; D016014:Linear Models; D008279:Magnetic Resonance Imaging; D008297:Male; D009275:Names; D009483:Neuropsychological Tests; D021641:Recognition, Psychology / Q000502:physiology*; D015203:Reproducibility of Results; D012660:Semantics; D013067:Speech Perception / Q000502:physiology*
keywords: audiovisual face-name pairs; audiovisual semantic integration; decoding; reproducibility; semantic familiarity