title: Semantic integration of audio-visual information of polyphonic characters in a sentence context: an event-related potential study.
abstract: In the Chinese language, a polyphone is a kind of special character that has more than one pronunciation, with each pronunciation corresponding to a different meaning. Here, we aimed to reveal the cognitive processing of audio-visual information integration of polyphones in a sentence context using the event-related potential (ERP) method. Sentences ending with polyphones were presented to subjects simultaneously in both an auditory and a visual modality. Four experimental conditions were set in which the visual presentations were the same, but the pronunciations of the polyphones were: the correct pronunciation; another pronunciation of the polyphone; a semantically appropriate pronunciation but not the pronunciation of the polyphone; or a semantically inappropriate pronunciation but also not the pronunciation of the polyphone. The behavioral results demonstrated significant differences in response accuracies when judging the semantic meanings of the audio-visual sentences, which reflected the different demands on cognitive resources. The ERP results showed that in the early stage, abnormal pronunciations were represented by the amplitude of the P200 component. Interestingly, because the phonological information mediated access to the lexical semantics, the amplitude and latency of the N400 component changed linearly across conditions, which may reflect the gradually increased semantic mismatch in the four conditions when integrating the auditory pronunciation with the visual information. Moreover, the amplitude of the late positive shift (LPS) showed a significant correlation with the behavioral response accuracies, demonstrating that the LPS component reveals the demand of cognitive resources for monitoring and resolving semantic conflicts when integrating the audio-visual information.
authors.affiliation: Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Yaguan Road No. 135, Jinnan District, Tianjin, 300350, People's Republic of China.
authors.forename: Hong
authors.initials: H
authors.lastname: Liu
authors.affiliation: Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Yaguan Road No. 135, Jinnan District, Tianjin, 300350, People's Republic of China.
authors.forename: Gaoyan
authors.initials: G
authors.lastname: Zhang
authors.affiliation: Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Yaguan Road No. 135, Jinnan District, Tianjin, 300350, People's Republic of China. liubaolin@tsinghua.edu.cn.|State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing, 100084, People's Republic of China. liubaolin@tsinghua.edu.cn.
authors.forename: Baolin
authors.initials: B
authors.lastname: Liu
mesh_terms: D000161:Acoustic Stimulation; D000328:Adult; D000704:Analysis of Variance; D001931:Brain Mapping; D004569:Electroencephalography; D005071:Evoked Potentials / Q000502:physiology*; D005260:Female; D006801:Humans; D008297:Male; D010700:Phonetics; D010775:Photic Stimulation; D011930:Reaction Time / Q000502:physiology; D011932:Reading; D012660:Semantics; D055815:Young Adult
keywords: Audio-visual semantic integration; ERPs; N400; Polyphone; Sentence context