title: Semantic integration of differently asynchronous audio-visual information in videos of real-world events in cognitive processing: an ERP study.
abstract: In the real world, some of the auditory and visual information received by the human brain are temporally asynchronous. How is such information integrated in cognitive processing in the brain? In this paper, we aimed to study the semantic integration of differently asynchronous audio-visual information in cognitive processing using ERP (event-related potential) method. Subjects were presented with videos of real world events, in which the auditory and visual information are temporally asynchronous. When the critical action was prior to the sound, sounds incongruous with the preceding critical actions elicited a N400 effect when compared to congruous condition. This result demonstrates that semantic contextual integration indexed by N400 also applies to cognitive processing of multisensory information. In addition, the N400 effect is early in latency when contrasted with other visually induced N400 studies. It is shown that cross modal information is facilitated in time when contrasted with visual information in isolation. When the sound was prior to the critical action, a larger late positive wave was observed under the incongruous condition compared to congruous condition. P600 might represent a reanalysis process, in which the mismatch between the critical action and the preceding sound was evaluated. It is shown that environmental sound may affect the cognitive processing of a visual event.
authors.affiliation: Department of Computer Science and Technology, Tsinghua University, Beijing 100084, PR China. liubaolin@tsinghua.edu.cn
authors.forename: Baolin
authors.initials: B
authors.lastname: Liu
authors.forename: Guangning
authors.initials: G
authors.lastname: Wu
authors.forename: Zhongning
authors.initials: Z
authors.lastname: Wang
authors.forename: Xiang
authors.initials: X
authors.lastname: Ji
mesh_terms: D000161:Acoustic Stimulation; D001307:Auditory Perception / Q000502:physiology*; D001921:Brain / Q000502:physiology*; D001931:Brain Mapping; D004569:Electroencephalography; D005071:Evoked Potentials / Q000502:physiology*; D005260:Female; D006801:Humans; D008297:Male; D010775:Photic Stimulation; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult