title: The integration of facial and vocal cues during emotional change perception: EEG markers.
abstract: The ability to detect emotional changes is of primary importance for social living. Though emotional signals are often conveyed by multiple modalities, how emotional changes in vocal and facial modalities integrate into a unified percept has yet to be directly investigated. To address this issue, we asked participants to detect emotional changes delivered by facial, vocal and facial-vocal expressions while behavioral responses and electroencephalogram were recorded. Behavioral results showed that bimodal emotional changes were detected with higher accuracy and shorter response latencies compared with each unimodal condition. Moreover, the detection of emotional change, regardless of modalities, was associated with enhanced amplitudes in the N2 and P3 component, as well as greater theta synchronization. More importantly, the P3 amplitudes and theta synchronization were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions. The superadditive responses in P3 amplitudes and theta synchronization were both positively correlated with the magnitude of the bimodal superadditivity in accuracy. These behavioral and electrophysiological data consistently illustrated an effect of audiovisual integration during the detection of emotional changes, which is most likely mediated by the P3 activity and theta oscillations in brain responses.
authors.affiliation: Key Laboratory of Behavior and Cognitive Psychology in Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi'an 710062, China, Key Laboratory of Modern Teaching Technology, Ministry of Education, Shaanxi Normal University, Xi'an 710062, China.
authors.forename: Xuhai
authors.initials: X
authors.lastname: Chen
authors.affiliation: Key Laboratory of Behavior and Cognitive Psychology in Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi'an 710062, China.
authors.forename: Zhihui
authors.initials: Z
authors.lastname: Pan
authors.affiliation: Key Laboratory of Behavior and Cognitive Psychology in Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi'an 710062, China.
authors.forename: Ping
authors.initials: P
authors.lastname: Wang
authors.affiliation: Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, China, and.
authors.forename: Xiaohong
authors.initials: X
authors.lastname: Yang
authors.affiliation: Key Laboratory of Behavior and Cognitive Psychology in Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi'an 710062, China.
authors.forename: Peng
authors.initials: P
authors.lastname: Liu
authors.affiliation: Key Laboratory of Behavior and Cognitive Psychology in Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi'an 710062, China.
authors.forename: Xuqun
authors.initials: X
authors.lastname: You
authors.affiliation: Key Laboratory of Cognition and Personality of Ministry of Education, School of Psychology, Southwest University, Chongqing 400715, China yuanjiajin168@126.com yuanjiaj@swu.edu.cn.
authors.forename: Jiajin
authors.initials: J
authors.lastname: Yuan
mesh_terms: D003463:Cues; D004569:Electroencephalography; D058407:Electroencephalography Phase Synchronization; D004644:Emotions; D005071:Evoked Potentials / Q000502:physiology; D005149:Facial Expression; D005260:Female; D006801:Humans; D008297:Male; D011597:Psychomotor Performance / Q000502:physiology; D011930:Reaction Time; D012938:Social Perception; D013826:Theta Rhythm / Q000502:physiology; D014831:Voice / Q000502:physiology*; D055815:Young Adult
keywords: bimodal integration; emotional change; event related potential; event related spectral perturbation