title: Electrophysiological evidence for speech-specific audiovisual integration.
abstract: Lip-read speech is integrated with heard speech at various neural levels. Here, we investigated the extent to which lip-read induced modulations of the auditory N1 and P2 (measured with EEG) are indicative of speech-specific audiovisual integration, and we explored to what extent the ERPs were modulated by phonetic audiovisual congruency. In order to disentangle speech-specific (phonetic) integration from non-speech integration, we used Sine-Wave Speech (SWS) that was perceived as speech by half of the participants (they were in speech-mode), while the other half was in non-speech mode. Results showed that the N1 obtained with audiovisual stimuli peaked earlier than the N1 evoked by auditory-only stimuli. This lip-read induced speeding up of the N1 occurred for listeners in speech and non-speech mode. In contrast, if listeners were in speech-mode, lip-read speech also modulated the auditory P2, but not if listeners were in non-speech mode, thus revealing speech-specific audiovisual binding. Comparing ERPs for phonetically congruent audiovisual stimuli with ERPs for incongruent stimuli revealed an effect of phonetic stimulus congruency that started at ~200 ms after (in)congruence became apparent. Critically, akin to the P2 suppression, congruency effects were only observed if listeners were in speech mode, and not if they were in non-speech mode. Using identical stimuli, we thus confirm that audiovisual binding involves (partially) different neural mechanisms for sound processing in speech and non-speech mode.
authors.affiliation: Basque Center on Cognition, Brain and Language, Paseo Mikeletegi 69, 2nd floor, 20009 Donostia, Spain; Tilburg University, Department of Cognitive Neuropsychology, P.O. Box 90153, Warandelaan 2, 5000 LE, Tilburg, the Netherlands.
authors.forename: Martijn
authors.initials: M
authors.lastname: Baart
authors.affiliation: Tilburg University, Department of Cognitive Neuropsychology, P.O. Box 90153, Warandelaan 2, 5000 LE, Tilburg, the Netherlands.
authors.forename: Jeroen J
authors.initials: JJ
authors.lastname: Stekelenburg
authors.affiliation: Tilburg University, Department of Cognitive Neuropsychology, P.O. Box 90153, Warandelaan 2, 5000 LE, Tilburg, the Netherlands. Electronic address: J.Vroomen@uvt.nl.
authors.forename: Jean
authors.initials: J
authors.lastname: Vroomen
mesh_terms: D000161:Acoustic Stimulation; D000293:Adolescent; D000328:Adult; D001307:Auditory Perception / Q000502:physiology; D001921:Brain / Q000502:physiology*; D004569:Electroencephalography; D005071:Evoked Potentials; D005072:Evoked Potentials, Auditory; D005260:Female; D006801:Humans; D008085:Lipreading; D008297:Male; D010700:Phonetics; D010775:Photic Stimulation; D013060:Speech; D013067:Speech Perception / Q000502:physiology*; D013997:Time Factors; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult
keywords: Audiovisual integration; Audiovisual speech; N1; P2; Sine-wave speech