title: Prediction across sensory modalities: A neurocomputational model of the McGurk effect.
abstract: The McGurk effect is a textbook illustration of the automaticity with which the human brain integrates audio-visual speech. It shows that even incongruent audiovisual (AV) speech stimuli can be combined into percepts that correspond neither to the auditory nor to the visual input, but to a mix of both. Typically, when presented with, e.g., visual /aga/ and acoustic /aba/ we perceive an illusory /ada/. In the inverse situation, however, when acoustic /aga/ is paired with visual /aba/, we perceive a combination of both stimuli, i.e., /abga/ or /agba/. Here we assessed the role of dynamic cross-modal predictions in the outcome of AV speech integration using a computational model that processes continuous audiovisual speech sensory inputs in a predictive coding framework. The model involves three processing levels: sensory units, units that encode the dynamics of stimuli, and multimodal recognition/identity units. The model exhibits a dynamic prediction behavior because evidence about speech tokens can be asynchronous across sensory modality, allowing for updating the activity of the recognition units from one modality while sending top-down predictions to the other modality. We explored the model's response to congruent and incongruent AV stimuli and found that, in the two-dimensional feature space spanned by the speech second formant and lip aperture, fusion stimuli are located in the neighborhood of congruent /ada/, which therefore provides a valid match. Conversely, stimuli that lead to combination percepts do not have a unique valid neighbor. In that case, acoustic and visual cues are both highly salient and generate conflicting predictions in the other modality that cannot be fused, forcing the elaboration of a combinatorial solution. We propose that dynamic predictive mechanisms play a decisive role in the dichotomous perception of incongruent audiovisual inputs.
authors.affiliation: Department of Basic Neurosciences, University of Geneva, Geneva, Switzerland. Electronic address: miren.olasagasti@unige.ch.
authors.forename: Itsaso
authors.initials: I
authors.lastname: Olasagasti
authors.affiliation: Department of Basic Neurosciences, University of Geneva, Geneva, Switzerland.
authors.forename: Sophie
authors.initials: S
authors.lastname: Bouton
authors.affiliation: Department of Basic Neurosciences, University of Geneva, Geneva, Switzerland.
authors.forename: Anne-Lise
authors.initials: AL
authors.lastname: Giraud
mesh_terms: D000161:Acoustic Stimulation; D000465:Algorithms; D001307:Auditory Perception / Q000502:physiology*; D003143:Communication Devices for People with Disabilities; D003198:Computer Simulation; D003463:Cues; D006801:Humans; D008046:Lip; D008959:Models, Neurological; D010775:Photic Stimulation; D012677:Sensation / Q000502:physiology*; D013067:Speech Perception / Q000502:physiology*; D014796:Visual Perception / Q000502:physiology
keywords: Audiovisual integration; Computational modeling; McGurk effect; Predictive coding