title: Cross-modal integration during vowel identification in audiovisual speech: a functional magnetic resonance imaging study.
abstract: To investigate the neural substrates of the perception of audiovisual speech, we conducted a functional magnetic resonance imaging study with 28 normal volunteers. We hypothesized that the constraint provided by visually-presented articulatory speech (mouth movements) would lessen the workload for speech identification if the two were concordant, but would increase the workload if the two were discordant. In auditory attention sessions, subjects were required to identify vowels based on auditory speech. Auditory vowel stimuli were presented with concordant or discordant visible articulation movements, unrelated lip movements, and without visual input. In visual attention sessions, subjects were required to identify vowels based on the visually-presented vowel articulation movements. The movements were presented with concordant or discordant uttered vowels and noise, and without sound. Irrespective of the attended modality, concordant conditions significantly shortened the reaction time, whereas discordant conditions lengthened the reaction time. Within the neural substrates that were commonly activated by auditory and visual tasks, the mid superior temporal sulcus showed greater activity for discordant stimuli than concordant stimuli. These findings suggest that the mid superior temporal sulcus plays an important role in the auditory-visual integration process underlying vowel identification.
authors.affiliation: Department of Physiological Sciences, The Graduate University for Advanced Studies (Sokendai), Kanagawa, Japan.
authors.forename: Mika
authors.initials: M
authors.lastname: Murase
authors.forename: Daisuke N
authors.initials: DN
authors.lastname: Saito
authors.forename: Takanori
authors.initials: T
authors.lastname: Kochiyama
authors.forename: Hiroki C
authors.initials: HC
authors.lastname: Tanabe
authors.forename: Satoshi
authors.initials: S
authors.lastname: Tanaka
authors.forename: Tokiko
authors.initials: T
authors.lastname: Harada
authors.forename: Yu
authors.initials: Y
authors.lastname: Aramaki
authors.forename: Manabu
authors.initials: M
authors.lastname: Honda
authors.forename: Norihiro
authors.initials: N
authors.lastname: Sadato
mesh_terms: D000161:Acoustic Stimulation; D000328:Adult; D001288:Attention / Q000502:physiology; D001921:Brain / Q000033:anatomy & histology / Q000502:physiology*; D001931:Brain Mapping; D004292:Dominance, Cerebral / Q000502:physiology; D005260:Female; D006801:Humans; D007802:Language; D007807:Language Tests; D008279:Magnetic Resonance Imaging; D008297:Male; D009415:Nerve Net / Q000033:anatomy & histology / Q000502:physiology; D009434:Neural Pathways / Q000033:anatomy & histology / Q000502:physiology; D010364:Pattern Recognition, Visual / Q000502:physiology; D010700:Phonetics; D010775:Photic Stimulation; D011930:Reaction Time / Q000502:physiology; D011932:Reading; D013067:Speech Perception / Q000502:physiology*; D013702:Temporal Lobe / Q000033:anatomy & histology / Q000502:physiology; D014705:Verbal Behavior / Q000502:physiology*