title: The contribution of dynamic visual cues to audiovisual speech perception.
abstract: Seeing a speaker's facial gestures can significantly improve speech comprehension, especially in noisy environments. However, the nature of the visual information from the speaker's facial movements that is relevant for this enhancement is still unclear. Like auditory speech signals, visual speech signals unfold over time and contain both dynamic configural information and luminance-defined local motion cues; two information sources that are thought to engage anatomically and functionally separate visual systems. Whereas, some past studies have highlighted the importance of local, luminance-defined motion cues in audiovisual speech perception, the contribution of dynamic configural information signalling changes in form over time has not yet been assessed. We therefore attempted to single out the contribution of dynamic configural information to audiovisual speech processing. To this aim, we measured word identification performance in noise using unimodal auditory stimuli, and with audiovisual stimuli. In the audiovisual condition, speaking faces were presented as point light displays achieved via motion capture of the original talker. Point light displays could be isoluminant, to minimise the contribution of effective luminance-defined local motion information, or with added luminance contrast, allowing the combined effect of dynamic configural cues and local motion cues. Audiovisual enhancement was found in both the isoluminant and contrast-based luminance conditions compared to an auditory-only condition, demonstrating, for the first time the specific contribution of dynamic configural cues to audiovisual speech improvement. These findings imply that globally processed changes in a speaker's facial shape contribute significantly towards the perception of articulatory gestures and the analysis of audiovisual speech.
authors.affiliation: Center for Visual Science and Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, USA. Electronic address: pjaekl@cvs.rochester.edu.
authors.forename: Philip
authors.initials: P
authors.lastname: Jaekl
authors.affiliation: UBC Vision Lab, Department of Psychology, University of British Colombia, Vancouver, BC, Canada.
authors.forename: Ana
authors.initials: A
authors.lastname: Pesquita
authors.affiliation: Department of Psychology, Queen's University, Kingston, ON, Canada.
authors.forename: Agnes
authors.initials: A
authors.lastname: Alsius
authors.affiliation: Department of Psychology, Queen's University, Kingston, ON, Canada.
authors.forename: Kevin
authors.initials: K
authors.lastname: Munhall
authors.affiliation: Centre for Brain and Cognition, Department of Information Technology and Communications, Universitat Pompeu Fabra, Spain; Institució Catalana de Recerca i Estudis Avançats (ICREA), Spain.
authors.forename: Salvador
authors.initials: S
authors.lastname: Soto-Faraco
mesh_terms: D000161:Acoustic Stimulation; D000328:Adult; D003463:Cues; D005260:Female; D005868:Gestures; D006801:Humans; D008297:Male; D008875:Middle Aged; D009622:Noise; D010775:Photic Stimulation; D013067:Speech Perception; D014796:Visual Perception; D055815:Young Adult
keywords: Audiovisual enhancement; Biological motion; Configural; Speech-in-noise; Visual form; Visual motion; Visual pathways