title: Cross-modal prediction in speech depends on prior linguistic experience.
abstract: The sight of a speaker's facial movements during the perception of a spoken message can benefit speech processing through online predictive mechanisms. Recent evidence suggests that these predictive mechanisms can operate across sensory modalities, that is, vision and audition. However, to date, behavioral and electrophysiological demonstrations of cross-modal prediction in speech have considered only the speaker's native language. Here, we address a question of current debate, namely whether the level of representation involved in cross-modal prediction is phonological or pre-phonological. We do this by testing participants in an unfamiliar language. If cross-modal prediction is predominantly based on phonological representations tuned to the phonemic categories of the native language of the listener, then it should be more effective in the listener's native language than in an unfamiliar one. We tested Spanish and English native speakers in an audiovisual matching paradigm that allowed us to evaluate visual-to-auditory prediction, using sentences in the participant's native language and in an unfamiliar language. The benefits of cross-modal prediction were only seen in the native language, regardless of the particular language or participant's linguistic background. This pattern of results implies that cross-modal visual-to-auditory prediction during speech processing makes strong use of phonological representations, rather than low-level spatiotemporal correlations across facial movements and sounds.
authors: Departament de Tecnologies de la Informació i les Comunicacions, Center of Brain and Cognition (CBC), Universitat Pompeu Fabra, c/Roc Boronat 138, 08018, Barcelona, Spain.
authors: Carolina
authors: C
authors: Sánchez-García
authors: James T
authors: JT
authors: Enns
authors: Salvador
authors: S
authors: Soto-Faraco
mesh_terms: D005260:Female; D006801:Humans; D007802:Language; D007858:Learning / Q000502:physiology*; D008297:Male; D013060:Speech / Q000502:physiology*; D013067:Speech Perception / Q000502:physiology*; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult
references: Trends Cogn Sci. 2007 Mar;11(3):105-10
references: 17254833
references: Lang Speech. 2003;46(Pt 2-3):183-216
references: 14748444
references: Brain Res. 2010 Apr 6;1323:84-93
references: 20117103
references: Psychol Res. 2007 Jan;71(1):4-12
references: 16362332
references: PLoS One. 2011;6(10):e25198
references: 21998642
references: Nat Neurosci. 2011 Jun;14(6):797-801
references: 21552273
references: J Exp Psychol Hum Percept Perform. 2005 Oct;31(5):912-918
references: 16262488
references: Nature. 1976 Dec 23-30;264(5588):746-8
references: 1012311
references: Eur J Neurosci. 2004 Oct;20(8):2225-34
references: 15450102
references: Science. 2007 May 25;316(5828):1159
references: 17525331
references: Dev Sci. 2008 Mar;11(2):306-20
references: 18333984
references: J Exp Child Psychol. 1997 Jul;66(1):85-110
references: 9226935
references: J Cogn Neurosci. 2010 Jul;22(7):1583-96
references: 19583474
references: Front Hum Neurosci. 2010 Mar 22;4:25
references: 20631856
references: PLoS Biol. 2010 Aug 10;8(8):e1000445
references: 20711473
references: Behav Res Methods Instrum Comput. 2003 Feb;35(1):116-24
references: 12723786
references: J Exp Psychol Hum Percept Perform. 1991 Feb;17(1):278-88
references: 1826317
references: Perception. 2003;32(1):111-20
references: 12613790
references: Cognition. 1985 Oct;21(1):1-36
references: 4075760
references: Brain Res Cogn Brain Res. 1996 Mar;3(2):131-41
references: 8713554
references: J Acoust Soc Am. 2001 May;109(5 Pt 1):2272-5
references: 11386581
references: Proc Natl Acad Sci U S A. 2005 Jan 25;102(4):1181-6
references: 15647358
references: Psychol Sci. 2001 Nov;12(6):445-9
references: 11760129
references: J Speech Lang Hear Res. 2007 Aug;50(4):940-67
references: 17675598
references: J Acoust Soc Am. 2011 Jul;130(1):1-4
references: 21786870
references: Psychol Rev. 1967 Nov;74(6):431-61
references: 4170865
references: Psychol Rev. 2008 Apr;115(2):357-95
references: 18426294
references: J Neurosci. 2009 Oct 28;29(43):13445-53
references: 19864557
references: Proc Natl Acad Sci U S A. 2001 Nov 6;98(23):13464-71
references: 11698690
references: Cognition. 2004 Sep;93(2):B69-78
references: 15147940
references: Percept Psychophys. 2000 Feb;62(2):233-52
references: 10723205
references: PLoS Comput Biol. 2009 Jul;5(7):e1000436
references: 19609344
references: J Exp Psychol Learn Mem Cogn. 2005 May;31(3):443-67
references: 15910130
references: Eur J Neurosci. 2002 Jan;15(2):399-402
references: 11849307
references: Cereb Cortex. 2007 Oct;17(10):2387-99
references: 17218482
references: PLoS One. 2009;4(3):e5047
references: 19333386
references: Proc Natl Acad Sci U S A. 2009 Jun 30;106(26):10598-602
references: 19541648
references: Neuroimage. 2005 Mar;25(1):76-89
references: 15734345
references: J Cogn Neurosci. 2007 Dec;19(12):1964-73
references: 17892381
references: Nat Neurosci. 2005 Aug;8(8):1117-21
references: 16007080
references: Child Dev. 1981 Mar;52(1):349-55
references: 7238150
references: Curr Biol. 2009 Apr 28;19(8):661-7
references: 19327997
references: Proc Natl Acad Sci U S A. 2012 Jan 31;109(5):1431-6
references: 22307596
references: Trends Cogn Sci. 2009 Jan;13(1):14-9
references: 19070534
references: Br J Psychol. 2001 May;92(Pt 2):339-55
references: 11417785
references: J Exp Psychol Hum Percept Perform. 1983 Oct;9(5):753-71
references: 6227688
references: Brain Res Cogn Brain Res. 2005 May;23(2-3):429-35
references: 15820649
references: Cognition. 1997 Sep;64(3):B9-17
references: 9426508
references: J Neurosci. 2012 Jun 20;32(25):8443-53
references: 22723684
references: J Acoust Soc Am. 2000 Sep;108(3 Pt 1):1197-208
references: 11008820
references: Cognition. 1999 Sep 30;72(2):111-23
references: 10553668
references: Annu Rev Psychol. 1999;50:509-35
references: 10074686
references: Philos Trans R Soc Lond B Biol Sci. 2008 Mar 12;363(1493):1071-86
references: 17890189
references: J Acoust Soc Am. 1984 Jun;75(6):1866-78
references: 6747097
references: Percept Psychophys. 2007 Feb;69(2):218-31
references: 17557592
references: Trends Cogn Sci. 2008 Mar;12(3):106-13
references: 18280772