title: Hearing and seeing meaning in speech and gesture: insights from brain and behaviour.
abstract: As we speak, we use not only the arbitrary form-meaning mappings of the speech channel but also motivated form-meaning correspondences, i.e. iconic gestures that accompany speech (e.g. inverted V-shaped hand wiggling across gesture space to demonstrate walking). This article reviews what we know about processing of semantic information from speech and iconic gestures in spoken languages during comprehension of such composite utterances. Several studies have shown that comprehension of iconic gestures involves brain activations known to be involved in semantic processing of speech: i.e. modulation of the electrophysiological recording component N400, which is sensitive to the ease of semantic integration of a word to previous context, and recruitment of the left-lateralized frontal-posterior temporal network (left inferior frontal gyrus (IFG), medial temporal gyrus (MTG) and superior temporal gyrus/sulcus (STG/S)). Furthermore, we integrate the information coming from both channels recruiting brain areas such as left IFG, posterior superior temporal sulcus (STS)/MTG and even motor cortex. Finally, this integration is flexible: the temporal synchrony between the iconic gesture and the speech segment, as well as the perceived communicative intent of the speaker, modulate the integration process. Whether these findings are special to gestures or are shared with actions or other visual accompaniments to speech (e.g. lips) or other visual symbols such as pictures are discussed, as well as the implications for a multimodal view of language.
authors.affiliation: Department of Linguistics, Radboud University Nijmegen, Erasmus Plain 1, 6500 HD, Nijmegen, The Netherlands Max Planck Institute for Psycholinguistics, Wundtlaan 1, Nijmegen 6525 JT, The Netherlands asliozu@mpi.nl.
authors.forename: Aslı
authors.initials: A
authors.lastname: Özyürek
mesh_terms: D001921:Brain / Q000502:physiology*; D032882:Comprehension / Q000502:physiology*; D005868:Gestures; D006309:Hearing / Q000502:physiology*; D006801:Humans; D012660:Semantics; D013060:Speech
keywords: brain; co-speech gestures; iconicity; multimodal language; semantics
references.citation: Front Psychol. 2010 Dec 31;1:227
references.pmid: 21833282
references.citation: Psychon Bull Rev. 2007 Feb;14(1):57-63
references.pmid: 17546731
references.citation: Philos Trans R Soc Lond B Biol Sci. 2014 Sep 19;369(1651):20130293
references.pmid: 25092661
references.citation: Science. 1980 Jan 11;207(4427):203-5
references.pmid: 7350657
references.citation: PLoS One. 2012;7(11):e51207
references.pmid: 23226488
references.citation: J Cogn Neurosci. 2009 Apr;21(4):821-36
references.pmid: 18578601
references.citation: Hum Brain Mapp. 2009 Oct;30(10):3309-24
references.pmid: 19350562
references.citation: Trends Cogn Sci. 2005 Sep;9(9):416-23
references.pmid: 16054419
references.citation: Philos Trans R Soc Lond B Biol Sci. 2014 Sep 19;369(1651):20130300
references.pmid: 25092668
references.citation: Neuroimage. 2007 Apr 15;35(3):1278-86
references.pmid: 17321757
references.citation: Neuroimage. 2008 Feb 15;39(4):2010-24
references.pmid: 18093845
references.citation: Neuropsychologia. 2006;44(2):178-90
references.pmid: 16005477
references.citation: Curr Biol. 2006 Sep 19;16(18):1818-23
references.pmid: 16979559
references.citation: Brain Lang. 2011 Dec;119(3):184-95
references.pmid: 21864890
references.citation: Hum Brain Mapp. 2011 Apr;32(4):520-33
references.pmid: 21391245
references.citation: Cereb Cortex. 2009 Jul;19(7):1493-503
references.pmid: 19001084
references.citation: Dev Sci. 2012 Mar;15(2):165-80
references.pmid: 22356173
references.citation: Cereb Cortex. 2011 May;21(5):1066-75
references.pmid: 20851853
references.citation: Brain Lang. 2004 Apr;89(1):253-60
references.pmid: 15010257
references.citation: Hum Brain Mapp. 2014 Mar;35(3):900-17
references.pmid: 23238964
references.citation: J Cogn Neurosci. 2007 Apr;19(4):605-16
references.pmid: 17381252
references.citation: Neuroimage. 2009 Oct 1;47(4):1992-2004
references.pmid: 19497376
references.citation: PLoS One. 2012;7(9):e46108
references.pmid: 23049951
references.citation: Philos Trans R Soc Lond B Biol Sci. 2007 May 29;362(1481):801-11
references.pmid: 17412680
references.citation: Neuroimage. 2010 Jan 1;49(1):875-84
references.pmid: 19733670
references.citation: J Cogn Neurosci. 2008 Jul;20(7):1235-49
references.pmid: 18284352
references.citation: J Deaf Stud Deaf Educ. 2005 Winter;10(1):3-37
references.pmid: 15585746
references.citation: Neuroimage. 2007 Feb 1;34(3):1270-9
references.pmid: 17142061
references.citation: Brain Lang. 2007 Jun;101(3):260-77
references.pmid: 17533001
references.citation: Psychol Sci. 2010 Feb;21(2):260-7
references.pmid: 20424055
references.citation: Cogn Sci. 2011 Jan-Feb;35(1):171-83
references.pmid: 21428996
references.citation: Neuroimage. 2009 Apr 15;45(3):1009-19
references.pmid: 19167505
references.citation: Brain Lang. 2007 Jun;101(3):278-89
references.pmid: 17416411
references.citation: Psychophysiology. 2005 Nov;42(6):654-67
references.pmid: 16364061
references.citation: Cereb Cortex. 2005 Jul;15(7):986-94
references.pmid: 15513929
references.citation: Philos Trans R Soc Lond B Biol Sci. 2008 Mar 12;363(1493):1001-10
references.pmid: 17827105
references.citation: Cereb Cortex. 2005 Aug;15(8):1261-9
references.pmid: 15635062
references.citation: Proc Natl Acad Sci U S A. 2009 Dec 8;106(49):20664-9
references.pmid: 19923436
references.citation: Cortex. 2012 Jul;48(7):857-70
references.pmid: 21397223
references.citation: J Neurosci. 2007 Jul 25;27(30):7881-7
references.pmid: 17652579
references.citation: Brain Lang. 2007 Jun;101(3):222-33
references.pmid: 16997367
references.citation: J Cogn Neurosci. 2007 Jul;19(7):1175-92
references.pmid: 17583993
references.citation: Psychol Sci. 2005 Feb;16(2):85-9
references.pmid: 15686572
references.citation: Curr Biol. 2009 Apr 28;19(8):661-7
references.pmid: 19327997
references.citation: Cereb Cortex. 2008 Nov;18(11):2574-85
references.pmid: 18308708
references.citation: Soc Cogn Affect Neurosci. 2015 Feb;10(2):255-61
references.pmid: 24652857
references.citation: J Cogn Neurosci. 2011 Aug;23(8):1845-54
references.pmid: 20201632
references.citation: Neuropsychologia. 2013 Aug;51(9):1716-25
references.pmid: 23727570
references.citation: Cereb Cortex. 2007 Oct;17(10):2322-33
references.pmid: 17159232
references.citation: J Cogn Neurosci. 2011 Jul;23(7):1648-63
references.pmid: 20350188
references.citation: Hum Brain Mapp. 2009 Nov;30(11):3509-26
references.pmid: 19384890
references.citation: Neuropsychologia. 2009 Jan;47(1):169-79
references.pmid: 18771673
references.citation: Cogn Sci. 2009;33(1):115
references.pmid: 20126430
references.citation: Cereb Cortex. 2001 Dec;11(12):1110-23
references.pmid: 11709482