title: Template construction grammar: from visual scene description to language comprehension and agrammatism.
abstract: How does the language system coordinate with our visual system to yield flexible integration of linguistic, perceptual, and world-knowledge information when we communicate about the world we perceive? Schema theory is a computational framework that allows the simulation of perceptuo-motor coordination programs on the basis of known brain operating principles such as cooperative computation and distributed processing. We present first its application to a model of language production, SemRep/TCG, which combines a semantic representation of visual scenes (SemRep) with Template Construction Grammar (TCG) as a means to generate verbal descriptions of a scene from its associated SemRep graph. SemRep/TCG combines the neurocomputational framework of schema theory with the representational format of construction grammar in a model linking eye-tracking data to visual scene descriptions. We then offer a conceptual extension of TCG to include language comprehension and address data on the role of both world knowledge and grammatical semantics in the comprehension performances of agrammatic aphasic patients. This extension introduces a distinction between heavy and light semantics. The TCG model of language comprehension offers a computational framework to quantitatively analyze the distributed dynamics of language processes, focusing on the interactions between grammatical, world knowledge, and visual information. In particular, it reveals interesting implications for the understanding of the various patterns of comprehension performances of agrammatic aphasics measured using sentence-picture matching tasks. This new step in the life cycle of the model serves as a basis for exploring the specific challenges that neurolinguistic computational modeling poses to the neuroinformatics community.
authors.affiliation: Neuroscience Graduate Program and USC Brain Project, University of Southern California, Los Angeles, CA, USA, barres@usc.edu.
authors.forename: Victor
authors.initials: V
authors.lastname: Barrès
authors.forename: Jinyong
authors.initials: J
authors.lastname: Lee
mesh_terms: D000818:Animals; D001921:Brain / Q000502:physiology; D032882:Comprehension / Q000502:physiology*; D003198:Computer Simulation; D016208:Databases, Factual; D006801:Humans; D007802:Language; D059167:Mirror Neurons / Q000502:physiology; D008959:Models, Neurological; D011323:Primates
references.citation: Brain Lang. 2005 Jul;94(1):43-53
references.pmid: 15896382
references.citation: Front Neuroinform. 2009 Jul 09;3:23
references.pmid: 19636392
references.citation: Neuroimage. 2006 May 1;30(4):1414-32
references.pmid: 16413796
references.citation: Science. 1995 Jun 16;268(5217):1632-4
references.pmid: 7777863
references.citation: Neuroimage. 2008 May 1;40(4):1633-42
references.pmid: 18328733
references.citation: J Mem Lang. 2007 Nov;57(4):544-569
references.pmid: 18978929
references.citation: Brain Res. 2007 May 18;1146:23-49
references.pmid: 17400197
references.citation: Brain Lang. 1999 Apr;67(2):134-47
references.pmid: 10092346
references.citation: Philos Trans R Soc Lond B Biol Sci. 2001 Aug 29;356(1412):1159-86
references.pmid: 11545697
references.citation: J Cogn Neurosci. 2006 Dec;18(12):2088-107
references.pmid: 17129193
references.citation: J Neurolinguistics. 2009 Jan 1;22(1):91-108
references.pmid: 20428488
references.citation: Trends Cogn Sci. 2002 Feb 1;6(2):78-84
references.pmid: 15866191
references.citation: Brain Lang. 1984 Mar;21(2):208-18
references.pmid: 6200177
references.citation: Neural Netw. 2000 Oct-Nov;13(8-9):975-97
references.pmid: 11156205
references.citation: Brain Lang. 1999 Oct 15;70(1):133-8
references.pmid: 10534377
references.citation: Front Neuroinform. 2012 Dec 27;6:30
references.pmid: 23293600
references.citation: Aphasiology. 2011;25(11):1286-1307
references.pmid: 22923879
references.citation: Trends Cogn Sci. 2009 Apr;13(4):175-81
references.pmid: 19223226
references.citation: J Exp Psychol Learn Mem Cogn. 2004 May;30(3):687-96
references.pmid: 15099136
references.citation: Psychol Rev. 2006 Oct;113(4):787-821
references.pmid: 17014303
references.citation: Cogn Sci. 2006 May 6;30(3):481-529
references.pmid: 21702823
references.citation: Brain Lang. 1976 Oct;3(4):572-82
references.pmid: 974731
references.citation: Brain Lang. 1989 Oct;37(3):419-39
references.pmid: 2478253
references.citation: Neuroimage. 2009 Oct 15;48(1):14-7
references.pmid: 19481158
references.citation: Neuroimage. 2006 May 1;30(4):1255-72
references.pmid: 16473023
references.citation: J Psycholinguist Res. 2009 Feb;38(1):1-9
references.pmid: 18521754
references.citation: Cognition. 1999 Dec 17;73(3):247-64
references.pmid: 10585516
references.citation: PLoS Comput Biol. 2005 Sep;1(4):e42
references.pmid: 16201007
references.citation: Brain Lang. 1999 May;67(3):242-7
references.pmid: 10210633
references.citation: Cogn Psychol. 2003 Sep;47(2):164-203
references.pmid: 12948517
references.citation: Brain Lang. 1998 Apr;62(2):255-97
references.pmid: 9576824
references.citation: Neuroinformatics. 2012 Apr;10(2):141-58
references.pmid: 22068921
references.citation: AJNR Am J Neuroradiol. 1998 Nov-Dec;19(10):1869-77
references.pmid: 9874539
references.citation: Brain Lang. 1982 May;16(1):61-72
references.pmid: 7104682
references.citation: J Neurosci. 2012 Oct 10;32(41):14125-31
references.pmid: 23055482
references.citation: Nat Rev Neurosci. 2002 Apr;3(4):319-21
references.pmid: 11967563
references.citation: Cogn Psychol. 2001 Jun;42(4):368-407
references.pmid: 11368528
references.citation: Neuroinformatics. 2014 Jan;12(1):5-26
references.pmid: 24234915
references.citation: Cognition. 2004 May-Jun;92(1-2):67-99
references.pmid: 15037127
references.citation: Cognition. 1996 Mar;58(3):289-308
references.pmid: 8871341
references.citation: Neural Netw. 2003 Nov;16(9):1237-60
references.pmid: 14622882
references.citation: Neural Netw. 2013 Jan;37:66-92
references.pmid: 23177656
references.citation: Cereb Cortex. 2012 Jul;22(7):1662-70
references.pmid: 21934094
references.citation: Behav Brain Sci. 2000 Feb;23(1):1-21; discussion 21-71
references.pmid: 11303337
references.citation: Trends Cogn Sci. 2005 Sep;9(9):416-23
references.pmid: 16054419
references.citation: Front Psychol. 2010 Dec 17;1:222
references.pmid: 21833277
references.citation: Hum Brain Mapp. 2005 May;25(1):185-98
references.pmid: 15846810
references.citation: Psychol Sci. 2011 Sep;22(9):1173-82
references.pmid: 21841148
references.citation: Front Neuroinform. 2011 Jun 27;5:4
references.pmid: 21743807
references.citation: Brain Res. 2008 Aug 15;1225:146-62
references.pmid: 18561901