title: Modeling language and cognition with deep unsupervised learning: a tutorial overview.
abstract: Deep unsupervised learning in stochastic recurrent neural networks with many layers of hidden units is a recent breakthrough in neural computation research. These networks build a hierarchy of progressively more complex distributed representations of the sensory data by fitting a hierarchical generative model. In this article we discuss the theoretical foundations of this approach and we review key issues related to training, testing and analysis of deep networks for modeling language and cognitive processing. The classic letter and word perception problem of McClelland and Rumelhart (1981) is used as a tutorial example to illustrate how structured and abstract representations may emerge from deep generative learning. We argue that the focus on deep architectures and generative (rather than discriminative) learning represents a crucial step forward for the connectionist modeling enterprise, because it offers a more plausible model of cortical learning as well as a way to bridge the gap between emergentist connectionist models and structured Bayesian models of cognition.
authors: Computational Cognitive Neuroscience Lab, Department of General Psychology, University of Padova Padova, Italy ; IRCCS San Camillo Neurorehabilitation Hospital Venice-Lido, Italy.
authors: Marco
authors: M
authors: Zorzi
authors: Alberto
authors: A
authors: Testolin
authors: Ivilin P
authors: IP
authors: Stoianov
keywords: connectionist modeling; deep learning; hierarchical generative models; neural networks; unsupervised learning; visual word recognition
references: Neural Comput. 1995 Sep;7(5):889-904
references: 7584891
references: Neural Comput. 2006 Jul;18(7):1527-54
references: 16764513
references: Cortex. 2004 Feb;40(1):194-6
references: 15174483
references: Neuron. 2007 Jul 5;55(1):143-56
references: 17610823
references: Trends Cogn Sci. 2010 Aug;14(8):357-64
references: 20576465
references: Psychol Rev. 1989 Oct;96(4):523-68
references: 2798649
references: Science. 2006 Jul 28;313(5786):504-7
references: 16873662
references: Cogn Psychol. 2010 Sep;61(2):106-51
references: 20510406
references: Cogn Process. 2012 Aug;13 Suppl 1:S141-6
references: 22802037
references: Nat Neurosci. 2012 Jan 08;15(2):194-6
references: 22231428
references: Front Psychol. 2013 May 06;4:251
references: 23653617
references: Trends Cogn Sci. 1998 Nov 1;2(11):455-62
references: 21227277
references: Psychol Rev. 1999 Jul;106(3):491-528
references: 10467896
references: Behav Brain Sci. 2011 Aug;34(4):169-88; disuccsion 188-231
references: 21864419
references: Trends Cogn Sci. 2010 Aug;14(8):348-56
references: 20598626
references: Psychol Rev. 1982 Jan;89(1):60-94
references: 7058229
references: J Math Biol. 1982;15(3):267-73
references: 7153672
references: Trends Cogn Sci. 2005 Jul;9(7):335-41
references: 15951224
references: Trends Cogn Sci. 2006 Jul;10(7):335-44
references: 16784883
references: Curr Opin Neurobiol. 2004 Aug;14(4):481-7
references: 15321069
references: Psychol Rev. 1958 Nov;65(6):386-408
references: 13602029
references: Nat Neurosci. 1999 Jan;2(1):79-87
references: 10195184
references: Philos Trans R Soc Lond B Biol Sci. 2010 Jan 12;365(1537):177-84
references: 20008395
references: Philos Trans R Soc Lond B Biol Sci. 2005 Apr 29;360(1456):815-36
references: 15937014
references: Trends Cogn Sci. 2006 Jul;10(7):287-91
references: 16807064
references: Psychol Rev. 2004 Apr;111(2):309-32
references: 15065912
references: Cogn Sci. 2014 Aug;38(6):1139-89
references: 25098813
references: Neural Comput. 2001 Jun;13(6):1199-241
references: 11387044
references: Neuron. 2012 Feb 9;73(3):415-34
references: 22325196
references: Wiley Interdiscip Rev Cogn Sci. 2011 Sep;2(5):580-593
references: 26302308
references: Nat Neurosci. 1999 Nov;2(11):1019-25
references: 10526343
references: IEEE Trans Pattern Anal Mach Intell. 1984 Jun;6(6):721-41
references: 22499653
references: Trends Cogn Sci. 2007 Oct;11(10):428-34
references: 17921042
references: Neural Comput. 2002 Aug;14(8):1771-800
references: 12180402
references: Nature. 1996 Jun 13;381(6583):607-9
references: 8637596
references: Cogn Sci. 2014 Aug;38(6):1078-101
references: 23800216
references: Psychon Bull Rev. 2002 Dec;9(4):829-35
references: 12613690
references: Cogn Sci. 2013 Jul;37(5):800-28
references: 23489148
references: Psychol Rev. 2007 Apr;114(2):273-315
references: 17500628
references: Psychol Rev. 2004 Jul;111(3):662-720
references: 15250780
references: Behav Brain Sci. 2013 Jun;36(3):181-204
references: 23663408
references: Psychol Rev. 1996 Jan;103(1):56-115
references: 8650300
references: Behav Brain Sci. 1998 Feb;21(1):1-17; discussion 17-54
references: 10097010
references: Psychol Rev. 1974 Mar;81(2):99-118
references: 4817613
references: Philos Trans R Soc Lond B Biol Sci. 1997 Aug 29;352(1358):1177-90
references: 9304685
references: IEEE Trans Pattern Anal Mach Intell. 2013 Aug;35(8):1798-828
references: 23787338