title: Reinforcement learning on slow features of high-dimensional input streams.
abstract: Humans and animals are able to learn complex behaviors based on a massive stream of sensory information from different modalities. Early animal studies have identified learning mechanisms that are based on reward and punishment such that animals tend to avoid actions that lead to punishment whereas rewarded actions are reinforced. However, most algorithms for reward-based learning are only applicable if the dimensionality of the state-space is sufficiently small or its structure is sufficiently simple. Therefore, the question arises how the problem of learning on high-dimensional data is solved in the brain. In this article, we propose a biologically plausible generic two-stage learning system that can directly be applied to raw high-dimensional input streams. The system is composed of a hierarchical slow feature analysis (SFA) network for preprocessing and a simple neural network on top that is trained based on rewards. We demonstrate by computer simulations that this generic architecture is able to learn quite demanding reinforcement learning tasks on high-dimensional visual input streams in a time that is comparable to the time needed when an explicit highly informative low-dimensional state-space representation is given instead of the high-dimensional visual input. The learning speed of the proposed architecture in a task similar to the Morris water maze task is comparable to that found in experimental studies with rats. This study thus supports the hypothesis that slowness learning is one important unsupervised learning principle utilized in the brain to form efficient state representations for behavioral learning.
authors.affiliation: Institute for Theoretical Computer Science, Graz University of Technology, Graz, Austria. legi@igi.tugraz.at
authors.forename: Robert
authors.initials: R
authors.lastname: Legenstein
authors.forename: Niko
authors.initials: N
authors.lastname: Wilbert
authors.forename: Laurenz
authors.initials: L
authors.lastname: Wiskott
mesh_terms: D000465:Algorithms; D000818:Animals; D001921:Brain / Q000502:physiology; D003198:Computer Simulation; D006801:Humans; D007858:Learning / Q000502:physiology; D008954:Models, Biological; D016571:Neural Networks, Computer; D011597:Psychomotor Performance / Q000502:physiology*; D051381:Rats; D012054:Reinforcement, Psychology; D014796:Visual Perception / Q000502:physiology
references.citation: PLoS Comput Biol. 2008 Oct;4(10):e1000180
references.pmid: 18846203
references.citation: Proc Natl Acad Sci U S A. 1991 May 15;88(10):4433-7
references.pmid: 1903542
references.citation: J Vis. 2005 Jul 20;5(6):579-602
references.pmid: 16097870
references.citation: PLoS Biol. 2006 May;4(5):e120
references.pmid: 16605306
references.citation: Nature. 2008 Mar 27;452(7186):436-41
references.pmid: 18368112
references.citation: Science. 1997 Mar 14;275(5306):1593-9
references.pmid: 9054347
references.citation: Prog Neurobiol. 1997 Feb;51(2):167-94
references.pmid: 9247963
references.citation: Front Neuroinform. 2009 Jan 08;2:8
references.pmid: 19169361
references.citation: Neural Comput. 1991 Summer;3(2):194-200
references.pmid: 31167302
references.citation: Neural Comput. 2011 Sep;23(9):2289-323
references.pmid: 21671784
references.citation: PLoS Comput Biol. 2007 Aug;3(8):e166
references.pmid: 17784780
references.citation: Neural Comput. 2007 Aug;19(8):2245-79
references.pmid: 17571943
references.citation: Neural Comput. 1991 Fall;3(3):312-320
references.pmid: 31167313
references.citation: Neural Comput. 2006 Jun;18(6):1318-48
references.pmid: 16764506
references.citation: Nat Neurosci. 2009 Mar;12(3):250-2
references.pmid: 19219040
references.citation: J Neurophysiol. 1998 Jul;80(1):1-27
references.pmid: 9658025
references.citation: J Cogn Neurosci. 1998 Jan;10(1):108-21
references.pmid: 9526086
references.citation: Neuron. 2003 Dec 18;40(6):1063-73
references.pmid: 14687542
references.citation: Neural Comput. 2009 Apr;21(4):911-59
references.pmid: 19018708
references.citation: PLoS Comput Biol. 2007 Jun;3(6):e112
references.pmid: 17604445
references.citation: Science. 2000 Dec 22;290(5500):2319-23
references.pmid: 11125149
references.citation: Biol Cybern. 2005 Jul;93(1):79-90
references.pmid: 16021516
references.citation: J Neurophysiol. 2000 Sep;84(3):1224-39
references.pmid: 10979997
references.citation: Nature. 1988 Oct 27;335(6193):817-20
references.pmid: 3185711
references.citation: Neuron. 2003 Mar 27;37(6):989-99
references.pmid: 12670427
references.citation: Neural Comput. 2007 Jun;19(6):1468-502
references.pmid: 17444757
references.citation: Neural Comput. 2009 Feb;21(2):301-39
references.pmid: 19196231
references.citation: Nature. 1982 Jun 24;297(5868):681-3
references.pmid: 7088155
references.citation: J Neurosci. 1996 Mar 1;16(5):1936-47
references.pmid: 8774460
references.citation: Nature. 1992 Jan 9;355(6356):161-3
references.pmid: 1729650
references.citation: Neural Netw. 2002 Jun-Jul;15(4-6):507-21
references.pmid: 12371508
references.citation: J Neurophysiol. 2007 Dec;98(6):3648-65
references.pmid: 17928565
references.citation: Neural Comput. 2002 Apr;14(4):715-70
references.pmid: 11936959
references.citation: PLoS Comput Biol. 2009 Dec;5(12):e1000586
references.pmid: 19997492
references.citation: Hippocampus. 2000;10(1):1-16
references.pmid: 10706212
references.citation: Phys Rev Lett. 2006 Jul 28;97(4):048104
references.pmid: 16907616
references.citation: Science. 2008 Sep 12;321(5895):1502-7
references.pmid: 18787171
references.citation: Network. 2003 Nov;14(4):765-88
references.pmid: 14653502
references.citation: Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Apr;69(4 Pt 1):041909
references.pmid: 15169045
references.citation: Cereb Cortex. 2007 Oct;17(10):2443-52
references.pmid: 17220510
references.citation: Int J Neural Syst. 1996 Dec;7(6):671-87
references.pmid: 9113528