Reinforcement learning on slow features of high-dimensional input streams.
Humans and animals are able to learn complex behaviors based on a massive stream of sensory information from different modalities. Early animal studies have identified learning mechanisms that are based on reward and punishment such that animals tend to avoid actions that lead to punishment whereas rewarded actions are reinforced. However, most algorithms for reward-based learning are only applicable if the dimensionality of the state-space is sufficiently small or its structure is sufficiently simple. Therefore, the question arises how the problem of learning on high-dimensional data is solved in the brain. In this article, we propose a biologically plausible generic two-stage learning system that can directly be applied to raw high-dimensional input streams. The system is composed of a hierarchical slow feature analysis (SFA) network for preprocessing and a simple neural network on top that is trained based on rewards. We demonstrate by computer simulations that this generic architecture is able to learn quite demanding reinforcement learning tasks on high-dimensional visual input streams in a time that is comparable to the time needed when an explicit highly informative low-dimensional state-space representation is given instead of the high-dimensional visual input. The learning speed of the proposed architecture in a task similar to the Morris water maze task is comparable to that found in experimental studies with rats. This study thus supports the hypothesis that slowness learning is one important unsupervised learning principle utilized in the brain to form efficient state representations for behavioral learning.
Legenstein
Robert
R
Institute for Theoretical Computer Science, Graz University of Technology, Graz, Austria. legi@igi.tugraz.at
Wilbert
Niko
N
Wiskott
Laurenz
L
D000465:Algorithms; D000818:Animals; D001921:Brain / Q000502:physiology; D003198:Computer Simulation; D006801:Humans; D007858:Learning / Q000502:physiology; D008954:Models, Biological; D016571:Neural Networks, Computer; D011597:Psychomotor Performance / Q000502:physiology*; D051381:Rats; D012054:Reinforcement, Psychology; D014796:Visual Perception / Q000502:physiology
PLoS Comput Biol. 2008 Oct;4(10):e1000180
18846203
Proc Natl Acad Sci U S A. 1991 May 15;88(10):4433-7
1903542
J Vis. 2005 Jul 20;5(6):579-602
16097870
PLoS Biol. 2006 May;4(5):e120
16605306
Nature. 2008 Mar 27;452(7186):436-41
18368112
Science. 1997 Mar 14;275(5306):1593-9
9054347
Prog Neurobiol. 1997 Feb;51(2):167-94
9247963
Front Neuroinform. 2009 Jan 08;2:8
19169361
Neural Comput. 1991 Summer;3(2):194-200
31167302
Neural Comput. 2011 Sep;23(9):2289-323
21671784
PLoS Comput Biol. 2007 Aug;3(8):e166
17784780
Neural Comput. 2007 Aug;19(8):2245-79
17571943
Neural Comput. 1991 Fall;3(3):312-320
31167313
Neural Comput. 2006 Jun;18(6):1318-48
16764506
Nat Neurosci. 2009 Mar;12(3):250-2
19219040
J Neurophysiol. 1998 Jul;80(1):1-27
9658025
J Cogn Neurosci. 1998 Jan;10(1):108-21
9526086
Neuron. 2003 Dec 18;40(6):1063-73
14687542
Neural Comput. 2009 Apr;21(4):911-59
19018708
PLoS Comput Biol. 2007 Jun;3(6):e112
17604445
Science. 2000 Dec 22;290(5500):2319-23
11125149
Biol Cybern. 2005 Jul;93(1):79-90
16021516
J Neurophysiol. 2000 Sep;84(3):1224-39
10979997
Nature. 1988 Oct 27;335(6193):817-20
3185711
Neuron. 2003 Mar 27;37(6):989-99
12670427
Neural Comput. 2007 Jun;19(6):1468-502
17444757
Neural Comput. 2009 Feb;21(2):301-39
19196231
Nature. 1982 Jun 24;297(5868):681-3
7088155
J Neurosci. 1996 Mar 1;16(5):1936-47
8774460
Nature. 1992 Jan 9;355(6356):161-3
1729650
Neural Netw. 2002 Jun-Jul;15(4-6):507-21
12371508
J Neurophysiol. 2007 Dec;98(6):3648-65
17928565
Neural Comput. 2002 Apr;14(4):715-70
11936959
PLoS Comput Biol. 2009 Dec;5(12):e1000586
19997492
Hippocampus. 2000;10(1):1-16
10706212
Phys Rev Lett. 2006 Jul 28;97(4):048104
16907616
Science. 2008 Sep 12;321(5895):1502-7
18787171
Network. 2003 Nov;14(4):765-88
14653502
Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Apr;69(4 Pt 1):041909
15169045
Cereb Cortex. 2007 Oct;17(10):2443-52
17220510
Int J Neural Syst. 1996 Dec;7(6):671-87
9113528