title: An imperfect dopaminergic error signal can drive temporal-difference learning.
abstract: An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.
authors: Institute of Neuroscience and Medicine (INM-6), Computational and Systems Neuroscience, Research Center Jülich, Jülich, Germany. w.potjans@fz-juelich.de
authors: Wiebke
authors: W
authors: Potjans
authors: Markus
authors: M
authors: Diesmann
authors: Abigail
authors: A
authors: Morrison
references: J Comput Neurosci. 1999 May-Jun;6(3):191-214
references: 10406133
references: J Neurosci. 2003 Oct 15;23(28):9395-402
references: 14561867
references: J Comput Neurosci. 2011 Jun;30(3):675-97
references: 20953686
references: Network. 2009;20(1):32-46
references: 19229732
references: J Neurophysiol. 1992 Jan;67(1):145-63
references: 1552316
references: Nat Neurosci. 2004 Dec;7(12):1345-52
references: 15558066
references: PLoS Comput Biol. 2008 Oct;4(10):e1000180
references: 18846203
references: Neuron. 2007 Apr 19;54(2):319-33
references: 17442251
references: Neuroscience. 1999;91(3):871-90
references: 10391468
references: J Neurosci. 2006 Dec 13;26(50):12921-42
references: 17167083
references: Acta Neurobiol Exp (Wars). 2007;67(4):481-8
references: 18320725
references: Neural Comput. 2007 Aug;19(8):2245-79
references: 17571943
references: Neuron. 2004 Jul 8;43(1):133-43
references: 15233923
references: Nature. 1995 Oct 26;377(6551):725-8
references: 7477260
references: J Neurosci. 1999 Dec 1;19(23):10502-11
references: 10575046
references: Neuroscience. 1999 Mar;89(3):799-812
references: 10199614
references: J Neurophysiol. 1998 Jul;80(1):1-27
references: 9658025
references: Science. 2004 Apr 16;304(5669):452-4
references: 15087550
references: Neurosci Lett. 1992 Jan 6;134(2):207-11
references: 1350333
references: J Cogn Neurosci. 1998 Jan;10(1):108-21
references: 9526086
references: Neuron. 2003 Dec 18;40(6):1063-73
references: 14687542
references: Nature. 2006 Aug 31;442(7106):1042-5
references: 16929307
references: Nature. 2002 Mar 28;416(6879):433-8
references: 11919633
references: Neural Comput. 2005 Feb;17(2):245-319
references: 15720770
references: Hippocampus. 1999;9(2):118-36
references: 10226773
references: J Neurosci. 1995 Apr;15(4):3092-103
references: 7722648
references: J Neurosci. 2007 May 2;27(18):4826-31
references: 17475790
references: Neural Comput. 2007 Jun;19(6):1468-502
references: 17444757
references: Science. 1996 Sep 6;273(5280):1399-402
references: 8703077
references: Neural Comput. 2009 Feb;21(2):301-39
references: 19196231
references: Biol Cybern. 2001 Jun;84(6):401-10
references: 11417052
references: J Neurosci. 1996 Mar 1;16(5):1936-47
references: 8774460
references: J Neurophysiol. 2007 Sep;98(3):1428-39
references: 17615124
references: Cereb Cortex. 1995 Mar-Apr;5(2):95-110
references: 7620294
references: Trends Neurosci. 2005 Jan;28(1):1-4
references: 15626490
references: Front Synaptic Neurosci. 2010 Oct 25;2:146
references: 21423532
references: Nature. 2007 Jun 28;447(7148):1111-5
references: 17522629
references: J Neurosci. 2010 Oct 6;30(40):13326-37
references: 20926659
references: PLoS Comput Biol. 2010 Feb 12;6(2):e1000670
references: 20169176
references: Neuroscience. 2000;96(4):651-6
references: 10727783
references: Nature. 2001 Sep 6;413(6851):67-70
references: 11544526
references: Science. 2005 Mar 11;307(5715):1642-5
references: 15761155
references: Neural Netw. 2002 Jun-Jul;15(4-6):507-21
references: 12371508
references: PLoS Comput Biol. 2008 Aug 08;4(8):e1000131
references: 18688266
references: J Neurosci. 2008 Mar 5;28(10):2435-46
references: 18322089
references: Front Comput Neurosci. 2011 Jan 05;4:154
references: 21258641
references: Curr Opin Neurobiol. 2000 Dec;10(6):732-9
references: 11240282
references: J Neurosci. 2001 May 1;21(9):3261-70
references: 11312311
references: Neuroscience. 2002;114(2):475-92
references: 12204216
references: J Neurophysiol. 2007 Dec;98(6):3648-65
references: 17928565
references: J Physiol. 1994 Sep 15;479 ( Pt 3):401-22
references: 7837097
references: PLoS Comput Biol. 2009 Dec;5(12):e1000586
references: 19997492
references: Nat Rev Neurosci. 2006 Dec;7(12):967-75
references: 17115078
references: PLoS One. 2009;4(5):e5464
references: 19421325
references: Neural Comput. 2007 Oct;19(10):2694-719
references: 17716008
references: Proc Natl Acad Sci U S A. 2006 Oct 10;103(41):15224-9
references: 17008410
references: Trends Neurosci. 1999 Apr;22(4):146-51
references: 10203849
references: Phys Rev E Stat Nonlin Soft Matter Phys. 2009 Nov;80(5 Pt 1):051902
references: 20365001
references: J Neurosci. 1994 Oct;14(10):6084-93
references: 7931564
references: Behav Neurosci. 1993 Feb;107(1):3-22
references: 8447956
references: PLoS Comput Biol. 2010 Sep 09;6(9):
references: 20856583
references: J Neurosci. 2004 Feb 18;24(7):1754-9
references: 14973252
references: Proc Natl Acad Sci U S A. 1982 Apr;79(8):2554-8
references: 6953413
references: Hippocampus. 2000;10(1):1-16
references: 10706212
references: Neural Netw. 2002 Jun-Jul;15(4-6):535-47
references: 12371510
references: Nat Neurosci. 2007 Jan;10(1):126-31
references: 17187065
references: Neuroscience. 2001;103(1):65-85
references: 11311788
references: Neural Netw. 2006 Oct;19(8):1075-90
references: 17015181
references: Eur J Neurosci. 1995 Sep 1;7(9):1889-94
references: 8528463
references: Science. 2003 Mar 21;299(5614):1898-902
references: 12649484
references: Neuroscience. 1987 Feb;20(2):365-83
references: 3035421
references: Nature. 2009 Jun 11;459(7248):837-41
references: 19448610
references: Neuroscience. 1994 Mar;59(2):229-43
references: 8008189
references: Nat Neurosci. 2010 Jan;13(1):112-9
references: 20010823
references: Neural Comput. 2001 Apr;13(4):841-62
references: 11255572
references: Neuroscience. 1998 Aug;85(4):1089-99
references: 9681948
references: Curr Opin Neurol. 2002 Aug;15(4):457-60
references: 12151843
references: J Neurosci. 2008 Oct 8;28(41):10386-98
references: 18842898
references: J Neurosci. 2006 Sep 13;26(37):9530-7
references: 16971537
references: Neuron. 2002 Oct 10;36(2):241-63
references: 12383780
references: Science. 1997 Mar 14;275(5306):1593-9
references: 9054347
references: Nat Neurosci. 2006 Aug;9(8):1057-63
references: 16862149
references: Neural Comput. 2008 Dec;20(12):3034-54
references: 18624657
references: Front Comput Neurosci. 2010 Nov 23;4:141
references: 21151370
references: Neural Netw. 2003 Jan;16(1):5-9
references: 12576101
references: Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Apr;69(4 Pt 1):041909
references: 15169045
references: Nature. 2004 Jun 10;429(6992):664-7
references: 15190354
references: Cereb Cortex. 2007 Oct;17(10):2443-52
references: 17220510
references: Biol Cybern. 2005 Jun;92(6):427-37
references: 15915357
references: J Neurosci. 2002 Aug 15;22(16):7308-20
references: 12177226
references: Neural Netw. 2002 Jun-Jul;15(4-6):603-16
references: 12371515
references: PLoS Comput Biol. 2009 Aug;5(8):e1000456
references: 19662159
references: Neural Comput. 2001 Oct;13(10):2221-37
references: 11570997