title: Dopaminergic Balance between Reward Maximization and Policy Complexity.
abstract: Previous reinforcement-learning models of the basal ganglia network have highlighted the role of dopamine in encoding the mismatch between prediction and reality. Far less attention has been paid to the computational goals and algorithms of the main-axis (actor). Here, we construct a top-down model of the basal ganglia with emphasis on the role of dopamine as both a reinforcement learning signal and as a pseudo-temperature signal controlling the general level of basal ganglia excitability and motor vigilance of the acting agent. We argue that the basal ganglia endow the thalamic-cortical networks with the optimal dynamic tradeoff between two constraints: minimizing the policy complexity (cost) and maximizing the expected future reward (gain). We show that this multi-dimensional optimization processes results in an experience-modulated version of the softmax behavioral policy. Thus, as in classical softmax behavioral policies, probability of actions are selected according to their estimated values and the pseudo-temperature, but in addition also vary according to the frequency of previous choices of these actions. We conclude that the computational goal of the basal ganglia is not to maximize cumulative (positive and negative) reward. Rather, the basal ganglia aim at optimization of independent gain and cost functions. Unlike previously suggested single-variable maximization processes, this multi-dimensional optimization process leads naturally to a softmax-like behavioral policy. We suggest that beyond its role in the modulation of the efficacy of the cortico-striatal synapses, dopamine directly affects striatal excitability and thus provides a pseudo-temperature signal that modulates the tradeoff between gain and cost. The resulting experience and dopamine modulated softmax policy can then serve as a theoretical framework to account for the broad range of behaviors and clinical states governed by the basal ganglia and dopamine systems.
authors: The Interdisciplinary Center for Neural Computation, The Hebrew University Jerusalem, Israel.
authors: Naama
authors: N
authors: Parush
authors: Naftali
authors: N
authors: Tishby
authors: Hagai
authors: H
authors: Bergman
references: Neuron. 2004 Jul 8;43(1):133-43
references: 15233923
references: Neurol Sci. 2010 Dec;31(6):751-6
references: 20859648
references: Brain Res. 1985 May 20;334(2):227-33
references: 3995318
references: J Neurophysiol. 2004 Oct;92(4):1973-81
references: 15115785
references: J Neurosci. 2008 Sep 24;28(39):9619-31
references: 18815248
references: Prog Neurobiol. 1996 Nov;50(4):381-425
references: 9004351
references: Brain Res. 1998 Mar 16;787(1):157-60
references: 9518590
references: Science. 2008 Aug 8;321(5890):848-51
references: 18687967
references: J Neurosci. 2009 Nov 18;29(46):14617-26
references: 19923294
references: Trends Neurosci. 1989 Oct;12(10):366-75
references: 2479133
references: J Neurophysiol. 2001 Jan;85(1):117-24
references: 11152712
references: J Neurosci. 2008 Nov 5;28(45):11603-14
references: 18987196
references: Neuron. 2004 Jan 22;41(2):269-80
references: 14741107
references: Brain Res Rev. 2008 Aug;58(2):272-81
references: 18054796
references: Brain. 2001 Mar;124(Pt 3):546-57
references: 11222455
references: Brain Res. 1991 Apr 26;547(1):152-61
references: 1677608
references: J Neurophysiol. 1999 Feb;81(2):858-74
references: 10036286
references: J Neurosci. 2009 Dec 2;29(48):15104-14
references: 19955362
references: Annu Rev Neurosci. 2000;23:185-215
references: 10845063
references: Neuron. 2008 Nov 26;60(4):543-54
references: 19038213
references: Trends Cogn Sci. 2006 Aug;10(8):375-81
references: 16843041
references: Trends Neurosci. 2007 May;30(5):228-35
references: 17408758
references: J Neurophysiol. 1983 May;49(5):1285-301
references: 6306173
references: Prog Neurobiol. 2003 Dec;71(6):439-73
references: 15013228
references: Trends Neurosci. 2010 Jun;33(6):267-76
references: 20207024
references: Neural Netw. 2002 Jun-Jul;15(4-6):507-21
references: 12371508
references: Nature. 2001 Sep 6;413(6851):67-70
references: 11544526
references: Neural Comput. 2007 Sep;19(9):2387-432
references: 17650064
references: J Neurosci. 2008 Mar 5;28(10):2435-46
references: 18322089
references: Adv Pharmacol. 1998;42:686-90
references: 9327992
references: Prog Brain Res. 2010;183:275-97
references: 20696325
references: J Neurosci. 2003 Oct 29;23(30):9913-23
references: 14586021
references: Neuropharmacology. 2007 Oct;53(5):583-7
references: 17709119
references: J Neurophysiol. 2001 Jul;86(1):249-60
references: 11431506
references: Trends Neurosci. 2003 Aug;26(8):423-8
references: 12900173
references: Behav Brain Res. 2009 Apr 12;199(1):141-56
references: 18950662
references: Prog Brain Res. 2007;160:209-26
references: 17499116
references: Trends Neurosci. 2000 Oct;23(10 Suppl):S48-56
references: 11052220
references: J Neurosci. 2003 May 15;23(10):4012-6
references: 12764086
references: J Neurosci. 2002 Sep 15;22(18):7850-5
references: 12223537
references: Science. 2003 Mar 21;299(5614):1898-902
references: 12649484
references: Mov Disord. 2007 Dec;22(16):2394-7
references: 17914720
references: Mov Disord. 1999 Jan;14(1):45-9
references: 9918343
references: Trends Neurosci. 2004 Aug;27(8):453-9
references: 15271492
references: J Neural Transm (Vienna). 1997;104(8-9):895-904
references: 9451721
references: Neuron. 2002 Oct 10;36(2):285-98
references: 12383782
references: J Pharmacol Exp Ther. 1998 Jul;286(1):272-81
references: 9655869
references: Science. 1997 Mar 14;275(5306):1593-9
references: 9054347
references: Ann Neurol. 1999 Nov;46(5):732-8
references: 10553990
references: Neuron. 2005 Jul 7;47(1):129-41
references: 15996553
references: Nat Neurosci. 2006 Aug;9(8):1057-63
references: 16862149
references: J Exp Anal Behav. 2005 Nov;84(3):555-79
references: 16596980
references: Arch Neurol. 1989 Oct;46(10):1070-5
references: 2508609
references: J Neurosci. 2007 Aug 1;27(31):8161-5
references: 17670959
references: J Neurophysiol. 2008 Dec;100(6):3244-52
references: 18842956