title: Variability in Dopamine Genes Dissociates Model-Based and Model-Free Reinforcement Learning.
abstract: Considerable evidence suggests that multiple learning systems can drive behavior. Choice can proceed reflexively from previous actions and their associated outcomes, as captured by "model-free" learning algorithms, or flexibly from prospective consideration of outcomes that might occur, as captured by "model-based" learning algorithms. However, differential contributions of dopamine to these systems are poorly understood. Dopamine is widely thought to support model-free learning by modulating plasticity in striatum. Model-based learning may also be affected by these striatal effects, or by other dopaminergic effects elsewhere, notably on prefrontal working memory function. Indeed, prominent demonstrations linking striatal dopamine to putatively model-free learning did not rule out model-based effects, whereas other studies have reported dopaminergic modulation of verifiably model-based learning, but without distinguishing a prefrontal versus striatal locus. To clarify the relationships between dopamine, neural systems, and learning strategies, we combine a genetic association approach in humans with two well-studied reinforcement learning tasks: one isolating model-based from model-free behavior and the other sensitive to key aspects of striatal plasticity. Prefrontal function was indexed by a polymorphism in the COMT gene, differences of which reflect dopamine levels in the prefrontal cortex. This polymorphism has been associated with differences in prefrontal activity and working memory. Striatal function was indexed by a gene coding for DARPP-32, which is densely expressed in the striatum where it is necessary for synaptic plasticity. We found evidence for our hypothesis that variations in prefrontal dopamine relate to model-based learning, whereas variations in striatal dopamine function relate to model-free learning.
  
  
  CONCLUSIONS
  Decisions can stem reflexively from their previously associated outcomes or flexibly from deliberative consideration of potential choice outcomes. Research implicates a dopamine-dependent striatal learning mechanism in the former type of choice. Although recent work has indicated that dopamine is also involved in flexible, goal-directed decision-making, it remains unclear whether it also contributes via striatum or via the dopamine-dependent working memory function of prefrontal cortex. We examined genetic indices of dopamine function in these regions and their relation to the two choice strategies. We found that striatal dopamine function related most clearly to the reflexive strategy, as previously shown, and that prefrontal dopamine related most clearly to the flexible strategy. These findings suggest that dissociable brain regions support dissociable choice strategies.
authors.affiliation: Center for Neural Science, New York University, New York, New York 10003, Department of Psychology, Columbia University, New York, New York 10027, bradley.doll@nyu.edu.
authors.forename: Bradley B
authors.initials: BB
authors.lastname: Doll
authors.affiliation: Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, Rhode Island 02912.
authors.forename: Kevin G
authors.initials: KG
authors.lastname: Bath
authors.affiliation: Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey 08540, Department of Psychology, Princeton University, Princeton, New Jersey 08540, and.
authors.forename: Nathaniel D
authors.initials: ND
authors.lastname: Daw
authors.affiliation: Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, Rhode Island 02912, Brown Institute for Brain Science, Brown University, Providence, Rhode Island 02912.
authors.forename: Michael J
authors.identifier: http://orcid.org/0000-0001-8451-0523
authors.initials: MJ
authors.lastname: Frank
mesh_terms: D000293:Adolescent; D000328:Adult; D001921:Brain / Q000502:physiology*; D001931:Brain Mapping; D002394:Catechol O-Methyltransferase / Q000235:genetics*; D003657:Decision Making / Q000502:physiology; D004298:Dopamine / Q000378:metabolism; D051937:Dopamine and cAMP-Regulated Phosphoprotein 32 / Q000235:genetics*; D005260:Female; D006801:Humans; D007858:Learning / Q000502:physiology*; D016015:Logistic Models; D008297:Male; D008960:Models, Psychological; D020641:Polymorphism, Single Nucleotide / Q000235:genetics*; D012054:Reinforcement, Psychology; D014163:Transfer, Psychology; D055815:Young Adult
chemical_list: D051937:Dopamine and cAMP-Regulated Phosphoprotein 32; C497928:PPP1R1B protein, human; D002394:Catechol O-Methyltransferase; D004298:Dopamine
keywords: decision-making; dopamine; genetics; reinforcement learning
references.citation: Pharmacol Rev. 1999 Dec;51(4):593-628
references.pmid: 10581325
references.citation: Eur J Neurosci. 2012 Apr;35(7):1024-35
references.pmid: 22487033
references.citation: Proc Natl Acad Sci U S A. 2011 Sep 13;108 Suppl 3:15647-54
references.pmid: 21389268
references.citation: Mol Psychiatry. 2010 Aug;15(8):810-5
references.pmid: 19255578
references.citation: Nat Neurosci. 2008 Aug;11(8):932-9
references.pmid: 18622401
references.citation: Neuroimage. 2005 Jan 1;24(1):244-52
references.pmid: 15588616
references.citation: Nat Neurosci. 2013 Jul;16(7):966-73
references.pmid: 23708143
references.citation: Brain Res Mol Brain Res. 1990 Feb;7(2):139-49
references.pmid: 2160041
references.citation: J Neurosci. 2011 Apr 6;31(14):5526-39
references.pmid: 21471389
references.citation: Neuron. 2013 Oct 16;80(2):312-25
references.pmid: 24139036
references.citation: J Exp Anal Behav. 2005 Nov;84(3):555-79
references.pmid: 16596980
references.citation: Eur J Neurosci. 2002 Jan;15(2):246-56
references.pmid: 11849292
references.citation: Eur J Neurosci. 2005 Jul;22(2):513-23
references.pmid: 16045504
references.citation: J Neurosci. 2011 Feb 2;31(5):1606-13
references.pmid: 21289169
references.citation: J Clin Invest. 2007 Mar;117(3):672-82
references.pmid: 17290303
references.citation: Curr Opin Neurobiol. 2012 Dec;22(6):1075-81
references.pmid: 22959354
references.citation: Psychol Rev. 2014 Jul;121(3):337-66
references.pmid: 25090423
references.citation: J Neurosci. 2011 Apr 20;31(16):6188-98
references.pmid: 21508242
references.citation: J Neural Transm Park Dis Dement Sect. 1990;2(2):101-12
references.pmid: 1977408
references.citation: J Child Adolesc Psychopharmacol. 2014 Oct;24(8):454-7
references.pmid: 25329975
references.citation: J Neurosci. 2004 Jun 9;24(23):5331-5
references.pmid: 15190105
references.citation: Nat Neurosci. 2015 May;18(5):767-72
references.pmid: 25799041
references.citation: J Neurophysiol. 1990 Jun;63(6):1385-400
references.pmid: 2358882
references.citation: Proc Natl Acad Sci U S A. 2007 Oct 9;104(41):16311-6
references.pmid: 17913879
references.citation: J Neurochem. 1992 Jul;59(1):326-30
references.pmid: 1613509
references.citation: Nature. 2008 Jun 12;453(7197):879-84
references.pmid: 18496528
references.citation: J Mem Lang. 2013 Apr;68(3):
references.pmid: 24403724
references.citation: J Comp Neurol. 1992 Sep 8;323(2):209-18
references.pmid: 1328330
references.citation: Nat Neurosci. 2012 Jun;15(6):816-8
references.pmid: 22544310
references.citation: Neuroimage. 2009 Jul 15;46(4):1004-17
references.pmid: 19306932
references.citation: Pharmacogenetics. 1996 Jun;6(3):243-50
references.pmid: 8807664
references.citation: Neuron. 2012 Aug 9;75(3):418-24
references.pmid: 22884326
references.citation: Proc Natl Acad Sci U S A. 2009 Oct 20;106(42):17951-6
references.pmid: 19822738
references.citation: Biol Psychiatry. 2008 Nov 1;64(9):739-49
references.pmid: 18620336
references.citation: Neuroimage. 2015 Apr 1;109:95-101
references.pmid: 25562824
references.citation: Psychol Rev. 1948 Jul;55(4):189-208
references.pmid: 18870876
references.citation: J Comp Neurol. 1990 Sep 15;299(3):327-48
references.pmid: 2229482
references.citation: Brain. 2016 Feb;139(Pt 2):355-64
references.pmid: 26685155
references.citation: J Cogn Neurosci. 2005 Jan;17(1):51-72
references.pmid: 15701239
references.citation: Psychol Sci. 2013 May;24(5):751-61
references.pmid: 23558545
references.citation: PLoS Comput Biol. 2006 Sep 8;2(9):e119
references.pmid: 16965177
references.citation: Lancet. 2003 Feb 15;361(9357):598-604
references.pmid: 12598158
references.citation: J Neurosci. 2000 Nov 15;20(22):8443-51
references.pmid: 11069952
references.citation: Science. 1991 Feb 22;251(4996):947-50
references.pmid: 1825731
references.citation: J Cogn Neurosci. 2010 Jul;22(7):1614-22
references.pmid: 19642882
references.citation: Proc Natl Acad Sci U S A. 2015 Feb 3;112(5):1595-600
references.pmid: 25605941
references.citation: Neuron. 2013 Nov 20;80(4):914-9
references.pmid: 24206669
references.citation: Eur J Pharmacol. 1998 Sep 4;356(2-3):127-37
references.pmid: 9774242
references.citation: Psychopharmacology (Berl). 2009 Jan;202(1-3):521-30
references.pmid: 18854987
references.citation: Neuroscience. 2003;116(1):127-37
references.pmid: 12535946
references.citation: Proc Natl Acad Sci U S A. 1998 Aug 18;95(17):9991-6
references.pmid: 9707588
references.citation: Pharmacogenetics. 1998 Jun;8(3):195-9
references.pmid: 9682265
references.citation: Neuron. 2010 May 27;66(4):585-95
references.pmid: 20510862
references.citation: Mol Psychiatry. 2008 Aug;13(8):821-7
references.pmid: 18317466
references.citation: Mol Psychiatry. 2015 Mar;20(3):345-52
references.pmid: 24840709
references.citation: Proc Natl Acad Sci U S A. 2001 Jun 5;98(12):6917-22
references.pmid: 11381111
references.citation: Proc Natl Acad Sci U S A. 2013 Dec 24;110(52):20941-6
references.pmid: 24324166
references.citation: Behav Neurosci. 2006 Jun;120(3):497-517
references.pmid: 16768602
references.citation: Neuron. 2011 Mar 24;69(6):1204-15
references.pmid: 21435563
references.citation: Neurosci Res. 1994 Nov;21(1):1-11
references.pmid: 7708289
references.citation: Nat Neurosci. 2009 Aug;12(8):1062-8
references.pmid: 19620978
references.citation: Neurobiol Learn Mem. 2015 Jan;117:4-13
references.pmid: 24846190
references.citation: Nat Neurosci. 2005 Dec;8(12):1704-11
references.pmid: 16286932
references.citation: Annu Rev Pharmacol Toxicol. 2004;44:269-96
references.pmid: 14744247
references.citation: Science. 2004 Dec 10;306(5703):1940-3
references.pmid: 15528409
references.citation: J Neurosci. 2009 Aug 5;29(31):9861-74
references.pmid: 19657038
references.citation: Neuron. 2014 Aug 6;83(3):551-7
references.pmid: 25066083
references.citation: Cogn Affect Behav Neurosci. 2015 Sep;15(3):523-36
references.pmid: 25801925
references.citation: Proc Natl Acad Sci U S A. 2005 Jan 11;102(2):491-6
references.pmid: 15608059
references.citation: J Neurosci. 1998 Apr 1;18(7):2697-708
references.pmid: 9502827