Learning, Reward, and Decision Making.
In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems. We consider the nature of the interactions between these systems and show how these interactions can lead to either adaptive or maladaptive behavioral outcomes. We then review evidence that an additional system guides inference concerning the hidden states of other agents, such as their beliefs, preferences, and intentions, in a social context. We also describe emerging evidence for an arbitration mechanism between model-based and model-free reinforcement learning, placing such a mechanism within the broader context of the hierarchical control of behavior.
Division of Humanities and Social Sciences and Computation and Neural Systems Program, California Institute of Technology, Pasadena, California 91125; email: jdoherty@caltech.edu.
John P
JP
O'Doherty
Division of Humanities and Social Sciences and Computation and Neural Systems Program, California Institute of Technology, Pasadena, California 91125; email: jdoherty@caltech.edu.
Jeffrey
J
Cockburn
Division of Humanities and Social Sciences and Computation and Neural Systems Program, California Institute of Technology, Pasadena, California 91125; email: jdoherty@caltech.edu.
Wolfgang M
WM
Pauli
D001921:Brain / Q000502:physiology; D003213:Conditioning, Psychological / Q000502:physiology; D003657:Decision Making / Q000502:physiology*; D006040:Goals; D006801:Humans; D007858:Learning / Q000502:physiology*; D009488:Neurosciences; D012054:Reinforcement, Psychology; D012201:Reward
Pavlovian; cognitive map; instrumental; model based; model free; outcome valuation
J Neurosci. 2010 Oct 20;30(42):14080-90
20962229
J Exp Anal Behav. 1973 Sep;20(2):163-81
4752087
Appetite. 2014 Aug;79:139-48
24743030
Science. 2003 Aug 22;301(5636):1104-7
12934011
Psychol Sci. 2009 Aug;20(8):955-62
19594859
Neuron. 2004 Oct 14;44(2):365-78
15473973
PLoS Comput Biol. 2013;9(2):e1002918
23436990
Neuron. 2014 Jan 22;81(2):267-279
24462094
Neuron. 2015 Jul 1;87(1):231-43
26139376
Neuron. 2014 Feb 5;81(3):687-99
24507199
Eur J Neurosci. 2004 Jan;19(1):181-9
14750976
J Neurosci. 2012 Jul 18;32(29):9878-86
22815503
Nat Neurosci. 2013 Apr;16(4):479-85
23416449
Curr Opin Neurobiol. 2004 Dec;14(6):769-76
15582382
Science. 2008 Feb 29;319(5867):1264-7
18309087
Neuroimage. 2009 Feb 1;44(3):1125-32
19010423
Eur J Neurosci. 2005 Jul;22(2):505-12
16045503
Trends Cogn Sci. 2012 Jan;16(1):72-80
22177032
Annu Rev Psychol. 2016;67:23-50
26253543
Neuron. 2014 Jun 18;82(6):1357-66
24881835
J Neurosci. 2011 Apr 20;31(16):6188-98
21508242
Neuron. 2013 Jul 24;79(2):217-40
23889930
J Neurosci. 2002 Dec 15;22(24):10829-37
12486176
J Cogn Neurosci. 2007 Dec;19(12):2082-99
17892391
J Neurosci. 2007 Sep 12;27(37):9984-8
17855612
Eur J Neurosci. 2003 Oct;18(7):2069-81
14622240
Neuron. 2008 May 8;58(3):451-63
18466754
Proc Natl Acad Sci U S A. 2008 Jan 22;105(3):1050-4
18195362
Nat Neurosci. 2015 May;18(5):767-72
25799041
J Neurosci. 2000 Jun 1;20(11):4233-9
10818159
Nature. 2006 Jun 15;441(7095):876-9
16778890
J Neurosci. 2013 Mar 27;33(13):5797-805
23536092
Neuron. 2005 Feb 3;45(3):459-67
15694331
Neuron. 2005 May 19;46(4):671-9
15944134
Neuron. 2007 Apr 5;54(1):51-8
17408577
Science. 2012 Oct 12;338(6104):270-3
23066083
PLoS Comput Biol. 2013;9(12):e1003364
24339762
J Neurosci. 1988 Jul;8(7):2517-29
2854842
Curr Opin Neurobiol. 2012 Dec;22(6):1027-38
22766486
Front Behav Neurosci. 2015 Nov 03;9:289
26578917
Front Psychol. 2013 Mar 04;4:92
23459512
J Neurosci. 2014 Nov 5;34(45):14833-44
25378151
J Exp Psychol Anim Behav Process. 1983 Jul;9(3):225-47
6153052
Nature. 2013 May 2;497(7447):74-9
23594744
J Neurosci. 1996 Mar 1;16(5):1936-47
8774460
J Neurophysiol. 2001 Oct;86(4):1916-36
11600651
Proc Natl Acad Sci U S A. 2015 Apr 21;112(16):5195-200
25848032
Philos Trans R Soc Lond B Biol Sci. 2003 Mar 29;358(1431):459-73
12689373
Nature. 2008 Nov 13;456(7219):245-9
19005555
Eur J Neurosci. 2009 Apr;29(8):1678-91
19385990
Brain Res. 2013 May 20;1511:46-64
23031636
J Neurosci. 2011 Feb 16;31(7):2700-5
21325538
Neurosci Biobehav Rev. 2014 Jun;43:259-68
24726573
Annu Rev Psychol. 2011;62:451-82
21126183
Neuron. 2002 Feb 28;33(5):815-26
11879657
Behav Neurosci. 2002 Feb;116(1):105-15
11898801
Nature. 2006 Sep 7;443(7107):85-8
16936716
Science. 2005 Nov 25;310(5752):1337-40
16311337
Neuron. 2013 Jul 10;79(1):191-201
23849203
Crit Rev Neurobiol. 1997;11(4):241-61
9336713
Neuroimage. 2015 Jan 15;105:145-55
25449744
J Neurosci. 2014 Feb 26;34(9):3279-90
24573287
Nat Neurosci. 2012 Mar 11;15(5):786-91
22406551
J Neurosci. 2012 Aug 29;32(35):12066-75
22933790
Nat Neurosci. 1999 Jun;2(6):563-7
10448222
Neuron. 2011 Jul 28;71(2):370-9
21791294
J Comp Physiol A. 1985 Sep;157(2):263-77
3939242
Psychol Rev. 1948 Jul;55(4):189-208
18870876
Nature. 2006 Feb 16;439(7078):865-70
16482160
J Neurosci. 2014 Jan 22;34(4):1380-96
24453328
J Neurosci. 2011 Feb 16;31(7):2474-80
21325514
Proc Natl Acad Sci U S A. 2015 Nov 10;112(45):13817-22
26460050
Neuron. 2006 Jun 1;50(5):781-9
16731515
Neuron. 2013 Dec 18;80(6):1558-71
24360551
Proc Natl Acad Sci U S A. 2011 Nov 1;108(44):18120-5
22006321
Science. 2014 Jun 27;344(6191):1481-6
24876345
Eur J Neurosci. 2009 Jun;29(11):2225-32
19490086
J Neurosci. 2007 Dec 12;27(50):13655-66
18077677
Neuropsychologia. 2003;41(2):147-55
12459213
Nat Rev Neurosci. 2008 Jul;9(7):545-56
18545266
Neuropsychopharmacology. 2010 Jan;35(1):48-69
19776734
Hum Brain Mapp. 2015 Feb;36(2):793-803
25393839
Brain Res. 1982 Apr 29;238(2):457-62
7093668
J Neurosci. 2013 Feb 6;33(6):2242-53
23392656
Brain Res. 1971 Nov;34(1):171-5
5124915
Nat Neurosci. 2016 Mar;19(3):404-13
26906507
Behav Brain Res. 2006 Jan 30;166(2):189-96
16153716
Science. 2012 Nov 16;338(6109):953-6
23162000
Science. 2003 Jul 11;301(5630):229-32
12855813
Neuron. 2006 May 18;50(4):531-4
16701204
J Exp Psychol Anim Behav Process. 1980 Jul;6(3):207-16
6153051
Learn Mem. 2014 Jul 15;21(8):368-79
25028395
Cereb Cortex. 2007 Sep;17 Suppl 1:i110-7
17548802
J Exp Psychol Anim Behav Process. 2010 Jul;36(3):402-8
20658871
Nat Neurosci. 2011 Feb;14(2):154-62
21270784
J Neurosci. 2009 Sep 30;29(39):12315-20
19793990
Neuron. 2011 May 26;70(4):731-41
21609828
Ann N Y Acad Sci. 2007 Dec;1121:336-54
17872400
Psychol Rev. 1967 May;74(3):151-82
5342881
Cereb Cortex. 1994 Nov-Dec;4(6):664-80
7703691
Neuron. 2012 May 10;74(3):582-94
22578508
Nat Rev Neurosci. 2002 Jul;3(7):553-62
12094211
Nat Neurosci. 2012 Sep;15(9):1290-7
22885851
Neuron. 2010 May 27;66(4):585-95
20510862
J Cogn Neurosci. 2012 Jan;24(1):106-18
21812568
Neuropharmacology. 1998 Apr-May;37(4-5):407-19
9704982
Curr Opin Neurobiol. 2012 Dec;22(6):956-62
22695048
Mol Psychiatry. 2015 Mar;20(3):345-52
24840709
Science. 1981 Jan 30;211(4481):504-6
7192881
Eur J Neurosci. 2003 Aug;18(3):695-703
12911766
Annu Rev Neurosci. 1997;20:303-30
9056716
J Neurosci. 2005 Mar 16;25(11):2771-80
15772337
Ann Neurosci. 2010 Jul;17(3):136-41
25205891
J Neurophysiol. 1989 Apr;61(4):780-98
2723720
J Neurosci. 2009 Sep 9;29(36):11330-8
19741139
Nat Neurosci. 2013 Apr;16(4):486-92
23416451
J Neurosci. 2007 Apr 11;27(15):4019-26
17428979
Annu Rev Neurosci. 2001;24:167-202
11283309
Neuron. 2006 Mar 2;49(5):765-75
16504951
Front Hum Neurosci. 2014 Jun 26;8:457
25018724
J Neurosci. 2010 Aug 11;30(32):10799-808
20702709
J Neurosci. 2010 Feb 17;30(7):2490-5
20164333
Neuron. 2012 Feb 9;73(3):595-607
22325209
J Neurosci. 2007 Nov 21;27(47):12860-7
18032658
Neuron. 2003 Apr 24;38(2):329-37
12718865
Nat Neurosci. 2005 Dec;8(12):1704-11
16286932
Cognition. 2009 Dec;113(3):262-280
18926527
Cogn Affect Behav Neurosci. 2014 Jun;14(2):473-92
24647659
Proc Natl Acad Sci U S A. 2010 Aug 10;107(32):14431-6
20660717
Science. 2003 Nov 14;302(5648):1181-5
14615530
Neuroimage. 2009 Aug 15;47(2):713-21
19442745
Neuron. 2015 Apr 22;86(2):591-602
25864634
Proc Natl Acad Sci U S A. 2008 May 6;105(18):6741-6
18427116
Nat Neurosci. 2001 Jan;4(1):95-102
11135651
Curr Opin Neurobiol. 2010 Apr;20(2):262-70
20338744
Science. 2004 Dec 10;306(5703):1940-3
15528409
Nature. 2006 May 11;441(7090):223-6
16633341
Neuron. 1999 Jun;23(2):209-26
10399928
Neuron. 2008 Sep 11;59(5):829-38
18786365
Eur J Neurosci. 2003 Oct;18(7):2059-68
14622239
J Neurophysiol. 2003 Sep;90(3):1865-76
12773496
Brain. 2001 Sep;124(Pt 9):1720-33
11522575
J Neurosci. 2011 Apr 6;31(14):5526-39
21471389
Eur J Neurosci. 2003 Apr;17(8):1680-94
12752386
J Neurosci. 2014 Apr 16;34(16):5431-46
24741034
J Neurophysiol. 1992 May;67(5):1335-45
1597716
J Neurosci. 2009 Jun 3;29(22):7278-89
19494150
Behav Brain Res. 1985 Aug;16(2-3):117-33
4041212
Neuron. 2012 Feb 23;73(4):789-802
22365551
Neuroreport. 2000 Mar 20;11(4):893-7
10757540
J Neurosci. 2013 Jul 24;33(30):12519-27
23884955
Neuroreport. 2001 Dec 4;12(17):3683-7
11726774
J Neurophysiol. 2006 Jan;95(1):301-10
16192329
J Neurosci. 2005 Jan 26;25(4):962-70
15673677
Proc Natl Acad Sci U S A. 2009 Oct 6;106(40):17199-204
19805082
J Neurosci. 2007 Nov 7;27(45):12176-89
17989284
Nat Neurosci. 1998 Jun;1(2):155-9
10195132
Neuron. 2001 Feb;29(2):537-45
11239442
Science. 1997 Mar 14;275(5306):1593-9
9054347
Nature. 1999 Jul 15;400(6741):233-8
10421364
J Neurosci. 2013 Oct 30;33(44):17385-97
24174671
Conscious Cogn. 2011 Jun;20(2):355-62
20510630
J Neurosci. 2015 Oct 21;35(42):14220-33
26490862
Nat Neurosci. 2014 Nov;17(11):1613-22
25306549
J Exp Psychol Anim Behav Process. 2004 Apr;30(2):104-17
15078120
J Neurosci. 2007 Aug 1;27(31):8170-3
17670961
Neural Netw. 1999 Oct;12(7-8):961-974
12662639
Philos Trans R Soc Lond B Biol Sci. 2008 Dec 12;363(1511):3787-800
18829426
Proc Natl Acad Sci U S A. 2016 Feb 16;113(7):1943-8
26831116
Nat Commun. 2014 Jul 23;5:4390
25055179
Psychopharmacology (Berl). 2007 Apr;191(3):599-607
16972103
J Neurosci. 2005 Aug 24;25(34):7763-70
16120777
Neuroimage. 2003 Aug;19(4):1439-48
12948701
Curr Opin Neurobiol. 1999 Apr;9(2):195-202
10322181
Annu Rev Neurosci. 2010;33:173-202
20331363
Nat Neurosci. 2013 Jul;16(7):777-9
23799468