Alterations in choice behavior by manipulations of world model.
How to compute initially unknown reward values makes up one of the key problems in reinforcement learning theory, with two basic approaches being used. Model-free algorithms rely on the accumulation of substantial amounts of experience to compute the value of actions, whereas in model-based learning, the agent seeks to learn the generative process for outcomes from which the value of actions can be predicted. Here we show that (i) "probability matching"-a consistent example of suboptimal choice behavior seen in humans-occurs in an optimal Bayesian model-based learner using a max decision rule that is initialized with ecologically plausible, but incorrect beliefs about the generative process for outcomes and (ii) human behavior can be strongly and predictably altered by the presence of cues suggestive of various generative processes, despite statistically identical outcome generation. These results suggest human decision making is rational and model based and not consistent with model-free learning.
Green
C S
CS
Department of Psychology, Center for Cognitive Science, University of Minnesota, Minneapolis, MN 55455, USA. csgreen@umn.edu
Benson
C
C
Kersten
D
D
Schrater
P
P
D001499:Bayes Theorem; D002755:Choice Behavior; D017753:Ecosystem; D005260:Female; D006801:Humans; D008297:Male; D008954:Models, Biological; D011336:Probability; D055815:Young Adult
J Exp Psychol. 1953 Jan;45(1):64-74
13035003
Can J Exp Psychol. 2004 Dec;58(4):221-8
15648726
Mem Cognit. 2003 Mar;31(2):243-51
12749466
J Exp Psychol. 1959 Jun;57(6):413-6
13654659
Neuropsychology. 2007 Sep;21(5):621-30
17784810
Can J Exp Psychol. 2002 Mar;56(1):58-63
11901961
J Neurosci. 2007 Nov 21;27(47):12860-7
18032658
Nat Neurosci. 2005 Dec;8(12):1704-11
16286932
Am J Psychol. 1955 Mar;68(1):106-16
14350105
Spat Vis. 1997;10(4):433-6
9176952
Cognition. 1999 Oct 26;72(3):269-304
10519925
Cognition. 2008 Dec;109(3):416-22
19019351
Nature. 2006 Jun 15;441(7095):876-9
16778890
Neuron. 2008 Jun 26;58(6):967-73
18579085
Am J Psychol. 1959 Dec;72:597-9
13822038
Nat Rev Neurosci. 2008 Apr;9(4):292-303
18319728
Spat Vis. 1997;10(4):437-42
9176953
J Exp Psychol. 1961 Oct;62:385-94
13889318