title: Alterations in choice behavior by manipulations of world model.
abstract: How to compute initially unknown reward values makes up one of the key problems in reinforcement learning theory, with two basic approaches being used. Model-free algorithms rely on the accumulation of substantial amounts of experience to compute the value of actions, whereas in model-based learning, the agent seeks to learn the generative process for outcomes from which the value of actions can be predicted. Here we show that (i) "probability matching"-a consistent example of suboptimal choice behavior seen in humans-occurs in an optimal Bayesian model-based learner using a max decision rule that is initialized with ecologically plausible, but incorrect beliefs about the generative process for outcomes and (ii) human behavior can be strongly and predictably altered by the presence of cues suggestive of various generative processes, despite statistically identical outcome generation. These results suggest human decision making is rational and model based and not consistent with model-free learning.
authors.affiliation: Department of Psychology, Center for Cognitive Science, University of Minnesota, Minneapolis, MN 55455, USA. csgreen@umn.edu
authors.forename: C S
authors.initials: CS
authors.lastname: Green
authors.forename: C
authors.initials: C
authors.lastname: Benson
authors.forename: D
authors.initials: D
authors.lastname: Kersten
authors.forename: P
authors.initials: P
authors.lastname: Schrater
mesh_terms: D001499:Bayes Theorem; D002755:Choice Behavior; D017753:Ecosystem; D005260:Female; D006801:Humans; D008297:Male; D008954:Models, Biological; D011336:Probability; D055815:Young Adult
references.citation: J Exp Psychol. 1953 Jan;45(1):64-74
references.pmid: 13035003
references.citation: Can J Exp Psychol. 2004 Dec;58(4):221-8
references.pmid: 15648726
references.citation: Mem Cognit. 2003 Mar;31(2):243-51
references.pmid: 12749466
references.citation: J Exp Psychol. 1959 Jun;57(6):413-6
references.pmid: 13654659
references.citation: Neuropsychology. 2007 Sep;21(5):621-30
references.pmid: 17784810
references.citation: Can J Exp Psychol. 2002 Mar;56(1):58-63
references.pmid: 11901961
references.citation: J Neurosci. 2007 Nov 21;27(47):12860-7
references.pmid: 18032658
references.citation: Nat Neurosci. 2005 Dec;8(12):1704-11
references.pmid: 16286932
references.citation: Am J Psychol. 1955 Mar;68(1):106-16
references.pmid: 14350105
references.citation: Spat Vis. 1997;10(4):433-6
references.pmid: 9176952
references.citation: Cognition. 1999 Oct 26;72(3):269-304
references.pmid: 10519925
references.citation: Cognition. 2008 Dec;109(3):416-22
references.pmid: 19019351
references.citation: Nature. 2006 Jun 15;441(7095):876-9
references.pmid: 16778890
references.citation: Neuron. 2008 Jun 26;58(6):967-73
references.pmid: 18579085
references.citation: Am J Psychol. 1959 Dec;72:597-9
references.pmid: 13822038
references.citation: Nat Rev Neurosci. 2008 Apr;9(4):292-303
references.pmid: 18319728
references.citation: Spat Vis. 1997;10(4):437-42
references.pmid: 9176953
references.citation: J Exp Psychol. 1961 Oct;62:385-94
references.pmid: 13889318