title: Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task.
abstract: The recently developed 'two-step' behavioural task promises to differentiate model-based from model-free reinforcement learning, while generating neurophysiologically-friendly decision datasets with parametric variation of decision variables. These desirable features have prompted its widespread adoption. Here, we analyse the interactions between a range of different strategies and the structure of transitions and outcomes in order to examine constraints on what can be learned from behavioural performance. The task involves a trade-off between the need for stochasticity, to allow strategies to be discriminated, and a need for determinism, so that it is worth subjects' investment of effort to exploit the contingencies optimally. We show through simulation that under certain conditions model-free strategies can masquerade as being model-based. We first show that seemingly innocuous modifications to the task structure can induce correlations between action values at the start of the trial and the subsequent trial events in such a way that analysis based on comparing successive trials can lead to erroneous conclusions. We confirm the power of a suggested correction to the analysis that can alleviate this problem. We then consider model-free reinforcement learning strategies that exploit correlations between where rewards are obtained and which actions have high expected value. These generate behaviour that appears model-based under these, and also more sophisticated, analyses. Exploiting the full potential of the two-step task as a tool for behavioural neuroscience requires an understanding of these issues.
authors.affiliation: Champalimaud Neuroscience Program, Champalimaud Centre for the Unknown, Lisbon, Portugal.|Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom.
authors.forename: Thomas
authors.initials: T
authors.lastname: Akam
authors.affiliation: Champalimaud Neuroscience Program, Champalimaud Centre for the Unknown, Lisbon, Portugal.
authors.forename: Rui
authors.initials: R
authors.lastname: Costa
authors.affiliation: Gatsby Computational Neuroscience Unit, UCL, London, United Kingdom.
authors.forename: Peter
authors.initials: P
authors.lastname: Dayan
mesh_terms: D002755:Choice Behavior / Q000502:physiology*; D003198:Computer Simulation; D006184:Habits; D006801:Humans; D008959:Models, Neurological; D015233:Models, Statistical; D012054:Reinforcement, Psychology; D012193:Reversal Learning / Q000502:physiology*; D013647:Task Performance and Analysis