Active Inference, epistemic value, and vicarious trial and error.
Balancing habitual and deliberate forms of choice entails a comparison of their respective merits-the former being faster but inflexible, and the latter slower but more versatile. Here, we show that arbitration between these two forms of control can be derived from first principles within an Active Inference scheme. We illustrate our arguments with simulations that reproduce rodent spatial decisions in T-mazes. In this context, deliberation has been associated with vicarious trial and error (VTE) behavior (i.e., the fact that rodents sometimes stop at decision points as if deliberating between choice alternatives), whose neurophysiological correlates are "forward sweeps" of hippocampal place cells in the arms of the maze under consideration. Crucially, forward sweeps arise early in learning and disappear shortly after, marking a transition from deliberative to habitual choice. Our simulations show that this transition emerges as the optimal solution to the trade-off between policies that maximize reward or extrinsic value (habitual policies) and those that also consider the epistemic value of exploratory behavior (deliberative or epistemic policies)-the latter requiring VTE and the retrieval of episodic information via forward sweeps. We thus offer a novel perspective on the optimality principles that engender forward sweeps and VTE, and on their role on deliberate choice.
Institute of Cognitive Sciences and Technologies, National Research Council, 00185 Rome, Italy giovanni.pezzulo@istc.cnr.it.
Giovanni
0000-0001-6813-8282
G
Pezzulo
Institute of Cognitive Sciences and Technologies, National Research Council, 00185 Rome, Italy La Sapienza University of Rome, Rome, 00185 Italy.
Emilio
E
Cartoni
The Wellcome Trust Centre for Neuroimaging, UCL, London WC1N 3BG, United Kingdom.
Francesco
F
Rigoli
Pascal Institute, Clermont University, 63000 Clermont-Ferrand, France.
Léo
L
Pio-Lopez
The Wellcome Trust Centre for Neuroimaging, UCL, London WC1N 3BG, United Kingdom.
Karl
K
Friston
D000818:Animals; D001499:Bayes Theorem; D003657:Decision Making; D006801:Humans; D018782:Maze Learning; D008960:Models, Psychological; D051381:Rats; D012201:Reward