title: Active inference and learning.
abstract: This paper offers an active inference account of choice behaviour and learning. It focuses on the distinction between goal-directed and habitual behaviour and how they contextualise each other. We show that habits emerge naturally (and autodidactically) from sequential policy optimisation when agents are equipped with state-action policies. In active inference, behaviour has explorative (epistemic) and exploitative (pragmatic) aspects that are sensitive to ambiguity and risk respectively, where epistemic (ambiguity-resolving) behaviour enables pragmatic (reward-seeking) behaviour and the subsequent emergence of habits. Although goal-directed and habitual policies are usually associated with model-based and model-free schemes, we find the more important distinction is between belief-free and belief-based schemes. The underlying (variational) belief updating provides a comprehensive (if metaphorical) process theory for several phenomena, including the transfer of dopamine responses, reversal learning, habit formation and devaluation. Finally, we show that active inference reduces to a classical (Bellman) scheme, in the absence of ambiguity.
authors.affiliation: The Wellcome Trust Centre for Neuroimaging, UCL, 12 Queen Square, London, United Kingdom. Electronic address: k.friston@ucl.ac.uk.
authors.forename: Karl
authors.initials: K
authors.lastname: Friston
authors.affiliation: The Wellcome Trust Centre for Neuroimaging, UCL, 12 Queen Square, London, United Kingdom; Max-Planck⿿UCL Centre for Computational Psychiatry and Ageing Research, London, United Kingdom. Electronic address: thomas.fitzgerald@ucl.ac.uk.
authors.forename: Thomas
authors.initials: T
authors.lastname: FitzGerald
authors.affiliation: The Wellcome Trust Centre for Neuroimaging, UCL, 12 Queen Square, London, United Kingdom. Electronic address: f.rigoli@ucl.ac.uk.
authors.forename: Francesco
authors.initials: F
authors.lastname: Rigoli
authors.affiliation: The Wellcome Trust Centre for Neuroimaging, UCL, 12 Queen Square, London, United Kingdom; Max-Planck⿿UCL Centre for Computational Psychiatry and Ageing Research, London, United Kingdom; Centre for Neurocognitive Research, University of Salzburg, Salzburg, Austria; Neuroscience Institute, Christian-Doppler-Klinik, Paracelsus Medical University Salzburg, Salzburg, Austria. Electronic address: philipp.schwartenbeck.12@ucl.ac.uk.
authors.forename: Philipp
authors.initials: P
authors.lastname: Schwartenbeck
authors.affiliation: Caltech Brain Imaging Center, California Institute of Technology, Pasadena, USA. Electronic address: jdoherty@hss.caltech.edu.
authors.forename: John
authors.initials: J
authors.lastname: O Doherty
authors.affiliation: Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy. Electronic address: giovanni.pezzulo@istc.cnr.it.
authors.forename: Giovanni
authors.initials: G
authors.lastname: Pezzulo
mesh_terms: D002755:Choice Behavior; D004298:Dopamine; D006184:Habits; D007858:Learning; D012201:Reward
chemical_list: D004298:Dopamine
keywords: Active inference; Bayesian inference; Bayesian surprise; Epistemic value; Exploitation; Exploration; Free energy; Goal-directed; Habit learning; Information gain