title: Active inference and epistemic value.
abstract: We offer a formal treatment of choice behavior based on the premise that agents minimize the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (or intrinsic) value. Minimizing expected free energy is therefore equivalent to maximizing extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximizing information gain or intrinsic value (or reducing uncertainty about the causes of valuable outcomes). The resulting scheme resolves the exploration-exploitation dilemma: Epistemic value is maximized until there is no further information gain, after which exploitation is assured through maximization of extrinsic value. This is formally consistent with the Infomax principle, generalizing formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk-sensitive (Kullback-Leibler) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems, ad hoc softmax parameters become the expected (Bayes-optimal) precision of beliefs about, or confidence in, policies. This article focuses on the basic theory, illustrating the ideas with simulations. A key aspect of these simulations is the similarity between precision updates and dopaminergic discharges observed in conditioning paradigms.
authors.affiliation: a The Wellcome Trust Centre for Neuroimaging , Institute of Neurology , London , UK.
authors.forename: Karl
authors.initials: K
authors.lastname: Friston
authors.affiliation: a The Wellcome Trust Centre for Neuroimaging , Institute of Neurology , London , UK.
authors.forename: Francesco
authors.initials: F
authors.lastname: Rigoli
authors.affiliation: b Centre for Robotics Research, Department of Informatics , King's College London , London , UK.
authors.forename: Dimitri
authors.initials: D
authors.lastname: Ognibene
authors.affiliation: a The Wellcome Trust Centre for Neuroimaging , Institute of Neurology , London , UK.|c Translational Neuromodeling Unit (TNU) , Institute for Biomedical Engineering, University of Zürich and ETH Zürich , Zürich , Switzerland.|d Laboratory for Social and Neural Systems Research (SNS Lab), Department of Economics , University of Zürich , Zürich , Switzerland.
authors.forename: Christoph
authors.initials: C
authors.lastname: Mathys
authors.affiliation: a The Wellcome Trust Centre for Neuroimaging , Institute of Neurology , London , UK.
authors.forename: Thomas
authors.initials: T
authors.lastname: Fitzgerald
authors.affiliation: e Institute of Cognitive Sciences and Technologies , National Research Council , Rome , Italy.
authors.forename: Giovanni
authors.initials: G
authors.lastname: Pezzulo
mesh_terms: D001499:Bayes Theorem; D002755:Choice Behavior / Q000502:physiology*; D003210:Concept Formation / Q000502:physiology; D003657:Decision Making / Q000502:physiology*; D006801:Humans; D019359:Knowledge; D008570:Memory, Short-Term / Q000502:physiology; D008960:Models, Psychological
keywords: Active inference; Agency; Bayesian inference; Bayesian surprise; Bounded rationality; Epistemic value; Exploitation; Exploration; Free energy; Information gain; Utility theory