title: The "proactive" model of learning: Integrative framework for model-free and model-based reinforcement learning utilizing the associative learning-based proactive brain concept.
abstract: Reinforcement learning (RL) is a powerful concept underlying forms of associative learning governed by the use of a scalar reward signal, with learning taking place if expectations are violated. RL may be assessed using model-based and model-free approaches. Model-based reinforcement learning involves the amygdala, the hippocampus, and the orbitofrontal cortex (OFC). The model-free system involves the pedunculopontine-tegmental nucleus (PPTgN), the ventral tegmental area (VTA) and the ventral striatum (VS). Based on the functional connectivity of VS, model-free and model based RL systems center on the VS that by integrating model-free signals (received as reward prediction error) and model-based reward related input computes value. Using the concept of reinforcement learning agent we propose that the VS serves as the value function component of the RL agent. Regarding the model utilized for model-based computations we turned to the proactive brain concept, which offers an ubiquitous function for the default network based on its great functional overlap with contextual associative areas. Hence, by means of the default network the brain continuously organizes its environment into context frames enabling the formulation of analogy-based association that are turned into predictions of what to expect. The OFC integrates reward-related information into context frames upon computing reward expectation by compiling stimulus-reward and context-reward information offered by the amygdala and hippocampus, respectively. Furthermore we suggest that the integration of model-based expectations regarding reward into the value signal is further supported by the efferent of the OFC that reach structures canonical for model-free learning (e.g., the PPTgN, VTA, and VS).
authors.affiliation: Department of Health Systems Management and Quality Management for Health Care, Faculty of Public Health, University of Debrecen.
authors.forename: Judit
authors.initials: J
authors.lastname: Zsuga
authors.affiliation: Department of Health Systems Management and Quality Management for Health Care, Faculty of Public Health, University of Debrecen.
authors.forename: Klara
authors.initials: K
authors.lastname: Biro
authors.affiliation: Department of Health Systems Management and Quality Management for Health Care, Faculty of Public Health, University of Debrecen.
authors.forename: Csaba
authors.initials: C
authors.lastname: Papp
authors.affiliation: Department of Health Systems Management and Quality Management for Health Care, Faculty of Public Health, University of Debrecen.
authors.forename: Gabor
authors.initials: G
authors.lastname: Tajti
authors.affiliation: Department of Pharmacology, Faculty of Pharmacy, University of Debrecen.
authors.forename: Rudolf
authors.initials: R
authors.lastname: Gesztelyi
mesh_terms: D000818:Animals; D001245:Association Learning / Q000502:physiology*; D001921:Brain / Q000502:physiology*; D003198:Computer Simulation; D006801:Humans; D008959:Models, Neurological; D009434:Neural Pathways / Q000502:physiology*; D012054:Reinforcement, Psychology