title: Corticostriatal circuit mechanisms of value-based action selection: Implementation of reinforcement learning algorithms and beyond.
abstract: Value-based action selection has been suggested to be realized in the corticostriatal local circuits through competition among neural populations. In this article, we review theoretical and experimental studies that have constructed and verified this notion, and provide new perspectives on how the local-circuit selection mechanisms implement reinforcement learning (RL) algorithms and computations beyond them. The striatal neurons are mostly inhibitory, and lateral inhibition among them has been classically proposed to realize "Winner-Take-All (WTA)" selection of the maximum-valued action (i.e., 'max' operation). Although this view has been challenged by the revealed weakness, sparseness, and asymmetry of lateral inhibition, which suggest more complex dynamics, WTA-like competition could still occur on short time scales. Unlike the striatal circuit, the cortical circuit contains recurrent excitation, which may enable retention or temporal integration of information and probabilistic "soft-max" selection. The striatal "max" circuit and the cortical "soft-max" circuit might co-implement an RL algorithm called Q-learning; the cortical circuit might also similarly serve for other algorithms such as SARSA. In these implementations, the cortical circuit presumably sustains activity representing the executed action, which negatively impacts dopamine neurons so that they can calculate reward-prediction-error. Regarding the suggested more complex dynamics of striatal, as well as cortical, circuits on long time scales, which could be viewed as a sequence of short WTA fragments, computational roles remain open: such a sequence might represent (1) sequential state-action-state transitions, constituting replay or simulation of the internal model, (2) a single state/action by the whole trajectory, or (3) probabilistic sampling of state/action.
authors.affiliation: Physical and Health Education, Graduate School of Education, The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan. Electronic address: morita@p.u-tokyo.ac.jp.
authors.forename: Kenji
authors.initials: K
authors.lastname: Morita
authors.affiliation: Institute for Advanced Simulation (IAS-6) and Institute of Neuroscience and Medicine (INM-6) and JARA BRAIN Institute I, Jülich Research Centre, Jülich, Germany.
authors.forename: Jenia
authors.initials: J
authors.lastname: Jitsev
authors.affiliation: Institute for Advanced Simulation (IAS-6) and Institute of Neuroscience and Medicine (INM-6) and JARA BRAIN Institute I, Jülich Research Centre, Jülich, Germany; Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr-University Bochum, Germany.
authors.forename: Abigail
authors.initials: A
authors.lastname: Morrison
mesh_terms: D000465:Algorithms; D000818:Animals; D002540:Cerebral Cortex / Q000502:physiology*; D002755:Choice Behavior / Q000502:physiology*; D003342:Corpus Striatum / Q000502:physiology*; D006801:Humans; D008959:Models, Neurological; D009043:Motor Activity / Q000502:physiology*; D009434:Neural Pathways / Q000502:physiology; D012054:Reinforcement, Psychology
keywords: Action selection; Corticostriatal; Nonlinear dynamics; Reinforcement learning; Reward prediction error; Winner-take-all