title: Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation.
abstract: It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of 'Go' or 'No-Go' selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of 'Go' values towards a goal, and (2) value-contrasts between 'Go' and 'No-Go' are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.
authors.affiliation: Department of Biological Sciences, Graduate School of Science, The University of Tokyo, Tokyo, Japan.
authors.forename: Ayaka
authors.identifier: http://orcid.org/0000-0002-6306-6600
authors.initials: A
authors.lastname: Kato
authors.affiliation: Physical and Health Education, Graduate School of Education, The University of Tokyo, Tokyo, Japan.
authors.forename: Kenji
authors.initials: K
authors.lastname: Morita
mesh_terms: D003198:Computer Simulation; D003342:Corpus Striatum / Q000502:physiology*; D003657:Decision Making / Q000502:physiology; D004298:Dopamine / Q000378:metabolism*; D059290:Dopaminergic Neurons / Q000502:physiology; D006801:Humans; D011939:Mental Recall / Q000502:physiology*; D008959:Models, Neurological; D009042:Motivation / Q000502:physiology*; D012054:Reinforcement, Psychology
chemical_list: D004298:Dopamine