title: Memory Transformation Enhances Reinforcement Learning in Dynamic Environments.
abstract: Over the course of systems consolidation, there is a switch from a reliance on detailed episodic memories to generalized schematic memories. This switch is sometimes referred to as "memory transformation." Here we demonstrate a previously unappreciated benefit of memory transformation, namely, its ability to enhance reinforcement learning in a dynamic environment. We developed a neural network that is trained to find rewards in a foraging task where reward locations are continuously changing. The network can use memories for specific locations (episodic memories) and statistical patterns of locations (schematic memories) to guide its search. We find that switching from an episodic to a schematic strategy over time leads to enhanced performance due to the tendency for the reward location to be highly correlated with itself in the short-term, but regress to a stable distribution in the long-term. We also show that the statistics of the environment determine the optimal utilization of both types of memory. Our work recasts the theoretical question of why memory transformation occurs, shifting the focus from the avoidance of memory interference toward the enhancement of reinforcement learning across multiple timescales.
  
  
  
  As time passes, memories transform from a highly detailed state to a more gist-like state, in a process called "memory transformation." Theories of memory transformation speak to its advantages in terms of reducing memory interference, increasing memory robustness, and building models of the environment. However, the role of memory transformation from the perspective of an agent that continuously acts and receives reward in its environment is not well explored. In this work, we demonstrate a view of memory transformation that defines it as a way of optimizing behavior across multiple timescales.
authors.affiliation: Institute of Medical Sciences, University of Toronto, Toronto, Ontario M5S 1AB, Canada.|Program in Neurosciences and Mental Health, Hospital for Sick Children, Toronto, Ontario M5G 1X8, Canada.
authors.forename: Adam
authors.initials: A
authors.lastname: Santoro
authors.affiliation: Institute of Medical Sciences, University of Toronto, Toronto, Ontario M5S 1AB, Canada, paul.frankland@sickkids.ca blake.richards@utoronto.ca.|Program in Neurosciences and Mental Health, Hospital for Sick Children, Toronto, Ontario M5G 1X8, Canada.|Department of Psychology, University of Toronto, Toronto, Ontario M5S 3G3, Canada.|Department of Physiology, University of Toronto, Toronto, Ontario M5S 1A8, Canada.
authors.forename: Paul W
authors.initials: PW
authors.lastname: Frankland
authors.affiliation: Department of Biological Sciences, University of Toronto Scarborough, Toronto, Ontario M1C 1A4, Canada, and paul.frankland@sickkids.ca blake.richards@utoronto.ca.|Department of Cell and Systems Biology, University of Toronto, Toronto, Ontario M5S 3G5, Canada.
authors.forename: Blake A
authors.identifier: 0000-0001-9662-2151
authors.initials: BA
authors.lastname: Richards
mesh_terms: D000222:Adaptation, Physiological / Q000502:physiology; D000818:Animals; D002755:Choice Behavior / Q000502:physiology; D003198:Computer Simulation; D006624:Hippocampus / Q000502:physiology*; D006801:Humans; D061212:Memory, Episodic; D008570:Memory, Short-Term / Q000502:physiology*; D011939:Mental Recall / Q000502:physiology*; D008959:Models, Neurological; D012201:Reward
keywords: computational modeling; decision making; episodic memory; memory transformation; reinforcement learning; schema