title: Efficient reinforcement learning of a reservoir network model of parametric working memory achieved with a cluster population winner-take-all readout mechanism.
abstract: The brain often has to make decisions based on information stored in working memory, but the neural circuitry underlying working memory is not fully understood. Many theoretical efforts have been focused on modeling the persistent delay period activity in the prefrontal areas that is believed to represent working memory. Recent experiments reveal that the delay period activity in the prefrontal cortex is neither static nor homogeneous as previously assumed. Models based on reservoir networks have been proposed to model such a dynamical activity pattern. The connections between neurons within a reservoir are random and do not require explicit tuning. Information storage does not depend on the stable states of the network. However, it is not clear how the encoded information can be retrieved for decision making with a biologically realistic algorithm. We therefore built a reservoir-based neural network to model the neuronal responses of the prefrontal cortex in a somatosensory delayed discrimination task. We first illustrate that the neurons in the reservoir exhibit a heterogeneous and dynamical delay period activity observed in previous experiments. Then we show that a cluster population circuit decodes the information from the reservoir with a winner-take-all mechanism and contributes to the decision making. Finally, we show that the model achieves a good performance rapidly by shaping only the readout with reinforcement learning. Our model reproduces important features of previous behavior and neurophysiology data. We illustrate for the first time how task-specific information stored in a reservoir network can be retrieved with a biologically plausible reinforcement learning training scheme.
authors.affiliation: State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China; Department of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China; and.
authors.forename: Zhenbo
authors.initials: Z
authors.lastname: Cheng
authors.affiliation: State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China;
authors.forename: Zhidong
authors.initials: Z
authors.lastname: Deng
authors.affiliation: State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China;
authors.forename: Xiaolin
authors.initials: X
authors.lastname: Hu
authors.affiliation: State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China;
authors.forename: Bo
authors.initials: B
authors.lastname: Zhang
authors.affiliation: Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai Institutes for Biological Sciences, Chinese Academy of Sciences, Shanghai, China tyang@ion.ac.cn.
authors.forename: Tianming
authors.identifier: http://orcid.org/0000-0001-6976-9246
authors.initials: T
authors.lastname: Yang
mesh_terms: D000818:Animals; D006801:Humans; D008570:Memory, Short-Term; D008959:Models, Neurological; D009474:Neurons / Q000502:physiology; D017397:Prefrontal Cortex / Q000166:cytology / Q000502:physiology; D012054:Reinforcement, Psychology
keywords: reinforcement learning; reservoir network; working memory