title: Reversal Learning in Humans and Gerbils: Dynamic Control Network Facilitates Learning.
abstract: Biologically plausible modeling of behavioral reinforcement learning tasks has seen great improvements over the past decades. Less work has been dedicated to tasks involving contingency reversals, i.e., tasks in which the original behavioral goal is reversed one or multiple times. The ability to adjust to such reversals is a key element of behavioral flexibility. Here, we investigate the neural mechanisms underlying contingency-reversal tasks. We first conduct experiments with humans and gerbils to demonstrate memory effects, including multiple reversals in which subjects (humans and animals) show a faster learning rate when a previously learned contingency re-appears. Motivated by recurrent mechanisms of learning and memory for object categories, we propose a network architecture which involves reinforcement learning to steer an orienting system that monitors the success in reward acquisition. We suggest that a model sensory system provides feature representations which are further processed by category-related subnetworks which constitute a neural analog of expert networks. Categories are selected dynamically in a competitive field and predict the expected reward. Learning occurs in sequentialized phases to selectively focus the weight adaptation to synapses in the hierarchical network and modulate their weight changes by a global modulator signal. The orienting subsystem itself learns to bias the competition in the presence of continuous monotonic reward accumulation. In case of sudden changes in the discrepancy of predicted and acquired reward the activated motor category can be switched. We suggest that this subsystem is composed of a hierarchically organized network of dis-inhibitory mechanisms, dubbed a dynamic control network (DCN), which resembles components of the basal ganglia. The DCN selectively activates an expert network, corresponding to the current behavioral strategy. The trace of the accumulated reward is monitored such that large sudden deviations from the monotonicity of its evolution trigger a reset after which another expert subnetwork can be activated-if it has already been established before-or new categories can be recruited and associated with novel behavioral patterns.
authors.affiliation: Faculty of Engineering, Computer Science and Psychology, Institute of Neural Information Processing, Ulm University Ulm, Germany.
authors.forename: Christian
authors.initials: C
authors.lastname: Jarvers
authors.affiliation: Faculty of Engineering, Computer Science and Psychology, Institute of Neural Information Processing, Ulm University Ulm, Germany.
authors.forename: Tobias
authors.initials: T
authors.lastname: Brosch
authors.affiliation: Special Lab Non-Invasive Brain Imaging, Leibniz Institute for Neurobiology Magdeburg, Germany.
authors.forename: André
authors.initials: A
authors.lastname: Brechmann
authors.affiliation: Department Systems Physiology, Leibniz Institute for Neurobiology Magdeburg, Germany.
authors.forename: Marie L
authors.initials: ML
authors.lastname: Woldeit
authors.affiliation: Department Systems Physiology, Leibniz Institute for Neurobiology Magdeburg, Germany.
authors.forename: Andreas L
authors.initials: AL
authors.lastname: Schulz
authors.affiliation: Department Systems Physiology, Leibniz Institute for Neurobiology Magdeburg, Germany.
authors.forename: Frank W
authors.initials: FW
authors.lastname: Ohl
authors.affiliation: Special Lab Non-Invasive Brain Imaging, Leibniz Institute for Neurobiology Magdeburg, Germany.
authors.forename: Marcel
authors.initials: M
authors.lastname: Lommerzheim
authors.affiliation: Faculty of Engineering, Computer Science and Psychology, Institute of Neural Information Processing, Ulm University Ulm, Germany.
authors.forename: Heiko
authors.initials: H
authors.lastname: Neumann
keywords: adaptive resonance theory; expert networks; recurrent neural networks; reinforcement learning; reversal learning; stability-plasticity