title: Inter-module credit assignment in modular reinforcement learning.
abstract: Critical issues in modular or hierarchical reinforcement learning (RL) are (i) how to decompose a task into sub-tasks, (ii) how to achieve independence of learning of sub-tasks, and (iii) how to assure optimality of the composite policy for the entire task. The second and last requirements are often under trade-off. We propose a method for propagating the reward for the entire task achievement between modules. This is done in the form of a 'modular reward', which is calculated from the temporal difference of the module gating signal and the value of the succeeding module. We implement modular reward for a multiple model-based reinforcement learning (MMRL) architecture and show its effectiveness in simulations of a pursuit task with hidden states and a continuous-time non-linear control task.
authors.affiliation: Human information science laboratories, ATR International, 2-2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan. samejima@atr.co.jp
authors.forename: Kazuyuki
authors.initials: K
authors.lastname: Samejima
authors.forename: Kenji
authors.initials: K
authors.lastname: Doya
authors.forename: Mitsuo
authors.initials: M
authors.lastname: Kawato
mesh_terms: D007858:Learning / Q000502:physiology; D008954:Models, Biological; D012054:Reinforcement, Psychology