Inter-module credit assignment in modular reinforcement learning.
Critical issues in modular or hierarchical reinforcement learning (RL) are (i) how to decompose a task into sub-tasks, (ii) how to achieve independence of learning of sub-tasks, and (iii) how to assure optimality of the composite policy for the entire task. The second and last requirements are often under trade-off. We propose a method for propagating the reward for the entire task achievement between modules. This is done in the form of a 'modular reward', which is calculated from the temporal difference of the module gating signal and the value of the succeeding module. We implement modular reward for a multiple model-based reinforcement learning (MMRL) architecture and show its effectiveness in simulations of a pursuit task with hidden states and a continuous-time non-linear control task.
Samejima
Kazuyuki
K
Human information science laboratories, ATR International, 2-2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan. samejima@atr.co.jp
Doya
Kenji
K
Kawato
Mitsuo
M
D007858:Learning / Q000502:physiology; D008954:Models, Biological; D012054:Reinforcement, Psychology