title: Learning Multirobot Hose Transportation and Deployment by Distributed Round-Robin Q-Learning.
abstract: Multi-Agent Reinforcement Learning (MARL) algorithms face two main difficulties: the curse of dimensionality, and environment non-stationarity due to the independent learning processes carried out by the agents concurrently. In this paper we formalize and prove the convergence of a Distributed Round Robin Q-learning (D-RR-QL) algorithm for cooperative systems. The computational complexity of this algorithm increases linearly with the number of agents. Moreover, it eliminates environment non sta tionarity by carrying a round-robin scheduling of the action selection and execution. That this learning scheme allows the implementation of Modular State-Action Vetoes (MSAV) in cooperative multi-agent systems, which speeds up learning convergence in over-constrained systems by vetoing state-action pairs which lead to undesired termination states (UTS) in the relevant state-action subspace. Each agent's local state-action value function learning is an independent process, including the MSAV policies. Coordination of locally optimal policies to obtain the global optimal joint policy is achieved by a greedy selection procedure using message passing. We show that D-RR-QL improves over state-of-the-art approaches, such as Distributed Q-Learning, Team Q-Learning and Coordinated Reinforcement Learning in a paradigmatic Linked Multi-Component Robotic System (L-MCRS) control problem: the hose transportation task. L-MCRS are over-constrained systems with many UTS induced by the interaction of the passive linking element and the active mobile robots.
authors.affiliation: Polytechnical Institute and Computational Intelligence Group, University of the Basque Country (UPV/EHU), Vitoria-Gasteiz, Spain.
authors.forename: Borja
authors.initials: B
authors.lastname: Fernandez-Gauna
authors.affiliation: Polytechnical Institute, University of the Basque Country (UPV/EHU), Vitoria-Gasteiz, Spain.
authors.forename: Ismael
authors.initials: I
authors.lastname: Etxeberria-Agiriano
authors.affiliation: Computational Intelligence Group, University of the Basque Country (UPV/EHU), San Sebastian, Spain, & ENGINE centre Wroclaw University of Technology (WrUT), Wroclaw, Poland.
authors.forename: Manuel
authors.initials: M
authors.lastname: Graña
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D007858:Learning; D008390:Markov Chains