A decentralized training algorithm for Echo State Networks in distributed big data applications.
The current big data deluge requires innovative solutions for performing efficient inference on large, heterogeneous amounts of information. Apart from the known challenges deriving from high volume and velocity, real-world big data applications may impose additional technological constraints, including the need for a fully decentralized training architecture. While several alternatives exist for training feed-forward neural networks in such a distributed setting, less attention has been devoted to the case of decentralized training of recurrent neural networks (RNNs). In this paper, we propose such an algorithm for a class of RNNs known as Echo State Networks. The algorithm is based on the well-known Alternating Direction Method of Multipliers optimization procedure. It is formulated only in terms of local exchanges between neighboring agents, without reliance on a coordinating node. Additionally, it does not require the communication of training patterns, which is a crucial component in realistic big data implementations. Experimental results on large scale artificial datasets show that it compares favorably with a fully centralized implementation, in terms of speed, efficiency and generalization accuracy.
Department of Information Engineering, Electronics and Telecommunications (DIET), "Sapienza" University of Rome, Via Eudossiana 18, 00184 Rome, Italy. Electronic address: simone.scardapane@uniroma1.it.
Simone
S
Scardapane
Department of Computer Science and Information Technology, La Trobe University, Melbourne, VIC 3086, Australia. Electronic address: dh.wang@latrobe.edu.au.
Dianhui
D
Wang
Department of Information Engineering, Electronics and Telecommunications (DIET), "Sapienza" University of Rome, Via Eudossiana 18, 00184 Rome, Italy. Electronic address: massimo.panella@uniroma1.it.
Massimo
M
Panella
D000465:Algorithms; D016208:Databases, Factual / Q000706:statistics & numerical data / Q000639:trends; D066264:Datasets as Topic / Q000639:trends; D006801:Humans; D016571:Neural Networks, Computer
Alternating Direction Method of Multipliers; Big data; Distributed learning; Echo State Network; Recurrent neural network