Recursive Bayesian recurrent neural networks for time-series modeling.
This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg-Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.
Mirikitani
Derrick T
DT
Department of Computing, Goldsmiths College, University of London, London, UK. D.T.Mirikitani@gold.ac.uk
Nikolaev
Nikolay
N
D000465:Algorithms; D001499:Bayes Theorem; D003198:Computer Simulation; D004560:Electricity; D007834:Lasers; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics; D011336:Probability; D013997:Time Factors