title: Recursive Bayesian recurrent neural networks for time-series modeling.
abstract: This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg-Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.
authors.affiliation: Department of Computing, Goldsmiths College, University of London, London, UK. D.T.Mirikitani@gold.ac.uk
authors.forename: Derrick T
authors.initials: DT
authors.lastname: Mirikitani
authors.forename: Nikolay
authors.initials: N
authors.lastname: Nikolaev
mesh_terms: D000465:Algorithms; D001499:Bayes Theorem; D003198:Computer Simulation; D004560:Electricity; D007834:Lasers; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics; D011336:Probability; D013997:Time Factors