Spurious valleys in the error surface of recurrent networks--analysis and avoidance.
This paper gives a detailed analysis of the error surfaces of certain recurrent networks and explains some difficulties encountered in training recurrent networks. We show that these error surfaces contain many spurious valleys, and we analyze the mechanisms that cause the valleys to appear. We demonstrate that the principle mechanism can be understood through the analysis of the roots of random polynomials. This paper also provides suggestions for improvements in batch training procedures that can help avoid the difficulties caused by spurious valleys, thereby improving training speed and reliability.
Horn
Jason
J
Agilent Technologies High Frequency Technology Center, Santa Clara, CA 95051 USA. jason@jasonhorn.com
De Jesús
Orlando
O
Hagan
Martin T
MT
D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics