A numerical study on learning curves in stochastic multilayer feedforward networks.
The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.
Müller
K R
KR
Department of Mathematical Engineering and Inf. Physics, University of Tokyo, Japan.
Finke
M
M
Murata
N
N
Schulten
K
K
Amari
S
S
D007858:Learning / Q000502:physiology*; D016571:Neural Networks, Computer; D009474:Neurons / Q000502:physiology*