title: On the selection of weight decay parameter for faulty networks.
abstract: The weight-decay technique is an effective approach to handle overfitting and weight fault. For fault-free networks, without an appropriate value of decay parameter, the trained network is either overfitted or underfitted. However, many existing results on the selection of decay parameter focus on fault-free networks only. It is well known that the weight-decay method can also suppress the effect of weight fault. For the faulty case, using a test set to select the decay parameter is not practice because there are huge number of possible faulty networks for a trained network. This paper develops two mean prediction error (MPE) formulae for predicting the performance of faulty radial basis function (RBF) networks. Two fault models, multiplicative weight noise and open weight fault, are considered. Our MPE formulae involve the training error and trained weights only. Besides, in our method, we do not need to generate a huge number of faulty networks to measure the test error for the fault situation. The MPE formulae allow us to select appropriate values of decay parameter for faulty networks. Our experiments showed that, although there are small differences between the true test errors (from the test set) and the MPE values, the MPE formulae can accurately locate the appropriate value of the decay parameter for minimizing the true test error of faulty networks.
authors.affiliation: Department of Electronic Engineering, City University of Hong Kong, Kowloon 852, Hong Kong. eeleungc@cityu.edu.hk
authors.forename: Chi Sing
authors.initials: CS
authors.lastname: Leung
authors.forename: Hong-Jiang
authors.initials: HJ
authors.lastname: Wang
authors.forename: John
authors.initials: J
authors.lastname: Sum
mesh_terms: D000465:Algorithms; D000818:Animals; D016477:Artifacts; D001185:Artificial Intelligence; D003198:Computer Simulation / Q000592:standards*; D005544:Forecasting / Q000379:methods; D006801:Humans; D016014:Linear Models; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics; D011237:Predictive Value of Tests