title: OCReP: An Optimally Conditioned Regularization for pseudoinversion based neural training.
abstract: In this paper we consider the training of single hidden layer neural networks by pseudoinversion, which, in spite of its popularity, is sometimes affected by numerical instability issues. Regularization is known to be effective in such cases, so that we introduce, in the framework of Tikhonov regularization, a matricial reformulation of the problem which allows us to use the condition number as a diagnostic tool for identification of instability. By imposing well-conditioning requirements on the relevant matrices, our theoretical analysis allows the identification of an optimal value for the regularization parameter from the standpoint of stability. We compare with the value derived by cross-validation for overfitting control and optimization of the generalization performance. We test our method for both regression and classification tasks. The proposed method is quite effective in terms of predictivity, often with some improvement on performance with respect to the reference cases considered. This approach, due to analytical determination of the regularization parameter, dramatically reduces the computational load required by many other techniques.
authors: University of Turin, Department of Computer Sciences, C.so Svizzera 185, 10149 Torino, Italy. Electronic address: cancelli@di.unito.it.
authors: Rossella
authors: R
authors: Cancelliere
authors: National Institute of Astrophysics, Astrophys. Observ. of Torino, Pino T.se (TO), Italy.
authors: Mario
authors: M
authors: Gai
authors: Laboratory of Computer Sciences, LIP6, Univ. Pierre et Marie Curie, Paris, France.
authors: Patrick
authors: P
authors: Gallinari
authors: University of Turin, Department of Computer Sciences, C.so Svizzera 185, 10149 Torino, Italy.
authors: Luca
authors: L
authors: Rubini
mesh_terms: D000465:Algorithms; D002965:Classification; D000069550:Machine Learning; D016571:Neural Networks, Computer; D015203:Reproducibility of Results
keywords: Condition number; Numerical instability; Pseudoinversion; Regularization parameter