title: Two-layer contractive encodings for learning stable nonlinear features.
abstract: Unsupervised learning of feature hierarchies is often a good strategy to initialize deep architectures for supervised learning. Most existing deep learning methods build these feature hierarchies layer by layer in a greedy fashion using either auto-encoders or restricted Boltzmann machines. Both yield encoders which compute linear projections of input followed by a smooth thresholding function. In this work, we demonstrate that these encoders fail to find stable features when the required computation is in the exclusive-or class. To overcome this limitation, we propose a two-layer encoder which is less restricted in the type of features it can learn. The proposed encoder is regularized by an extension of previous work on contractive regularization. This proposed two-layer contractive encoder potentially poses a more difficult optimization problem, and we further propose to linearly transform hidden neurons of the encoder to make learning easier. We demonstrate the advantages of the two-layer encoders qualitatively on artificially constructed datasets as well as commonly used benchmark datasets. We also conduct experiments on a semi-supervised learning task and show the benefits of the proposed two-layer encoders trained with the linear transformation of perceptrons.
authors.affiliation: Autonomous Intelligent Systems, Computer Science Institute VI, University of Bonn, Germany. Electronic address: schulz@ais.uni-bonn.de.
authors.forename: Hannes
authors.initials: H
authors.lastname: Schulz
authors.affiliation: Department of Information and Computer Science, Aalto University School of Science, Finland.
authors.forename: Kyunghyun
authors.initials: K
authors.lastname: Cho
authors.affiliation: Department of Information and Computer Science, Aalto University School of Science, Finland.
authors.forename: Tapani
authors.initials: T
authors.lastname: Raiko
authors.affiliation: Autonomous Intelligent Systems, Computer Science Institute VI, University of Bonn, Germany.
authors.forename: Sven
authors.initials: S
authors.lastname: Behnke
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D016571:Neural Networks, Computer
keywords: Deep learning; Linear transformation; Multi-layer perceptron; Pretraining; Semi-supervised learning; Two-layer contractive encoding