title: Learning eigenfunctions links spectral embedding and kernel PCA.
abstract: In this letter, we show a direct relation between spectral embedding methods and kernel principal components analysis and how both are special cases of a more general learning problem: learning the principal eigenfunctions of an operator defined from a kernel and the unknown data-generating density. Whereas spectral embedding methods provided only coordinates for the training points, the analysis justifies a simple extension to out-of-sample examples (the Nyström formula) for multidimensional scaling (MDS), spectral clustering, Laplacian eigenmaps, locally linear embedding (LLE), and Isomap. The analysis provides, for all such spectral embedding methods, the definition of a loss function, whose empirical average is minimized by the traditional algorithms. The asymptotic expected value of that loss defines a generalization performance and clarifies what these algorithms are trying to learn. Experiments with LLE, Isomap, spectral clustering, and MDS show that this out-of-sample embedding formula generalizes well, with a level of error comparable to the effect of small perturbations of the training set on the embedding.
authors.affiliation: Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, H3C 3J7, Canada. bengioy@iro.umontreal.ca
authors.forename: Yoshua
authors.initials: Y
authors.lastname: Bengio
authors.forename: Olivier
authors.initials: O
authors.lastname: Delalleau
authors.forename: Nicolas
authors.initials: N
authors.lastname: Le Roux
authors.forename: Jean-François
authors.initials: JF
authors.lastname: Paiement
authors.forename: Pascal
authors.initials: P
authors.lastname: Vincent
authors.forename: Marie
authors.initials: M
authors.lastname: Ouimet
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D016000:Cluster Analysis; D005793:Generalization, Psychological; D006801:Humans; D007858:Learning / Q000502:physiology*; D015233:Models, Statistical; D016571:Neural Networks, Computer