Autoencoder
Reducing the Dimensionality of
Data with Neural Networks, Science, 2006
High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors.
Encoder and Decoder
Multilayer “encoder” network to transform the high-dimensional data into a low-dimensional code and a similar “decoder” network to recover the data from the code.
Basic structure
Unfolding and Fine-tuning
Stacked Encoder
A single layer of binary features is not the best way to model the structure … The first layer of feature detectors then become the visible units for learning the next RBM. This layer-by-layer learning can be repeated as many times as desired.
The layer-by-layer learning algorithm is nonetheless a very effective way to pretrain the weights of a deep autoencoder. Each layer of features captures strong, high-order correlations between the activities of units in the layer below. For a wide variety of data sets, this is an efficient way to progressively reveal low-dimensional, nonlinear structure.
Summary
- The autoencoder can work as a tool to reduce the dimensionality of data by learning the low-dimensional codes.
- The pretraining precedure of autoencoder can provide good initial weights for the neural networks.