
Building autoencoders in Keras - fchollet
http://blog.keras.io/building-autoencoders-in-keras.html
======
hooloovoo_zoo
What a lovely tutorial. It should also be noted that autoencoders are useful
for supervised learning as feature generator for more effective (problem
specific) techniques than NNs like GBMs.

~~~
nomailing
Could you please elaborate on this. I would really like to know if
autoencoders are still useful for classification if I have only labels for a
small part of my training data. Is unsupervised pretraining still useful or
was it completely replaced by other techniques as the article somehow seems to
suggest?

~~~
Homunculiheaded
A single layer autoencoder with n nodes is equivalent to doing PCA and taking
the first n principal components. If you're familiar with PCA in natural
language processing, which is called Latent Semantic Analysis (or Indexing),
projecting high dimensional data on a lower dimensional surface can actually
improve your features. This is because similar words will project onto the
same Principal component allowing you to model some semantic information.

Autoencoders with more than 1 layer are more interesting because you end up
doing what is essentially non-linear PCA by projecting your data onto a curved
manifold. This famous paper, "Reducing the Dimensionality of Data with Neural
Networks" [0], by Hinton shows the improvement in how linearly separable
documents become once multi-layer autoencoders are used.

The old argument was that unsupervised pretraining helps get proper weights
faster, but this has largely been disproven. However, I do believe AEs assist
in semi-supervised learning because they project the initial data into a more
useful space. As you can seen in the article I linked the projected data are
much more linearly separable.

And as a practical evidence: I used a 5 layer AE in the kaggle black box
competition [1] to eventually outrank of team of Hinton's grad students. The
problem had a larger unsupervised data set with a small number of labels.
Using the autoencoders before the MLP ended up nearly doubling our team's
score.

[0]
[https://www.cs.toronto.edu/~hinton/science.pdf](https://www.cs.toronto.edu/~hinton/science.pdf)
[1] [https://www.kaggle.com/c/challenges-in-representation-
learni...](https://www.kaggle.com/c/challenges-in-representation-learning-the-
black-box-learning-challenge/leaderboard)

~~~
nomailing
Thank you for the answer. That makes very much sense.

Just a side note: As far as I know a single layer autoencoder and PCA are only
equivalent if all units have no activation function (linear activation
function), which is usually not the case.

------
isseu
I don't understand the image denoising. He put noise in the images, but never
use them in the code.. It's an error from the author or I missed something?

edit: Author fixed it

------
glial
I hadn't heard of Keras, but it looks easy to use. Can anyone comment on how
it compares to the other deep learning helper packages out there?

~~~
ogrisel
It's both simple to use and very easy to customize to build ad-hoc
architectures with custom nodes. The development is very active and it's well
documented.

It can also use either tensorflow or theano as runtime.

------
callesgg
if you just want it and dont want to build it:
[http://getnarrative.com/](http://getnarrative.com/)

~~~
msandford
This has nothing to do with autoencoders.

