

Breaking Linear Classifiers on ImageNet - mattxxx
http://karpathy.github.io/2015/03/30/breaking-convnets/

======
jostmey
I'd be interested in seeing if this same problem appears in generative models.
It seems like a flaw of discriminative models.

Convolutional Boltzmann machines have been developed, and neural networks
trained with Backpropagation can be pre-trained with generative models. Would
this help alleviate the problem?

~~~
Houshalter
The original paper found that nets pretrained as autoencoders were the least
susceptible to it, but still failed. IIRC the follow up paper
([http://arxiv.org/abs/1412.6572](http://arxiv.org/abs/1412.6572)) tested it
or at least claimed it wasn't a solution.

------
wodenokoto
I tried replicated Nguyen et. al's experiment on MNIST. I decided to do it
using different classifiers and that "state of the art" was less than 5
percent 0-1 error on testing set.

I ended up with a ~98% accurate Gaussian kernel SVM classifier and my
generated images all looked like numbers.

I also got ~96% using KNN, but haven't figured out how I would measure
confidence in such a classifier (average distance?)

Any ideas?

------
panic
Can our own visual system be fooled in a similar way, or is the response of
the neurons involved too non-linear?

~~~
Houshalter
Possibly. We can't backpropagate through all the neurons in a human brain. If
we could we might be able to create real BLITs
[http://www.infinityplus.co.uk/stories/blit.htm](http://www.infinityplus.co.uk/stories/blit.htm)
.

------
irascible
What are the implications of this with self driving cars or biometric security
devices?

Also.. In the "incomprehensible noise" pattern classes, I can pretty clearly
see features that I would identify as a robin, cheetah, armadillo, and lesser
panda. I don't think this means the classifiers are being fooled, they are
just not being trained to recognize the class of "noisy image that looks kind
of like something but is actally noise"

~~~
wodenokoto
I think the required noise / subspaces are unlikely to be met in real world
scenarios. That is, you won't take a picture of a spider where the sun just
happens to shine in the right way and your lens is in just the right angle to
make an image recognized as a giraf.

It may become an attack vector, but one the is very difficult to fit to the
classifier you are targeting (your car may be using online learning, so there
is a chance that an image generated on one self driving car won't work on
another of the same model) then you have to show this carefully generated
image to the car, but if you show it to the camera, it will already be
distorted beyond its fooling capability. due to angle, lighting, lenses etc.

