
How to trick a neural network into thinking a panda is a vulture - jvns
https://codewords.recurse.com/issues/five/why-do-neural-networks-think-a-panda-is-a-vulture
======
IshKebab
I like the visualisations on this page:
[http://colah.github.io/posts/2014-03-NN-Manifolds-
Topology/](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/)
(Especially "The Easy Way Out" one.

It suggests that because neural networks cannot properly separate the classes,
the often mush them together in the input-space so the "panda" region of
image-space has loads of other classes speckled though it. If you purposely
find those speckles you can make it think that a panda is a vulture.

That blog mentions adding smoothness constraints, so that you can't suddenly
go from panda to vulture with a tiny change in an image. But I wonder if an
easier solution (ok, hack) is just to add different noise to the image 10
times or so, run it through the network and then combine the results.

Essentially you'd be doing Monte Carlo sampling of the image space around the
input image. Or kind of blurring the image space.

Just an idea anyway. I don't really know what I'm talking about.

~~~
fundamental
My understanding from reading a few 'adversarial example' papers is that it's
also possible to talk about the classifier as having a poor margin between
classes. An acceptable output classifier for the convolutional net would be to
place the decision boundary arbitrarily close to the known examples, but this
doesn't make much sense given how large the semantic differences actually are.

There are multiple options to try to enhance the distance to the decision
boundary (such as the adversarial examples referenced in the post), but I
think the recent work from microsoft (I think it's published Dec 2015) on
replacing pooling layers in the convolutional neural net with something
analogous random forests might be the best option so far. More or less each
decision tree will end up pushing values to 0/1 in the non-linear region,
which mitigates some of the concern about overly linear systems and it places
the decision boundary at a somewhat arbitrary location between classes. In
aggregate when these locations are combined the resulting classifier has a
larger margin without explicitly needing adversaries. So instead of sampling
the image space, you're effectively sampling the classifier space.

If you want, I can dig up the citation, but searching for deep neural nets and
random forests should get you to the paper all the same.

~~~
mziel
For the lazy:

[http://research.microsoft.com/pubs/255952/ICCV15_DeepNDF_mai...](http://research.microsoft.com/pubs/255952/ICCV15_DeepNDF_main.pdf)

From abstract:

 _" (...) we introduce a stochastic and differentiable decision tree model,
which steers the representation learning usually conducted in the initial
layers of a (deep) convolutional network. (...)"_

------
chestervonwinch
I'm not sure if this provides any insight, but I think it's interesting to
visualize this type of thing for the much simpler case of a two-class problem
in two dimensions:

[http://i.imgur.com/NdzdH5j.png](http://i.imgur.com/NdzdH5j.png)

On the left are the training vectors, color-coded by label; the background
color-codes the probability output by the learned network at each point in the
plane. On the right is the gradient field of the network's output
corresponding to the blue class. The gradient field shows, at any point, the
local direction of greatest increase toward the blue class.

~~~
eli_gottlieb
Interesting. So most of the training points for Blue were in the "bulge" near
where they were surrounded by the Red class, but at almost every point, the
network believes that moving further "deeper" into the Blue "zone" increases
the "blueness".

------
schiffern
I wonder if simulated saccading would address this problem?

Rank the image by entropy, and re-run the neural network centered on those
points with a radially-increasing gaussian blur applied to the subimage
(approximating the sensitivity of the human visual field).

------
blazespin
For some reason this reminds me of the man who mistook his wife for a hat.

[https://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_f...](https://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat)

~~~
spikels
Make me think of this and optical illusions.

[http://www.ritsumei.ac.jp/~akitaoka/index-e.html](http://www.ritsumei.ac.jp/~akitaoka/index-e.html)

------
guard-of-terra
Neural net can see some subtle pattern invisible to human eye, decide it is
very representative, and make a guess based on that.

Pre-processing seems to be an answer, normalize the image, blank out areas
obviously uninteresting to human eye, add dithering.

~~~
argonaut
Define "uninteresting."

Also, it kind of defeats the purpose of neural networks to do substantial
feature engineering like that.

~~~
guard-of-terra
Something human can't see is uninteresting. Fractional values of pixel color
is one thing.

Our human eyes have a lot of filters (hardware and software-based) before
recognition takes place.

~~~
orting
There is a lot of stuff the human eye cant see that is very interesting. One
of the challenges in medical imaging is getting accurate labelling of images.
For many labellings we see large inter- and intra-observer variability. We
have both the problem that humans see something that is not interesting and
miss something that is interesting.

I currently work on estimating emphysema extent in CT lung scans. Emphysema
can be very diffuse and it is not possible to label individual pixels, so
instead we try to learn the local emphysema pattern from a global label.
Neural networks are interesting for this problem because the learn the
features, but it is also a "problem" because the features might not make
physically sense, which could make it hard to transfer the model and convince
clinicians that they should use it.

~~~
guard-of-terra
For that kind of task, you might want to filter out other things.

We should just be realistic. We want to take real image, except it might be
tinkered with, and make neural net tell us what we see on it, except we also
want it to see what we can't see, and we want it to answer as accurate as
possible, except we also want short and definitive answer.

We also kind of want it to admit that image always contains more than one
thing, but kind of don't.

------
natch
Cool! So now I have this Python notebook running (I guess) on docker, and it
tells me:

    
    
        [I 08:37:21.591 NotebookApp] Writing notebook server cookie secret to /.local/share/jupyter/runtime/notebook_cookie_secret
        [I 08:37:21.757 NotebookApp] Serving notebooks from local directory: /neural-nets
        [I 08:37:21.758 NotebookApp] 0 active kernels 
        [I 08:37:21.759 NotebookApp] The IPython Notebook is running at: http://0.0.0.0:8888/
        [I 08:37:21.759 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    

What next? Should I be looking for a python notebook tutorial, or a docker
tutorial, or both?

The author lists some commands to run, but it isn't clear where to type those
commands. All I see is a terminal with the above output.

~~~
avian
I don't know anything about Docker, but to use iPython Notebook, you need to
open it in a browser. The command line starts a local web server. It says it
is running at "[http://0.0.0.0:8888/"](http://0.0.0.0:8888/") in your case
(which sounds invalid). Usually, Notebook opens the browser for you
automatically when it starts. I guess Docker interferes with that.

~~~
antsar
0.0.0.0 is valid. It means the server is listening on all available addresses
(as opposed to, say, only 127.0.0.1)

------
dnautics
this really suggests that while neural networks are good, they aren't doing
what we are doing when we classify images.

~~~
blazespin
Maybe, it's hard to say. I remember when my kids were really young they used
to see really random things in clouds and sand or whatever - just like a
neural net would.

~~~
kuschku
There’s even a name (and a subreddit) for the phenomenon:
[https://www.reddit.com/r/pareidolia](https://www.reddit.com/r/pareidolia)

It shows that a human brain is just a neural network with decades of training.

------
lordnacho
Quite interesting. Sounds like he's subtly modifying the images at the points
where the various classes are sensitive.

I don't think any humans are going to be fooled by this procedure, so what
algorithm do people use to classify? There must be other algos out there that
are more plausible as explanations for what humans (and animals) use?

~~~
eli_gottlieb
>I don't think any humans are going to be fooled by this procedure, so what
algorithm do people use to classify? There must be other algos out there that
are more plausible as explanations for what humans (and animals) use?

Personally, it seems very plausible to me that humans and animals use
generative modeling (which was behind the "human-level concept learning" paper
published this month) rather than discriminative (like typical neural
networks).

------
extrapolate
Great read! Time to troll some of my machine learning co-workers.

"Can you classify this image of a panda? Oh, curious, it thinks it is a
vulture..."

~~~
rrmm
There are tons of anecdotes i remember from AI classes about neural nets
learning something other than what you expected them to learn from your
training set.

For example, one story involved training a classifier to recognize an overhead
image with tanks vs without. It turns out it ended up learning which days were
sunny and which were overcast.

This sort of thing happens when training people from examples too: From the
mundane cases in school, to the AA587 crash in Queens NYC.

~~~
cbr
> ended up learning which days were sunny and which were overcast.

The earliest source I can find for this is
[https://neil.fraser.name/writing/tank/](https://neil.fraser.name/writing/tank/)
but it says it "might be apocryphal".

~~~
cbr
Actually, there's also the 1992 paper "What Artificial Experts Can and Cannot
Do" [1] which gives this anecdote and calls it a "legend".

[1] [http://www.jefftk.com/dreyfus92.pdf](http://www.jefftk.com/dreyfus92.pdf)

~~~
rrmm
Nice sleuthing! I don't even recall where I first heard this story myself. The
paper was an interesting read and still largely applicable even now in this
3rd or 4th coming of NN's.

~~~
cbr
Expanded this into a post: [http://www.jefftk.com/p/detecting-
tanks](http://www.jefftk.com/p/detecting-tanks)

------
awqrre
Could processing the image before and after adding a small blur to the image
detect this kind of thing?

~~~
sp332
You would lose information from the texture of the image that way. It's
already downscaled to 224 pixels square, so you'd really be restricting the
amount of data available to the function.

~~~
awqrre
of course you would loose some information but it is probably not enough to
mis-identify most objects..

------
Qantourisc
What happens if you retrain the network with these bugs ? (As in tell the
vulture is not a panda ?)

~~~
jvns
the network gets smarter! (you can read the paper to find out more =D)

------
pmalynin
[https://news.ycombinator.com/item?id=10742390](https://news.ycombinator.com/item?id=10742390)

------
ohblahitsme
Wow this post is fantastic! Great read Julia!

~~~
sundarurfriend
I should probably begin reading author names when starting to read articles -
knowing this was written by Julia Evans would have clued me in on how fun a
read this was going to be, and motivated me to cut down on the open-tab-
procrastination hours!

> So now we’ve seen the network do a correct thing, and we’ve seen it make an
> adorable mistake by accident (the queen is wearing a shower cap ).

This is such a Julia sentence, including the suddenly appropriate emoji, it
should probably have clued me in.

Edit: HN is breaking my heart with its anti-emoji stance, so (re)read the
article to see the quote in its full glory.

------
gcb0
This is a great example of why computer scientists get a bad rep.

zero understanding of something so simple, laid out as black magic. And a lot
more time spent fiddling with the thing as a toddler playing with a toy than
it would be required to read two or three articles that explained it
correctly.

~~~
jvns
which articles would you suggest?

