
Adversarial Examples That Fool Both Computer Vision and Time-Limited Humans - networked
https://arxiv.org/abs/1802.08195
======
rahimnathwani
The interesting data are in "Table Supp. 1" at the bottom of page 14.

I opened the paper hoping to see some examples of images that look to me like
one thing on first glance, and something else on closer inspection. The best
image is the one with the spider on a blurred-snake background, and that's not
going to trick anyone who looks at it for more than a second.

The humans were shown each image for either 63ms or 71ms. That's 1-2 frames of
a movie. So whilst the result is important, it's not as surprising as you
might expect.

~~~
gxigzigxigxi
To be fair to the computers, that’s about the speed we train image classifiers
to run at, if not faster. It seems likely that if we were willing to tolerate
a second or two of latency we might be able to devise architectures that are
able to see through some of the images that confuse current architectures.

~~~
lisper
The problem is not that we're not willing to tolerate latency. The problem is
that the model of how neural networks are trained is completely different from
how humans learn to see. When a neural net is trained, it is shown a static
image, weights are tweaked until the output is correct, and then it is shown a
_completely different_ static image and the process is repeated. Neural net
learning is iterative, discrete, and supervised. Human visual learning, by
contrast, is continuous and largely unsupervised. We don't see snapshots, we
see continuously varying images. Furthermore, we actively interact with the
world by manipulating objects and shifting our gaze, and that information is
also incorporated into our visual learning. Finally, humans have very advanced
feature detectors built in to our brains by evolution. We don't learn to see
cats, we have cat-detectors built in to our brains by our DNA, which learned
to detect cats because that was a useful skill in our ancestral environment,
when cats were a lot bigger and could eat us. We do learn that the thing that
our cat-detectors detect is called a "cat", but we don't "learn" what a cat
(or a human) _looks like_. That's built in to our brain wiring. (There are
some things that we do learn what they look like, like cars, which obviously
didn't exist in the ancestral environment. That's why all humans can tell the
difference between a cat and a dog, but not everyone can identify whether
something is a Honda or a Toyota.)

The point is: the process that humans go through when they learn is completely
different than the process that contemporary neural nets go through. No one
has yet come up with a theory that combines all of the features of human
learning into an implementable algorithm. It will surely happen eventually,
but there are at least a few more conceptual breakthroughs that will need to
happen. Minor tweaks to back-propagation won't do it.

~~~
Game_Ender
Do you have a citation on humans having from birth classifieds for things like
cats?

~~~
theoh
I think the GP didn't mean to imply that humans have an innate cat
representation from birth.

It makes more sense to interpret the comment as saying that humans don't learn
an internal _image_ representation. Humans _do_ learn representations of
bridges, aircraft, cats, etc. But those are built on top of an image
processing/representation system that we are born with, analogous to raster
graphics?

Edit: Maybe I'm misreading the comment. What's definitely built in at birth is
things like edge and orientation detectors. A zebra detector would be a
surprise.

~~~
lisper
> I think the GP didn't mean to imply that humans have an innate cat
> representation from birth.

Actually, that is what I meant, though I don't have a reference for cat
detectors per se. But there is ample evidence for innate feature detectors of
comparable complexity (e.g. human facial expressions), even if the actual
target is something other than cats.

------
zkms
Speaking of similarity between CV and human visual systems; it was kinda
striking to me just how psychedelic/"trippy" the "universal adversarial
perturbations" (see page 5 in
[https://arxiv.org/pdf/1610.08401.pdf](https://arxiv.org/pdf/1610.08401.pdf))
look.

Indeed, the look of visual hallucination "form constants" is intimately
related to the machinery in the visual cortex that detects
edges/contours/surfaces:
[https://www.math.uh.edu/~dynamics/reprints/papers/nc.pdf](https://www.math.uh.edu/~dynamics/reprints/papers/nc.pdf)
and such visual hallucinations can be elicited without any drugs or
physiological interventions/defects -- just via diffuse flickering light:
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3182860/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3182860/)

~~~
shoo
There's a video demo of the first paper here:
[https://www.youtube.com/watch?v=jhOu5yhe0rc](https://www.youtube.com/watch?v=jhOu5yhe0rc)

i was trying to build a bit of intuition about this paper by considering a
trivial case where there are two classes, and the classifier is linear, in
very low dimension. consider the following trivial example:

[https://en.wikipedia.org/wiki/Linear_classifier#/media/File:...](https://en.wikipedia.org/wiki/Linear_classifier#/media/File:Svm_separating_hyperplanes.png)

in this example, the decision boundary from different classifiers is shown as
H_1, H_2, H_3. The "universal" perturbation for each of the three classifiers
would be a small vector normal to the each classifier's decision boundary.
This paper defines "universal" perturbation with respect to the choice of
input from the population of inputs, but each "universal" perturbation is
optimised specifically to target a single model (aka classifier).

both H_1 and H_2 do a reasonable job of separating the two classes, but the
H_1 decision boundary with the smaller margin is more vulnerable to
misclassification if inputs are perturbed by a small vector normal to its
decision boundary.

You can imagine translating the input space a bit to the right -- this would
result in H_1 misclassifying say 2 out of the 18 data points shown, whereas
H_2 (the SVM generated decision boundary with maximal margin) classifier would
not experience any errors.

------
dalbasal
Interesting. Anyone in the space have some meta/context to share?

I may have this wrong but it _seems_ the authors are very interested in a
particular subset of classification errors common to machines and humans.

What is that subsets of "transferable" mistakes are trying to find? What does
the existence of a particular subset of "adversarial example" (sneakily
doctored images) tell us either about human or machine brains?

~~~
mannykannot
This is what the authors say: "A rigorous investigation of the above question
creates an opportunity both for machine learning to gain knowledge from
neuroscience, and for neuroscience to gain knowledge from machine learning."

------
dontreact
Does this mean we can stop seeing adversarial examples as a deep fundamental
flaw of deep neural networks? Seems like human system experiences them as
well!

~~~
saagarjha
Aren't optical illusions just adversarial examples for people?

~~~
aetherson
No, and I wish people would stop that meme.

Humans can obviously be tricked, in a variety of ways. But adversarial images
take advantage of the fact that image-recognizing neural networks do not fit
their image recognition into a full fledged understanding of the world like we
do. So a few pixels here and there can make a truck look like a panda and the
algorithm never says, "But wait, pandas are mostly black and white and this is
mostly yellow," or, "But I don't see legs anywhere, or ears."

Optical illusions mostly don't cause high level image misclassifications. To
the extent that they are anything similar, they're the reverse: using our
general world understanding to cause glitches in our information processing,
such as cases where you think something is darker or lighter than it is, or
bigger or smaller, or bent or straight. Those are your mind applying rules
that are based on "how the world usually appears at a high level" to an image
where those rules do not in fact apply.

~~~
saagarjha
I’m no machine learning expert, but it seems to me that neural networks just
don’t really work like people do, as much as people would like to claim that
they’ve created something that works like the human brain.

~~~
aetherson
Sure. So, then, why did you posit a direct parallel between optical illusions
and adversarial images?

~~~
saagarjha
Because that’s another thing I can now mention when people tell me neural
networks are like the human brain, since this is an example they often like to
bring up.

------
ardy42
I'm looking forward to when these are integrated into Recaptcha...

------
tomcam
Aren’t we all time-limited humans?

~~~
aneeqdhk
On a long enough timeline the survival rate for everyone drops to zero.

