
Robust Physical-World Attacks on Machine Learning Models - earlenceferns
https://arxiv.org/abs/1707.08945
======
swordswinger12
This paper is a fairly convincing counter-argument to another recent work
([https://arxiv.org/abs/1707.03501](https://arxiv.org/abs/1707.03501)) on
physical adversarial examples for autonomous vehicles. The other paper argued
there was "no need to worry" about such physical attacks.

~~~
hellbanner
In Design of Future Things, ex-Apple UX designer Don Norman interviews
(anonymous) engineers of a self-driving car:

DN: "It sped up when we left the freeway instead of slowing down on the exit,
that was dangerous. Why did that happen?"

ENG: "Car saw it was on a straightaway. We'll add a rule to handle that."

DN: "So you have to add rules for every possible situation? Doesn't that mean
that the car is always at risk for what it doesn't know about yet?"

ENG: "That's Not-A-Problem. We will classify everything."

~~~
jacquesm
That's an incredibly stupid answer. It is precisely this kind of thinking that
makes me worry about sharing the road with alpha grade self driving hardware.
There is real potential for carnage here, at highway speeds it doesn't take
that big of a software bug to get a lot of people killed.

------
mynameisvinn124
i understand why we'd like to have near-perfect classification but do we need
100% accuracy in practice?

for example, the paper identifies an instance where a stop sign was
misclassified as a speed limit under various conditions.

however, wouldnt applying bayesian priors ("how often do i see speed limits at
intersections?") presumably negate misclassifications?

rather than chase the elusive ~100% accuracy from a single model, why not take
a layered, ensemble approach?

~~~
throwawayjava
I think the traffic sign example was a nice setting for explaining the
research. But of course really good examples always risk interfering with
communication as much as they assist communication. I think that might've
happened here.

Concretely, I have a hard time believing that adversarial traffic signs will
be the Achilles heel of self-driving cars. Vandalizing signs is already
illegal, and if doing so causes or was intended to cause a crash, law
enforcement will throw the book at the culprit. And this already happens to
human drivers. See e.g., [http://globegazette.com/news/local/prank-turns-to-
peril-vand...](http://globegazette.com/news/local/prank-turns-to-peril-
vandalism-called-likely-cause-of-
serious/article_ae3c15e9-f3c0-5b00-a21d-2804a4d08806.html) or
[http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-
st...](http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-stop-sign)

Besides which, you can also just rip the sign out of the ground or replace the
stop sign with an actual speed limit sign. From a SDC perspective, this is
kind of the equivalent of using super clever software vulnerabilities when you
have physical access to an un-encrypted HDD. Impressive technical tour de
force, but misses the forest for the trees if your goal is security.

Traffic sign vandalism is a social problem with a social solution. The first
person who tries something like this in production with malicious intent
toward SDCs will be charged with some variation of attempted or actual
manslaughter/murder and opponents of SDCs will switch to strategies other than
domestic terrorism to advance their agenda.

This research is still very interesting and useful, of course, but primarily
for other reasons. I just don't see traffic sign vandalism intended to confuse
DNNs as a serious -- or unique -- threat to self-driving cars.

~~~
pessimizer
> Traffic sign vandalism is a social problem with a social solution. The first
> person who tries something like this in production with malicious intent
> toward SDCs will be charged with some variation of attempted or actual
> manslaughter/murder and opponents of SDCs will switch to strategies other
> than domestic terrorism to advance their agenda.

It's tunnelvision to assume that the only attackers would be individual
radical luddites. There do exist terrorist organizations and state actors who
intend to run risks to do damage. And unless you're planning to monitor and
respond quickly to all potential threats to all stop signs at all times (and
all other possible attacks), what's to stop a group of 40 terrorists from
covering their faces, stealing motorcycles, and putting 10,000 stickers on
10,000 signs in 10,000 intersections in 20 different cities in 1 night?

~~~
throwawayjava
What's to stop 40 terrorists from covering their faces, stealing motorcycles,
and clipping brake cables (whatever you get the point...) on 10,000 cars
parked along 10,000 streets in 20 different cities in 1 night?

...Or just blowing up a shitload of random houses or whatever.

Basically, the exact same thing as what would stop your scenario. Is it
perfect? Even good? Again, no. But striving for good security in a public
setting is how you get TSA.

If there are 40 committed and sophisticated terrorists willing to act, there
are a _lot_ of easier targets...

Again, I think the "clever localhost software vuln" analogy is apt. It's not
that this isn't one possible attack vector. And no attack vector should be
ignored. But... really?

