
Slight Street Sign Modifications Can Fool Machine Learning Algorithms - itcrowd
http://spectrum.ieee.org/cars-that-think/transportation/sensors/slight-street-sign-modifications-can-fool-machine-learning-algorithms
======
mannykannot
Whenever this sort of issue comes up, there are a number of responses pointing
out that humans can be fooled too, and I am guessing that the generally
unstated implication is that therefore it is not a big deal. Two important
differences that this point of view overlooks are that humans are usually able
to tell when they are not fully understanding their visual input, and they are
able to respond appropriately, which includes acting cautiously and taking
actions that will help resolve the uncertainty. Artificial visual systems that
have, at best, only a rudimentary understanding of what they are looking at,
are in no position to act this way, and can assign high confidence values to
what we would regard as ludicrous interpretations of the scene.

These things will be fixed in time. There is nothing to be gained by
pretending that they are not problems.

~~~
DannyBee
They aren't real problems because nobody smart is trying to directly hook up a
road sign classifier to a steering wheel. They are trying to build complex
systems where this is just one signal.

If they were trying to directly use this info, this would be the least of the
serious issues. For example, there are plenty of stop signs well hidden by
trees. Such a car would probably last about a mile in a non freeway setting.

A trivial model of "hey, does this make any sense appearing at this place on a
map" defeats this, etc.

Of course this stuff is it is infancy and can do dumb things. But this is just
isn't that big a deal.

Humans also act much worse than you imply. Look at the number of people who
accelerate into crashes instead of braking, etc. Again, doesn't mean AI should
get a pass, and humans are infinitely more complex, but we shouldn't be held
up as amazing at this either.

(That's actually the part that worries me. That we _aren 't_ good at it)

~~~
mannykannot
The fragility of AI decision-making in general absolutely is an issue. Feel
free to post a submission when it has been solved to a level that matches or
exceeds human capabilities.

~~~
rm999
I build AI/ML systems for a living and it's not as big of a deal as you think.
Production machine learning systems built by experienced teams will almost
always have business logic protections built-in to stop the machine from doing
something totally stupid. For example, an auto-bidder may have capped
velocities or auto-shutoff mechanisms (those examples of bidders spending
billions of dollars in minutes are exceedingly sloppy).

This is not unlike the human body's deeply-evolved reflexes - before the
neocortex can even process that a pan is boiling hot, the nervous system
bypasses the brain and and tells the hands to drop it.

In the case of driverless cars, DannyBee is absolutely correct: truly
driverless cars will not make decisions from a single data point. I have some
past experience with unmanned aerial vehicles and those things had insane
amounts of redundancy in almost every signal and decision making process;
driverless cars are more complex and will come decades later and will probably
take this even further.

~~~
oneshot908
As another such practitioner, why oh why would we ignore hard-learned
unambiguous signals and wisdom because we jumped on the machine learning
bandwagon? We embraced machine learning to detect and uncover the things we
don't see, not to replace the things we already know. There's no conflict here
whatsoever.

------
fennecfoxen
Related, but different: Autonomous Trap 001

[http://jamesbridle.com/works/autonomous-
trap-001](http://jamesbridle.com/works/autonomous-trap-001)

They used salt to construct a circle with a solid line on the inside ("do not
cross") and a dashed line on the outside ("come on in").

~~~
DanBC
Everyone makes it sound like this art project was tried with a real autonomous
car, but it hasn't been.

This is just a guy sprinkling salt around his regular car.

~~~
hellbanner
So that was just him driving in and not an autonomous car?
[https://vimeo.com/208642358](https://vimeo.com/208642358)

The car drives into the salt circle then stops.

~~~
mikeash
Yes. From the interview above:

Is this actually an autonomous car, or is it conceptual?

I don't actually have a self-driving car, unfortunately....

~~~
lima
We're talking about it on HN, so I'd say he did it exactly right :)

------
thedevil
In my state, there's a (fast) road that turns down under a bridge. From a
distance, it kind of looks like I'm going to crash into the bridge.

Even though I know the road wouldn't drive directly into the bridge, I slow
down a little and look carefully to make sure I'm actually not going to crash
into the bridge.

When my perception doesn't fit my internal model, I gather more data (look at
different parts of the bridge and what other cars are doing), or transform the
data (ie turn my head slightly and look at the bridge and road from different
angles)

Edit: Likewise, when someone's tone doesn't match their words, I gather more
data (look at their body language).

Have any researchers experimented with neural nets to do the same? I haven't
noticed any posts here about that.

~~~
moyix
There was a paper that claimed those sorts of rotations and movements can help
alleviate adversarial examples:

[https://arxiv.org/abs/1707.03501](https://arxiv.org/abs/1707.03501)

However, OpenAI quickly refuted it by creating adversarial examples that
continue to fool the classifier even when rotated, scaled, etc:

[https://blog.openai.com/robust-adversarial-
inputs/](https://blog.openai.com/robust-adversarial-inputs/)

So it looks like there's no "easy" way out here. Multiple types of sensors
_may_ help, but it seems likely that it will still be possible to construct
examples that fool network over all sensor inputs at once.

Ian Goodfellow and Nicolas Papernot have a good blog on machine learning
security issues. One relevant post on why this is such a hard problem:

[http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-...](http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-
attacking-machine-learning-is-easier-than-defending-it.html)

~~~
pishpash
Is it a big surprise that you can construct adversarial examples for
algorithms? Don't humans have the same class of problems with optical
illusions? And those are not even adversarial, just confusing.

If we constructed truly adversarial examples for human neurology, I bet they
would be equally insane.

~~~
blauditore
In some sense, yes, optical illusions are simlar to such "adversarial
examples". But if you think about it, any kind of image is somewhat delusive,
since we perceive it as whatever object it depicts while actually staring at a
piece of paper with some ink on it.

Also, adversarial in this case seems to refer to images perceived differently
by machines than by humans, so it's not really possible to create such ones
for humans.

~~~
kthejoker2
No, adversarial simply means deliberately trying to engineer false positives
and negatives. This can be done against humans, machines, ants, trees, viruses
...

~~~
blauditore
What is a false positive or false negative in this case? The "ground truth"
here is what human perceive.

~~~
kthejoker2
Optical illusion?

------
panic
The road environment is designed for people to see and understand. Without a
human-emulating "general AI", it's unlikely that every sign and surface
marking out there will be legible to a machine. IMO, the whole premise of
training these machines to see roads like a human is flawed. Their developers
should be working with government agencies (like the FHWA in the US) to create
new standards that are aware of the capabilities of the machine.

~~~
hacker_9
I agree and I think the whole self driving push should be incremental anyway:

1\. To begin with it is only used on motorways/highways as they are long
straight roads, where the AI can take over the boring part and the driver can
be left to take over if something complex happens.

2\. Shared AI maps across all cars that can route everyone to/from work. As
this is shared, it can balance the load and result in the least amount of
traffic jams. Humans still do the driving.

3\. The government sets up AI cameras that monitor city routes, and cars can
tap into this information as they drive. The benefit here would be seeing the
road from all angles, as well as massive computers bigger than cars doing all
the number crunching. Over time more and more city routes could become
approved to be fully driven by AI.

That's my approach to an incremental design at least.

------
geophile
This article says that the training set was small. And presumably the
misclassified images were unlike images in the training set. If the training
set includes images intended to mislead, then wouldn't the classifier then be
more tolerant?

------
mishraka
This is the price we all pay for lazily defining a complex cognitive task as a
mere image <-> label problem.

------
Rjevski
The right solution to this is to have an official database of signs and their
GPS coordinates provided by the government or whoever is responsible for road
safety, free of charge (you're not directly paying for having signs on the
road, why would you have to pay for an electronic version of that?).

Road signs were made for humans because we don't have the ability to connect
to the internet and fetch the data in less than a second, but autonomous card
do, so why not use it?

~~~
castis
Introducing a single point of failure (1 database, across the internet),
organized by a largely complex system (the DoT, possibly) to cope with an edge
case doesn't feel like a very elegant solution.

~~~
pimmen
A more costly, less elegant solution whose problem is already shared by street
signs in general (meaning we could plan for it) would be RFID tags or
something that tells the computer what sign this is if it can't read the sign.
You could also use this for training, so it learns to filter away poorly drawn
swastikas from the sign.

An attack on something like this would scale very poorly, as you would need
physical access to all street signs. Issuing RFID to a street sign would be
just another step along the manufacturing process of the sign, or as a step to
the mounting of the sign.

I can't take credit for this idea though, it's already been [explored in a
paper]([https://link.springer.com/chapter/10.1007/978-3-642-41647-7_...](https://link.springer.com/chapter/10.1007/978-3-642-41647-7_15)).

~~~
tyingq
>An attack on something like this would scale very poorly, as you would need
physical access to all street signs

I wouldn't underestimate the eventual legions of unemployed truck drivers.

They will probably lose eventually, but I expect a spirited effort.

~~~
pimmen
They would be most successful where population and sign density is high, which
would be the cities. During such an attack, public transport could move around
people while the authorities would take care of the attackers.

The problem for them is that where they would cause the most harm, rural areas
(since they are in most dire need of supplies a few towns away and don't have
any good alternatives to cars), is where such an attack would be the hardest
to implement. Canada, Australia, Iceland and Alaska has many roads where vital
street signs can be tens of miles apart from each other, as well any actual
people who might be effected by this. Also, demographic movements is working
to their disadvantage; more and more people everyday are moving to large
cities.

~~~
tyingq
In the US they can focus on just the interstates. And, they are unemployed,
already used to long boring trips, and have established communication networks
between them. Oh, and disenfranchised friends at the various rural truck stops
and motels that will also be razed by self driving tech.

------
pishpash
If you had a diversity of differently trained algorithms, they would not admit
the same sorts of adversarial examples. The risk isn't that adversarial
examples exist -- an exponentially small number of them always exist with any
representation. The risk is that you can search for them en masse like offline
cracking of passwords. If you could do that with the human mind, I hesitate to
think what you would find.

~~~
brownbat
Cognitive biases, optical illusions probably.

------
infruset
Couldn't slightly randomizing the image before inputting it to the neural
network invalidate such manipulations?

------
ragebol
Yet humans are totally unphased by these modifications. What makes humans
still excel at these edge cases?

~~~
acomjean
Our brains are pretty good at image processing. Plus we have an inherent
understanding the world, including the way objects rotate and change shape.
We're looking for the stop sign (big red hexegon).

Though I've seen a new stop sign added recently, and the number of cars that
don't see it is remarkable.

I'm not an expert on AI though ( a few classed at university and I specialized
in something else).

Its the general classifiers that give us trouble. We want to show it 100s of
stop sign pictures and have it figure out when we show it new one. But we're
not asking it, is this a stop sign, we're asking "what is this".

We can write software that probably is good at finding hexagons and colors and
thus stop signs. Take facial recognition, its remarkably good at this point,
but its doing one thing ( though I've seen a computer id 3 people in a photo
with 2 in it , because a third person's photo was in the background.)

~~~
userbinator
_We 're looking for the stop sign (big red hexegon)._

 _Though I 've seen a new stop sign added recently, and the number of cars
that don't see it is remarkable._

These two sentences together show that our brains aren't just looking for the
signs; we're also looking at many other aspects of the situation and even
taking into account past experience (e.g. is this an intersection? Have I seen
a stop sign here before? If I'm new to this area, I'm likely going to be far
more alert to the signage.)

If someone planted a (non-modified) stop sign on the side of a highway, where
the road is completely straight and with no intersection, I bet some drivers
won't even see it, those who do will be puzzled, and approximately none of
them will even try to stop.

------
hellbanner
[https://news.ycombinator.com/item?id=14883900](https://news.ycombinator.com/item?id=14883900)

"We'll just cover all of the possible use cases" \- Self Driving Car Engineer

------
craigyk
The first time I learned of adversarial research my initial thought was GD, in
our automated AI-driven future, the people who master this are going to live
like wizards.

~~~
resu_nimda
You might really like [https://cvdazzle.com/](https://cvdazzle.com/), they at
least look the part!

------
nine_k
What makes me wonder is how slight modifications make the algorithm miss
gross, highly visible features, e.g mistake a blue sign for a red sign, or an
upward-pointing triangle for a downward-pointing. I suspect it won't be very
hard to make the algorithm pay more attention to it, by specially teaching it
to tell between such differences, and maybe by running several separate
networks taught to tell apart particular narrow features, not complete signs.

~~~
s_kilk
It seems to indicate that these machines are nowhere near as smart as they
appear to be based on earlier successes. I'm starting to get worried about the
possibility of another AI Winter if it turns out that reality and hype are too
far apart.

~~~
jgalt212
The AI Winter will come only when the investment money runs out. As the Fed is
keen to raise rates as slow as humanly possible (similar to 2004, and unlike
1994), I think the AI practitioners still have a quite a bit of runway left.

------
JackFr
I would also hope that the car uses contextual geolocation info. That a speed
limit sign is not posted on the corner of a 4-way intersection, and a stop
sign is not typically put on the side of a limited access highway. In fact I
would expect that most of the driving regulations should be encoded in the
map. Anomalies would be treated with extreme caution (and reported back to the
home office).

~~~
tomelders
I would expect that some of the more basic rules win out. e.g. Don't crash
into another vehicle. Do hit a pedestrian etc etc.

------
smoyer
Because CCD based cameras see so much more IR than we do, you could probably
make a set of IR reflective/absorptive stickers that could be placed on street
signs that would be virtually invisible to humans but totally alter what the
CCD sees.

OT: My favorite street sign graffiti is the hula-hoop stickers that artists
put on pedestrian walking signs.

------
drawkbox
I have wondered this about mirrors and LIDAR, sounds like it is being worked
on [1]

[1]
[http://ieeexplore.ieee.org/document/5409636/](http://ieeexplore.ieee.org/document/5409636/)

------
pteredactyl
I don't understand the use of 'adverserial attack.' Makes it sounds like the
machines, or the creators of the machines are at war. When in reality a stop
sign sticker adds texture to a mostly mundane public realm.

~~~
pishpash
Adversarial is a technical term.

------
yters
Give a general machine learning algorithm a long enough string of 1s with a
few missing, and it will have no clue what fills the missing digits. ML is
inherently incapable of matching human cognition.

------
kumarvvr
Sometimes it seems that NN systems and Deep learning systems are a dead end.

Are there any other promising technologies that can replace or at-least
augment current Machine Learning systems?

------
pacaro
I would assume that you could somewhat trivially defend against this by
validating the top n hypotheses from your classifier against reference images.

------
thanatropism
I want to know now how I can paint my face slightly to fool Facebook's (or
some other widespread) facial recognition system.

~~~
userbinator
Perhaps not "slightly" but there's this:
[https://cvdazzle.com/](https://cvdazzle.com/)

To other humans, those are very much recognisable humans, but face detectors
won't think they are.

------
whipoodle
Sorry I'm late, boss. Someone put a sticker on a street sign so my car drove
into a bakery.

------
roryisok
The first thing my mind went to was the image of muttley putting up a detour
sign

------
ape4
Some context knowledge would help. For example, approaching an intersection...
should I stop here? Oh, yah, there is a distorted red sign.

Also, figuring out the type of sign for the outline. Then the icon inside
seems like an approach that could work.

------
windlessstorm
Future humans only street.

------
dilemma
Sounds like Artificial Intelligence is Stupid.

~~~
visarga
It's not fair to compare with AI. In this case, a visual network is more like
a reflex. They can fool the equivalent of a human reflex. The vision neural
net feeds into a "world model" where such inconsistencies are resolved on a
more abstract level. The same world model is being used to plan the path of
the car. Even if the vision net makes an error, the internal model has ways to
detect that there's a perception error if it doesn't make sense in the
context.

------
bitL
Alright, so this is an image augmentation problem. Another training set with
white noise variations, random unrelated pixellation/overlay texts can solve
this. Simply your training wasn't general enough.

~~~
visarga
Constructing adversarial examples is a sophisticated task, but it's being
done.

~~~
bitL
Sure, and it should go hand in hand with appropriate image augmentation. Maybe
make something like a GAN that would try to generate anti-obfuscation image
augmentation for every new adversarial example...?

Our eyes can be fooled easily anyway.

------
jameshart
Slight street sign modifications can completely fool humans too. Luckily: 1)
most people aren't assholes so don't want to cause traffic accidents; and 2)
we have laws and police organizations to track down and punish assholes who do

~~~
dogma1138
Slight sign modification will not fool humans at least not on this manner.

This looks like a couple of bumper stickers will cause problems even when they
do not obstruct the actual sign.

This is a pretty big issue that will have to be dealt with.

~~~
jameshart
Except that in a world where self-driving vehicles are relying on visual
recognition of signage, it would become obvious PRETTY QUICKLY if the
modifications to a STOP sign made it look like a 65mph speedlimit, and it
would be able to be treated exactly as severely as if someone had covered the
stop sign with a 65mph speedlimit sign.

We're not talking about modifications causing a sign to accidentally be
mistaken for something else, we are talking about deliberate modifications to
road signs that cause vehicles to misinterpret them.

If you modified a sign so that most users still perceived it as a stop sign
but colorblind people misread it as a speed limit you'd be doing exactly the
same kind of thing.

I mean, it's not like there's some free-speech right to graffiti on road
signage in the first place, let alone to modify it so that some road users
will misunderstand the sign's meaning. If you interfere with a roadsign in
order to deliberately confuse road users, you are a criminal.

------
cs702
IN OTHER NEWS:

Images with Zero Modifications Can Completely Fool Human Sight!

Examples:
[http://www.ritsumei.ac.jp/~akitaoka/index-e.html](http://www.ritsumei.ac.jp/~akitaoka/index-e.html)

(In case it's not obvious, the implication is that it's possible to engineer
sign street modifications that fool human beings too.)

~~~
dogma1138
And there is a reason why we design signs the way we do and not use optical
illusions.

~~~
djmips
This is about fooling vision systems with adversarial modifications.

Everyone needs to get out of themselves and see that the human vision systems
can be fooled with adversarial mods as well, just not the same bugs as
computer vision...

