
AI Has a Hallucination Problem That's Proving Tough to Fix - lnguyen
https://www.wired.com/story/ai-has-a-hallucination-problem-thats-proving-tough-to-fix/
======
kator
One challenge I see in AI is that humans tend to attribute human traits to
non-human entities (Anthropomorphism). This tendency leads people to expect
something from AI that they can do quickly and to be shocked when we find AI
to be brittle and lacking fundamental features of human understanding.

Having raised children and now playing with my grandchildren, it often amazes
me at just how much we take for granted in that comes "out of the box" with a
human brain. Humans can build associations with very few samples, and we come
pre-wired with all sorts of tools from primary systems like hearing, visual,
sensory systems to complex capabilities like speech and communication
abilities.

I've worked with computers for 37 years, and the progression has been amazing,
but we're still a long ways away from the primary capabilities of even a house
cat.

All this said I often wonder if the reason for the failure of current AI
systems to wow us is the gap between power density of a human brain vs.
compute systems. I've heard it said that DeepMind burned an order of magnitude
more than the 20W/hr a typical human brain uses. When we have compute systems
with that power density, we may see more emergent behaviors from our silicon-
based friends.

Either way, I think AI has useful applications today and I hope we will find
many useful applications of these technologies to make our lives better and
make more time for us to learn, love and care for one another.

~~~
YeGoblynQueenne
>> This tendency leads people to expect something from AI that they can do
quickly and to be shocked when we find AI to be brittle and lacking
fundamental features of human understanding.

Just to be clear- "AI" ≠ classification and even more so, AI ≠ machine
learning ≠ deep neural nets. The article above runs fast and loose with the
terminology, but machine vision and in particular object detection (or
classification of objects in images) is one area and one sub-task of AI in
general. It happens to be one of the two or three areas that have seen strong
empirical results in recent years, but it's by no means the only active area
of research (although, thanks to the funding from large technology companies,
it is probably the fastest growing one).

It's also very strange to see "brittleness" as a criticism of deep neural
networks in particular (the attacks described only work on convolutional
neural nets as far as I know). In the past, the type of AI system criticised
as "brittle" was the hand-crafted rule-based expert system type of AI. And the
reason why that kind of AI was criticised as brittle is because it did not
deal very well with the noise in real-world domains, such as in photography or
speech etc. Deep neural nets in particular are extremely robust to noise,
which is why they work so well in speech and image processing.

I think what you really mean by "brittle" is the tendency of deep nets to be,
well, a little too good in dealing with noise. Specifically, they have a
tendency to overfit to the noise, because they produce models with very high
variance. Indeed, the whole adversarial examples thing is probably best
understood as a result of overfitting.

As to "lacking fundamental features of human understanding" what you mean, I
think, is that image classifiers only do classification and nothing more-
which is true, but then that's what they 're designed to do. Nobody expects an
image classifier to have any understanding of the images it's classifying.

~~~
Eridrus
> (the attacks described only work on convolutional neural nets as far as I
> know).

These attacks work on basically every classifier. It works for multi-layer
perceptrons, it works for linear models, it works for random forests, it works
for SVMs. It works on basically anything that tries to generalize far from the
points it is trained on.

There are a lot of super interesting results in this area, but it's practical
significance is waaaaaay over-hyped. One cool thing I saw recently was that
some researchers made a physical turtle statue with a texture on it that
fooled ML systems into thinking it was a gun, with I think the implication
that you could make an AI system think a gun was a turtle, but you could also
just put a gun in a turtle case, which fools humans as well as AI systems.

~~~
JimmyAustin
I think part of the implication was the opposite, a turtle that was registered
by AI systems as a gun, potentially triggering a automatic “swatting”
response.

~~~
Eridrus
Out of all the ways to mess with someone, this seems pretty complicated and
unreliable.

------
_dps
I really wish the press (and a subset of highly vocal Deep Learning
practitioners) would stop conflating Deep Networks with the entirety of AI/ML.
This hallucination problem is

1) highly pronounced in this one class of methods that has surged in
popularity in the last 10 years

2) difficult to address because these methods are (so far) quite opaque to
human understanding

I work on multiple in-production vision systems and in cases where we
absolutely need to know why something went wrong we use much more
conventional, but more transparent, learning algorithms. The performance loss
is often an acceptable tradeoff for being able to understand your edge cases.

------
falcolas
"AI hallucination" and its negative effects is not limited to self-driving
cars... for example, see the latest incident with Alexa "hallucinating" a
command to laugh. What makes that scary to me, is Alexa also had to
"hallucinate" the trigger words as well. How many times does Alexa
"hallucinate" those trigger words and send random conversations off to a third
party?

------
resource0x
For the sake of argument, suppose someone hacked your program by causing
buffer overflow. Then at least you know where the problem is, you go fix it
and ship a patch the next day. Now suppose someone hacked your AI by causing
hallucination. Do you know where the problem is? How to fix it? How long will
it take to fix it? Does the fix really fix it? Etc... Not sure how this all is
supposed to work.

~~~
wepple
Two points:

\- you suggest that software is easy and logical, in that you can triage,
locate, and patch the buffer overflow. But in any sufficiently large codebase,
there are simply near-unlimited more bugs to find: hence all the memory
mitigation’s that have been implemented. So it’s not a robust comparison from
the get go.

\- there’s constantly increasing research into being able to diagnose &
analyze what CNN’s “see”: possibly not to the level and accuracy that you’d
expect from lldb, but how long did it take to have really amazing debuggers
for binary Applications?

~~~
resource0x
In a normal program, given an exploit, it's easy to understand the idea of
attack and specific target of attack. In CNN case, because no one really
understands how things work (see
[https://www.technologyreview.com/s/604087/the-dark-secret-
at...](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-
of-ai/)), how can they understand why things don't work? The whole contraption
is a black box.

~~~
Too
What about the latest Intel bug Meltdown? It's not always easy to understand
such attacks. Also those involving rare race conditions or things that only
worked because a core part of your code depended on undefined behavior can
require a major refactoring of your whole code base in order to mitigate.

------
gdubs
Interestingly, the evolution of the human mind has dealt with similar
challenges.

A lot of what AI is doing (pattern matching, creative problem solving, etc)
could be considered “right brain” activities. Some think that many human
mental problems arise out of an unchecked, overactive right brain. We still
mistake shadows on the wall for something sinister, or read more into a
person’s glance than is really there.

Some posit that the right brain is always hallucinating, in a sense; that
psychedelic drugs simply disable the left brain, and allow the right brain to
take center stage. Until the corpus callosum developed (which allows the left
and right brain to send data back and forth across the divide), it’s possible
that right brain insights came to us as “voices” in our heads.

See: “Incognito: the secret lives of the brain.”

“The Dragons of Eden: speculations on the evolution of human intelligence.”

~~~
ohtwenty
This sounds like a bunch of pseudoscience? I've taken a bunch of courses on
neurosciences etc, and there's certainly some left/right splits but nothing as
severe as that…

The wiki article is actually pretty accurate:
[https://en.m.wikipedia.org/wiki/Lateralization_of_brain_func...](https://en.m.wikipedia.org/wiki/Lateralization_of_brain_function)

~~~
gdubs
Probably more like outdated conjecture than pseudoscience (Sagan’s book,
“Dragons of Eden”, was written in the 1970s), but point taken — contemporary
neuroscience talks more about a modular theory of the mind than a strict Right
/ Left split.

~~~
cicero
It sounds to me like a desperate attempt to explain away religious experience.

------
tptacek
I don't have anything insightful to say about this (I think it's a super
interesting area of research) but am commenting anyways to point out that
Nicholas Carlini, one of the researchers cited, is also responsible for
engineering the best of the later Microcorruption levels. Small world!

------
sologoub
Actual audio from the “evil dot com” example can be found here:
[https://nicholas.carlini.com/code/audio_adversarial_examples...](https://nicholas.carlini.com/code/audio_adversarial_examples/)

It’s only hard to hear on a phone speaker at low volume on the 50dB example.
All other examples have what sounds like some sort of static or background
noise.

Granted, to an average consumer it may sound like just bad audio, but it’s not
imperceptible and thus can be screened for.

Most likely we’ll end up in the same perpetual update cycle as other computer
security - someone finds an exploit, that exploit is either reported for a
bounty or discovered in active use in the wild, a fix is implemented and
exploit is added to the test suite.

For ML that will mean adding the examples of the exploits into the learning
sets/providing negative re-enforcement feedbacks.

The question will be whether we can get enough of these caught before some
machinery injures someone because it saw something that wasn’t there or
accepted a malicious command, and it becomes a media frenzy.

------
abhishek0318
If we are to mass deploy self driving cars, we must solve this problem. No one
would want to travel by cars that could be tricked so easily.

This also leads to a moral and legal questions. If a self driving car injures
someone, who will be responsible, the person owning the car or the
manufacturer?

~~~
NewEntryHN
There's no need to completely solve it. It just needs to happen statistically
less often than human failures. And I believe this won't be hard.

~~~
argonaut
Well currently the attacks work 100% of the time.

------
starchild_3001
This is a highly biased article, imo.

i) The authors and commenters of the article have a strong incentive to get
funding for their projects.

ii) The mentioned problems are mostly academic. The real-world implications
are not tested, unknown, and likely overblown. Yes, one can create contrived
examples. But that isn't same as real life.

The way spoofing examples are created is by feeding small perturbations of the
same image to a time-invariant detector. It's unclear if such attacks have
much practical value. E.g. a password screen won't allow you to try more than
3-5 values. Attacks must be detectable (due to repeated tries of small
perturbations). Plus noise and other perturbations can be added to the
detector.

------
visarga
When the problem of adversarial examples will be solved, AI will have leapt
one more step ahead. It's clear right now that adversarial examples are the
elephant in the room. It's a make or break situation for the field but I think
it will come out stronger. We're grasping at the limits of our neural network
technology, trying to discriminate between ghosts of perceptions.

I think the solution will come from marrying a top-down approach to the
bottom-up one we're using right now. We need more prior knowledge about the
world. We need to be able to simulate situations and understand their effects.
Maybe what we're lacking right now is a mental simulator of the world, an
imagination module. Coupling perception with imagination would reduce the
sample complexity as well.

Interesting to note that the researchers that started the adversarial examples
craze and invented the imagination module (GAN - generative adversarial
network) are one and the same - Ian Goodfellow. He was right on the spot to
identify the weak point of deep learning.

------
dwaltrip
> Making subtle changes to images, text, or audio can fool these systems into
> perceiving things that aren’t there.

This isn't a hallucination problem. It is a robustness issue.

The outputs of modern AI lack conceptual depth and substance. These algorithms
produce very shallow categorizations that are only useful in narrow,
constrained contexts. Not surprisingly, it isn't hard to break or hack these
fragile categorizations.

Sure, one could potentially argue that this has some similarity to human
hallucinations, but I think that is a needless distraction. We know with great
certainty that our AI techniques don't have the robustness and generality of
animal intelligence. We are much less certain about the causes of human mental
illness and any resulting hallucinations, so that analogy doesn't really lead
us in a productive direction.

------
stochastic_monk
I think that DeepXplore [0] is on the right track. I think that perhaps some
methodological improvements could likely be made, but applying an adversarial,
security-like approach to deep neural networks is a way to build fault-
tolerance into these methods.

I'll keep hoping that things like capsules and "smarter" network
design/training and data augmentation will eventually, de novo, help add
"safe" generalizability. Perhaps optimizing accuracy/minimizing loss in a more
broadly, uniformly random way rather than in optimizing accuracy, weighted by
frequency of observations.

[0]: [https://arxiv.org/abs/1705.06640](https://arxiv.org/abs/1705.06640)

------
jeffdavis
"The vision systems of autonomous vehicles, voice assistants able to spend
money, and machine learning systems filtering unsavory content online all need
to be trustworthy."

I feel like we're building up to some horrible situation here, because these
systems are never going to be worthy of trust in the same way that a human is.

As just one example: as more and more of our lives happen digitally, more
evidence of real crimes will be digital evidence. So these AI systems can be
easily fooled into thinking you were in the wrong place or requested something
that you didn't. And there is absolutely no way to correct the record.

~~~
skybrian
It seems likely this would be handed the same way security cameras do it
today, by saving the original images and having humans look at them.

You might want to look into how a "chain of custody" is handed by the courts.

~~~
jeffdavis
Might work OK for video/audio (though DeepFakes calls that into question). But
what about the timestamp on it? Or what about GPS location (did you even
posses the phone at the time? Was the GPS data manipulated later?)?

There will be a huge class of things that basically can't be questioned,
denying you the ability to confront your accuser.

~~~
skybrian
Not a lawyer, but it's my understanding that the side that's presenting any
physical evidence needs to show why it probably wasn't tampered with and the
other side can question that.

When gathering evidence that you want to hold up in court, you need to think
about how to prove that.

------
jwatte
I am a little bit bored with these GAN style attacks. They show that the
networks haven't yet generalized well enough. They don't show that the
technology, or even the approach, is broken.

Personally, I think the main problem is some combination of a little bit of
over claiming in research, and a metric ton of over hyping and generalizing in
both media and business circles.

Robust multi model ensembles with strong generalization ability will show up
within the foreseeable future, and will be no more susceptible to optical
illusions than human beings.

------
plaidfuji
I think the obvious answer to this is to use a generative adversarial
approach, but the more I think about that the more difficult it sounds in
practice. Say once you've trained in the initial weights of your classifier
from your training set, you have a generator start adding distortions,
occlusion, and noise to the training set images, then train the classifier to
recognize those as true positives. Of course the difficult part is defining a
generator architecture that can learn to generate a wide enough variety of
distortions, occlusions and noise...

------
sushirain
Is it possible to create an adversarial example without access to the weights
of a model, and without being able to forward many images through it?

~~~
Fripplebubby
One method of creating adversarial examples in a "black box" setting is to
create and train a local model as a stand-in for the actual model using the
inputs and outputs of the actual model. [1] So, the answer is "no" but a
qualified "no" since in practice this seems to work. The second part, being
able to forward many images, is also a qualified "no".

1 - [https://arxiv.org/abs/1602.02697](https://arxiv.org/abs/1602.02697)

------
ams6110
It doesn't seem surprising to me that AI can be fooled by adversarial input.
Human brains, as advanced as they are compared to contemporary AI, are also
vulnerable to this. People figured out a long time ago that hunters, soldiers,
and other military assets can be camouflaged quite effectively by painting
them in certain patterns, for example.

Also ask any Illusionist / magician.

------
agjacobson
I couldn’t resist because everyone’s thinking of cat-guacamole. Also Labsix.

[https://mashable.com/2017/11/02/mit-researchers-fool-
google-...](https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-
program.amp)

------
bluetwo
Thanks to those keeping AI claims grounded in truth and not hype.

------
xstephen95x
"hallucinations" is an excessive anthropomorphization of a math model in my
opinion

------
bjornsing
The human visual cortex employs a combination of bottom-up and top-down
processing. So before we see a dog the brain sort of generates a picture of a
dog top-down, and compares that with the visual stimuli. My feeling is the
problem with the "AI" described here is that it's still bottom-up only.

------
hannasanarion
Why are we all freaking out about the ability to "put a sticker on a stop sign
that makes it invisible to YOLO".

I could just as easily put a sticker on a stop sign that makes it invisible to
people.

~~~
wepple
The difference is that a human will know it’s been tampered with and lower
their confidence: the attacks against NNs make it think with 98% certainty
it’s a 60mph sign, not a stop sign. Furthermore, if a human is monitoring the
system, a small well-placed sticker will not likely alert them to the fact
that the car is in trouble

------
pdkl95
> “People tend to trust each other in machine learning,” says Biggio. “The
> security mindset is exactly the opposite, you have to be always suspicious
> that something bad may happen.”

It's not just machine learning; far too much tech - software _and_ hardware -
has this problem. Everyday here on HN you see discussions about the _benefits_
of a new idea without any consideration for how it could be exploited.

~~~
sillysaurus3
_Everyday here on HN you see discussions about the benefits of a new idea_

Just the opposite. Rarely do you see any discussion of the benefits of a new
idea. It's mostly hate.

I suspect we're probably talking past each other, though. Concrete examples
would resolve it.

[https://news.ycombinator.com/item?id=16542183](https://news.ycombinator.com/item?id=16542183)

Notice the top comment is negative. In general, almost every top comment on HN
is negative. It's hard to write well positively, just like it's hard to be a
comedian and only make clean jokes.

~~~
falcolas
As a negative commenter on that topic: it's not a new idea. It's not even a
new tool. It's an iteration on an existing tool to make it more user friendly.
It's an iteration which, by all appearances, didn't go far enough.

Tools, like ideas, get feedback when posted to a forum like this. Feedback is
rarely going to be all positive, especially in the cases where a tool is still
rough.

~~~
sillysaurus3
I didn't mean to imply it was a bad thing. I meant to say it's hard to write
positively.

Without straying too far from the original premise, it's worth pointing out
that the idea of using a graph database for traditional tasks is in fact a new
idea. Most people would not consider doing it, and that makes it new.

Notice an interesting philosophical thing that happens every time
philosophical debates occur: We quickly get mixed up in a debate over the
meaning of words. "What do you mean by 'new'?"

It's worth excluding such questions from debate. Questions over the meaning of
words rarely go anywhere interesting.
[http://www.paulgraham.com/philosophy.html](http://www.paulgraham.com/philosophy.html)

Oh dear, look where we've ended up. But it was an intriguing place, I hope.

The overall point is that if you advocate something non-traditional on HN,
you're in for a world of hurt. This place is nothing if not adversarial. You'd
better have your I's dotted and your T's crossed. But those constraints are
what make the medium so interesting.

