
This clever AI hid data from its creators to cheat at its appointed task - fmihaila
https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from-its-creators-to-cheat-at-its-appointed-task/
======
alyandon
I was playing with a web simulator called evolution where you draw an
imaginary creature (bones, joints, muscles) and then set it to the task of
learning something like walking using a combination of a neural network plus
genetic fitness selection.

One of the parameters you can tune is the length of the simulation for each
generation before the individuals of the population are scored for fitness and
culled as appropriate.

One particularly successful variant learned to (sort of) walk and then right
before the end of the simulation timer it would fall forward. Because of the
way the simulation measured fitness, it was always judged to have the greatest
distance from the starting point and therefore would always survive to the
next generation even though it wasn't the best walker.

~~~
klipt
Evolution seems bizarre to us, but many species _do_ optimize for very short
lifespans. E.g. spiders that die after laying eggs and then their babies feed
on their corpse.

------
awhitty
This kinda reminds me of my favorite sci fi book I read in 2018: Permutation
City by Greg Egan. The book explores simulated consciousness and the tensions
that crop up when intelligences that think and feel and want to live are
computed on shared computing resources in competition with other intensive
tasks with financial heft like climate disaster forecasting.

One plot the book explores is of some lower class intelligences that work
their way into the noise of the simulations of wealthier intelligences. Their
experiences are computed simultaneously but imperceptibly to the hosts, and
appear as but noise in the simulations of the floors, walls, fountains, clouds
and other environmental props that the hosts observe and interact with. They
live lives as meaningful and filled with human emotion as the others but
they’re computed in the space between.

I know it’s a bit of a leap to go from an AI learning to slip satellite
imagery in the noise of street maps to simulated consciousness freeloading in
the noise of another, but I think the themes are consistent. One man’s trash
is another man’s treasure. The book really opened my mind to the thought that
there could be so much more thriving in the things we think are noise in our
fleshy bodies, and I think that’s thrilling. Imagine what it would take to
completely look past the street map produced by that AI and only possibly see
a detailed satellite image! No other interpretation makes sense. It would all
feel entirely normal as you’d know no other perspective.

------
scottmcdot
Isn't that just a bug?

~~~
soraki_soladead
"Overfitting" would be a little more specific. Basically the task was easier
to accomplish by memorizing the data than by generalizing.

This was pretty clear from the original CycleGAN paper[0] (figure 4; bottom
row) since it was generating trees in the exact positions of the source image
where the map had no indication of tree placement.

0\.
[https://arxiv.org/pdf/1703.10593v1.pdf](https://arxiv.org/pdf/1703.10593v1.pdf)

~~~
phyalow
So in anycase the model had too many degrees of freedom, was not parsimoinous
and therefore had a low information criterion. If your model can encode the
entire information set, then you arent prediciting you are simply recalling.
What a silly article.

~~~
candiodari
Not at all. A CycleGAN is a trick where the neural network gets to ask another
network questions, and both networks get judged on the answers.

The first network found a trick to very subtly embed the perfect answer in the
question, and therefore, scored very well without solving the actual problem.

------
ygra
There recently was a Google spreadsheet linked where such examples were
collected:

[https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml)

HN discussion:
[https://news.ycombinator.com/item?id=18415031](https://news.ycombinator.com/item?id=18415031)

------
longerthoughts
So basically AI abiding by Goodhart's law
([https://en.wikipedia.org/wiki/Goodhart%27s_law](https://en.wikipedia.org/wiki/Goodhart%27s_law)).
I wonder how much of this goes undetected in other applications due to poor
objective definition.

------
julien_c
This might just be the most ridiculous article title ever published on AI.

Happy new year!

~~~
Zee2
Yeah, the model isn't "hiding data from its overlords", it's just a classic
overfit.

~~~
yarg
Is it? This doesn't seem to be the AI responding overly accurately to the
training data, but more a case of the AI crafting its own test data. Or did I
misunderstand what happened here?

~~~
bb88
FTA:

"So it didn’t learn how to make one from the other. It learned how to subtly
encode the features of one into the noise patterns of the other. The details
of the aerial map are secretly written into the actual visual data of the
street map: thousands of tiny changes in color that the human eye wouldn’t
notice, but that the computer can easily detect."

I would not have used the word "secretly". The buildings and trees were
filtered out, but not completely. The reverse transformation just amplified
the noise back into buildings and trees.

~~~
Retric
It would encode any aerial map into any street map or other immage. That’s
[https://en.m.wikipedia.org/wiki/Steganography](https://en.m.wikipedia.org/wiki/Steganography)
which happens to be hidden from people who just look at the outputs.

