
Clever Machines Learn How to Be Curious - he0001
https://www.quantamagazine.org/clever-machines-learn-how-to-be-curious-20170919/
======
cgmg
> For example, imagine a television displaying nothing but static on its
> screen. Such a thing would quickly engage the curiosity of a purely novelty-
> seeking agent, because a square of randomly flickering visual noise is, by
> definition, totally unpredictable from one moment to the next. Since every
> pattern of static appears entirely novel to the agent, its intrinsic reward
> function will ensure that it can never cease paying attention to this
> single, useless feature of the environment — and it becomes trapped.

Reminds me of
[http://people.idsia.ch/~juergen/interest.html](http://people.idsia.ch/~juergen/interest.html)

> To discourage the controller from focusing on truly unpredictable, random
> inputs (such as uninteresting details of white noise), later approaches
> model the expected progress of the predictor: parts of the world where the
> predictor fails to learn (no data compression progress!) become less
> interesting than those where its predictions improve.

This seems like a better/more general solution than the one presented in the
article, since the agent could otherwise get stuck with noise over which it
_does_ have causal influence. The agent could even _generate_ , using its
actuators, noise that is unpredictable and chaotic, but not interesting.

~~~
goialoq
If the agent can generate unpredictable but learnable noise, then the agent is
learning about its own ability, which is very interesting.

~~~
cgmg
Not sure what you mean by 'unpredictable but learnable'. Can you clarify?

------
MrQuincle
I see Oudeyer mentioned, nice.

There is definitely other research that has to be mentioned in this field.

Most prominently Karl Friston's work on free energy. It's a proper Bayesian
formulation of optimizing for surprise, in this case minimization. The
challenge is randomness in actions or environment. In a deterministic
environment it might make sense to maximize surprise, however in a stochastic
environment it makes sense to minimize it. You don't want a robot that
navigates to those parts of space that are totally chaotic.

Surprisingly, by minimizing surprise an agent can develop knowledge of the
world on the long term as long as that world is dynamic enough. There are no
dark rooms to retreat.

Ralf Der argues for something similar from a dynamical perspective:
[https://www.informatik.uni-leipzig.de/~der/](https://www.informatik.uni-
leipzig.de/~der/). It's about minimization towards (a set of) equilibrium
states.

------
grondilu
It seems to me that if they can crack that, it's basically game over. They'll
just implement it into a robot (either real or simulated) with cameras and
arms, throw toys at it, and let it learn by itself like babies do.

Am I oversimplifying much?

~~~
joering2
I think you do and that's the whole point.

We humans learn how to walk or keep balance by falling. It would be even fun
to constantly fall; however there is pain involved, so we learn to do anything
NOT to fall.

I think the first one who will design simple 2-leg machine and lock it in the
room for a year and let it learn whatever it wants with simple objective "stay
tall, don't fall" will be able to walk Boston Dynamics machines on a leech to
a local park.

~~~
grondilu
> It would be even fun to constantly fall;

You wouldn't go far, though. There is more information to gain in exploring
the environment than on perpetually falling.

------
icc97
In the section of Avoiding the Novelty Trap:

> This abstraction incorporates only features of the environment that have the
> potential to affect the agent (or that the agent can influence)

I like the parallel between this and various courses I had about working
efficiently / not getting too stressed by only focusing on what you can
influence.

------
anigbrowl
Roll on the day. I've been wishing for a chatbot/virtual assistant that asks
questions for quite a while now.

------
Havoc
Being curious is the essence of human intelligence.

Either demonstrate hard-AI or tone down that title a bit.

~~~
kobeya
Did you read the article? It's about how to make AI programs experience and
act on curiosity.

