

Genetic Algorithm Walkers - clukic
http://rednuht.org/genetic_walkers/

======
Animats
I'm currently stuck at 1233 after 250 generations. The last improvement was at
gen 183. Walking gets part way down the first downslope, then falls. The
current champions are far better than any of their mutations, so a local
maximum has been reached. This is the fate of most evolutionary AI.

Once on the downslope, the controller for flat ground won't work, so now it
has to jump to something substantially more complex to progress. Making such
jumps was an unsolved problem with genetic algorithms 20 years ago, and I
commented back then that whoever solved it would win a Nobel Prize. Still
waiting.

I suspect that a solution lies in work on neural nets where one net is
trained, then a second, smaller, net is trained to match the first. This is a
form of "abstraction". Somewhere in that space is something very important to
AI.

~~~
Animats
Just to see if any further improvement was possible, I left this running for
over 18,000 generations. 98% of the improvement had happened by gen 183. The
last minor improvement was at gen 7313. There's no sign of a controller
evolving that can handle non-flat terrain.

I used to work on legged locomotion. Once you get off the flat, traction
control ("anti-slip") starts to dominate the problem. It looks like this
simulation isn't going to evolve anti-slip control.

------
saganus
A bit OT, but I remember a very cool Quake2 mod that basically built a neural
network so that bots could learn how to play.

The useful part was that you could save the NN to a file, and so you could let
Q2 run for hours (mostly I left it at night so it wouldn't prevent me from
playing :) ) and the hext day I would save the progress.

After about a week of doing this everynight, the bots evolved from just
jumping around in their spawn points without moving, to actually run around
the level and shoot the other bots on sight.

They didn't really have a good aim, but sometimes they would land a rocket or
a rail. I always wondered how much time would I have to let the bots evolve so
that they would become competitive.

Unfortunately after some time I lost my NN files and lost the project page.
Some time later I found again the project but it was not updated and the
download files were broken.

Wish I had saved the mod :(

Now, honest question: would this kind of approach work for, say, programming a
Hello World program? or would the number of variables and possibilities is too
big?

It would be interesting to put several of these bots to compete against each
other, but in different languages and see which one is "easier" to grasp by
the bot.

~~~
Houshalter
Someone made a GA evolve a hello world program in brainfuck. It's definitely
possible, but incredibly inefficient.

~~~
Florin_Andrei
If you look at the trillions of creatures thrown at the evolutionary game
every day in real life, Mother Nature herself is not terribly efficient
either.

------
sinwave
I suspect that the best way to tackle this problem is to treat "learning to
walk" as a reinforcement learning problem. This way you can evaluate actions
within each episode rather than waiting until the end of each trial to adjust
parameters encoding the walking strategy.

As karpathy suggests below (and he's certainly much more qualified than me),
an evolutionary method such as GA's, while apparently fairly effective, could
well be wasting valuable information learned in real time through interaction.

------
lukev
One interesting thing is that progress very much happens by fits and starts,
as it waits for the "lucky mutation" that will enable evolution to continue.

For example,

    
    
      0	Bedaaa Ceeici	6.07
      1	Bocodo Bidobo	105.93
      3	Aibebe Docoeo	107.74
      7	Diaebe Eocoeu	107.88
      10	Ciaabe Eocoeo	107.95
      25	Diaebe Facodu	108.19
      28	Biaebe Eocoeo	108.20
      30	Biaeae Eacoeo	108.47
      35	Beaebi Fucieo	109.88
      36	Biaebi Eucici	203.60
      42	Biaibi Fuceci	204.65
      45	Aiaibi Fuceeo	206.30
      47	Aiaibi Fucici	206.60
      48	Aeaebe Eoceeo	412.96
      56	Beaebe Euceeo	414.05
      59	Beaebi Fubiei	415.76
      73	Beaebi Focieu	519.01
      75	Beaebi Eocido	519.39
      96	Baaebi Gidido	521.00
      99	Baaebe Focedo	627.14
    

There are pretty massive jumps at generation 36, 48, 73 and 99.

By the way, this is at 50% mutation probability and 25% mutation amount. It
got stuck way earlier with large less frequent mutations (as one would expect,
if the probability of a beneficial mutation occurring is the limiting factor).

~~~
fivedogit
This is called punctuated equilibrium. Evolution happens in fits and starts
after periods of relative non-change.

[http://en.wikipedia.org/wiki/Punctuated_equilibrium](http://en.wikipedia.org/wiki/Punctuated_equilibrium)

~~~
im3w1l
If you look at how the scores for the walkers increase as they walk, you will
see that it jumps in increments of 100. I guess it is some kind of "milestone
bonus".

~~~
codys
The page says "bonus points are given for each proper step forward".

So it appears they are optimizing for whatever their idea of a "proper step"
is.

~~~
rednuht
That's every time the legs switch places. Without that bonus, you'd get a lot
of QWOP-ish individuals dragging one foot behind. :)

------
kendallpark
Ah. Aiaeca Eeaeci rocking it at 1048.50 at gen 256.

EDIT I've heavily tweaked my config throughout depending on whether I thought
I was trapped at a local maxima, etc. Right now I think I've settled on for
late-game:

10% mutation prob 1% mutation amount 3 to copy

Currently: 1056.91 @ 338.

------
freehunter
With a Core i7, I wasn't expecting the simulation to run so slow, even at the
lowest settings.. Maybe Firefox on RHEL 6 isn't the optimal platform for this
little toy?

\--edit: it runs much, much faster on Firefox with Windows 8.1. Not a Core i7,
but an AMD 8-core thing. Must be Firefox on Linux's implementation, unless
it's a video card/driver thing.

~~~
aselzer
Runs pretty fast with Chromium on Linux, but in Firefox it's pretty slow as
well. That's with a 2-core 1.7GHz i7

It's probably the physics simulation. Though random JS benchmarks show Chrome
and Firefox to be approximately equal in JS performance on my computer, Chrome
beating FF slighly on sunspider and the other way around with Octane.

------
Houshalter
What exactly is evolving? How is the genome represented, how does it control
the walking?

~~~
rednuht
The genome is represented by a set of parameters that control the rotation
speed on each joint, 3 parameters per joint.

The rotation speed for each joint on each simulation step is given by:

    
    
      x * cos(y + z * simulation_steps);
    

where x, y, and z are the parameters. I've experimented with more
sophisticated models, but it was hard to get any kind of evolution in the
attention span people give to a browser game. :D

~~~
karpathy
I've worked on similar projects during my Master's degree. An important trick
I learned was to make the controller a function of some kind of discrete
feedback in the world (e.g. when the foot hits the ground). In your equation
above for example, a simple way to incorporate this would be to reset
simulation_steps to zero every time a foot hits the ground.

This is also an important feature in the SIMBICON controller, which is
arguably the simplest and most robust walker system.
([http://www.cs.ubc.ca/~van/papers/Simbicon.htm](http://www.cs.ubc.ca/~van/papers/Simbicon.htm))

EDIT: and yes, Reinforcement Learning is much more effective way of attacking
this type of problem (e.g. see some recent work from Sergey Levine
[http://www.eecs.berkeley.edu/~svlevine/](http://www.eecs.berkeley.edu/~svlevine/)),
but I also agree with the author that GA are fun :)

------
Aqwis
Fun, but even after hundreds of generations the walkers are still pretty bad.
How good would they be after thousands or millions of generations? Is it even
feasible to create a decent walker using genetic algorithms?

~~~
Houshalter
These simulations are very sensitive to the parameters, some of which we don't
even have control over (e.g. population size or what method to select the next
generation.) And especially the way the genome is represented.

It doesn't really say how it works, but it doesn't seem like a very natural
way to do walking. E.g. here is are evolved walkers in a more complicated 3d
simulation: [http://vimeo.com/79098420](http://vimeo.com/79098420) They seem
to get very good after just a few generations compared to this.

~~~
jamesk_au
Incredible. It even evolves a walker resembling a variant of a kangaroo (at
[01:57]). With a target speed of 1.0 m/s, the legs alternate in taking small
steps, but at 2.0 m/s, both legs are used to jump.

------
V-2
I'd like to see a genetic algorithm evolutionarily figuring out what
parameters give you best performing walkers in shortest time ;) Meta-genetic
if you will

~~~
deong
It's a pretty well known field of research (though not specifically for toy
walking problems). Genetic algorithms are a member of the class of problems
called "metaheuristics", so your "meta" prefix is sort of already there. So
they went up a level and called them "hyperheuristics".

------
nhzmju
I was just about to give up at 70th generation, when the walkers suddenly
evolved from 1 to 4 steps in 5 generations.

Currently at generation 120 with 7 steps, looks like it is "evolving" in
bursts, with long periods of nothing.

Well it is stuck at 9 steps at gen 400+.

    
    
      327	aibobe baeado	948.93
    
      >> 548   aibobe baeado   1058.76
    

I wonder if the author got any further. I would say this is close to the
limit.

~~~
saganus
It's interesting how I got:

0 Cicebi Bicedi 206.20

1 Cibebi Dibodi 208.03

4 Cobebi Bibedi 208.54

6 Cobebi Bobedi 208.68

8 Cocebi Boaedi 209.53

14 Cocibi Boaedi 310.47

20 Cocebi Boaedi 312.73

36 Cocebi Boaedi 312.85

38 Cocebi Biaedi 312.89

40 Cocebi Bobedi 314.06

58 Cocebi Bobedi 417.71

119 Cicebi Bibodi 522.37

331 Cicebi Bibuei 624.26

And at ~500 generations the champions are still the same 3. I have fiddled
with the parameters several times to no avail. I've tried very little mutation
probability (1-5%) with low mutation amount (1-10%) but also 75-100 mut. prob
with 1-5% mutation amount and any figures in between and nothing seems to make
it go out of the local minima.

Is there any way to get out of this? or when this happens in nature the
species simply goes extinct or gets eaten by everyone else?

~~~
saganus
Ah.. seems like I finally got out of it at gen ~530 by changing the amount of
champions to copy to increase the amount of gene variation in the pool.

I guess it kinda stuck into an inbreeding sort of loop.

------
ColinWright
So is this really a genetic algorithm, or is it hill-climbing? Are you
combining genomes from different individuals? Or are you just mutating a
single genome? There are no controls for crossing genomes, so it seems likely
to be hill-climbing.

If not, how are you mapping the genome to the phenome?

~~~
Houshalter
In the source code their is a function for mating.

------
mmanfrin
Reminds me of this really fantastic video of finding the gaits of different
sized/legged 'animals' at different speeds:
[http://vimeo.com/79098420](http://vimeo.com/79098420)

------
gus_massa
I just got one that evolved kneeling to last a few more seconds:
[http://imgur.com/mii9N8J](http://imgur.com/mii9N8J)

------
hellbanner
Please make one for QWOP
[http://www.foddy.net/Athletics.html](http://www.foddy.net/Athletics.html)

~~~
maaaats
There already is, actually. I read the paper a while back. Video:
[https://www.youtube.com/watch?v=eWxFI3NHtT8](https://www.youtube.com/watch?v=eWxFI3NHtT8)
Paper:
[http://dl.acm.org/citation.cfm?id=2598248](http://dl.acm.org/citation.cfm?id=2598248)

~~~
hellbanner
You made my day. Nice find!

------
V-2
Reminded me about
[http://rednuht.org/genetic_cars_2/](http://rednuht.org/genetic_cars_2/)

If you're disappointed with walkers' performance, try cars, they improve much
more. The principle remains the same of course. Attention: addictive.

~~~
codys
Yep, I remember that and [http://boxcar2d.com/](http://boxcar2d.com/) ending
up on HN some time ago

------
shkkmo
You seem to have some bugs with the copying of champions and/or the identical
reproduction of walks at the very short simulation length. Without having
adjusted simulation parameters I no longer have my highest scoring walker
saved as a champion.

~~~
rednuht
Thanks, I'll look into it!

------
ehtd
Very cool! I did my master thesis in Genetic Algorithms and bipedal robots.
You can watch the results here:
[https://www.youtube.com/watch?v=SYONnbvsxz0](https://www.youtube.com/watch?v=SYONnbvsxz0)

------
clemsen
This looks like fun! It seems that the copied champions are simulated in every
run. As long there is no random influence you can probably skip this
simulation, as the simulation of the champions is also most likely the most
CPU intensive.

EDIT: Grammar.

~~~
nhzmju
I was thinking the same thing. Since the gene certainly isn't very long the
best champions could be stored, until a better champion is found.

------
hcarvalhoalves
Converged pretty fast to the first step on the 10th gen, then got stuck having
a seizure and falling on it's head "forever".

I bet this would be interesting evolving a simpler movement mechanic, like in
bacteria.

------
sago
This is how NaturalMotion started. I remember their CEO Torsten evolving
muscle controllers on the Mathengine physics engine (1999?). Get it right, and
you get to build a pretty nice business out of it!

------
bryogenic
You could optimize the no-graphics/high speed even more by not simulating the
copied champions (since their end score is pre-determined)

------
comboy
I always spend way too much time on these sites. Not a new idea, but the
visualization is awesome.

------
yuashizuki
its not improving after gen 190. Its stuck.

~~~
nhzmju
How may steps/score? Try changing the parameters.

~~~
yuashizuki
I did it was runnig at 0fps with max speed. I even changed the mutation part
but was still no improving. I am running dual core machine.

