
Hyper Networks: An RNN that changes the weights of an RNN - GChevalier
http://blog.otoro.net/2016/09/28/hyper-networks/
======
Marazan
Haven't read the article yet but back in the 90s/00s there was a thing for
using Genetic Algorithms to evolve optimal NN topologies.

Edit: Cool, mentions it right out the gate.

~~~
api
Hybrid systems combining different ML techniques were generally in back then.
These should be revisited with today's processing power and data set
availability.

------
martinmusio7
This is an additional abstract dimension we add to it. As long as it helps, I
encourage to go this road. But I also fear on the other hand that it gets
harder and harder to understand NNs. Especially, if someone just starts with
NNs.

~~~
jacquesm
> Especially, if someone just starts with NNs.

That goes for any new field you are interested in. If this bothers you I have
a suggestion: pretend the present does not exist and go back to the beginnings
of the field, then do a 'faster than realtime' replay to allow you to see the
steps as they were made in the past until you catch up with the present.
Typically this will take you anywhere from 1 to 5 years depending on how fast
you work for something complex but you'll come away with a much deeper level
of understanding than you'd get if you just looked at what is being done
today.

It will also help you to not go down all the blind alleys of history because
you are now aware of them.

------
GChevalier
I wonder if it can be applied a 3rd meta-time. E.g.: an RNN would edit the
weights of another one which edits the weights of the main one. And maybe add
the L2L meta optimizer in that?

------
notgood
Lets go deeper, lets create a RNN that changes the weights of this RNN!

I fear someday we may accidentally discover (and disappoint ourselves) that
consciousness (AKA the mind) its just a self-modifying binary system and not
as magical as we like to assume... probably still centuries away even if
that's the case.

~~~
andybak
I've also pondered this but I worry that there's a degree of magical thinking
here.

Here's one thing too complex for us to understand (a conscious brain).

Here's another thing with superficial similarities (highly complex neural nets
of one kind or another).

Maybe this complex thing is the same as this other complex thing!

However - I'm still stuck at the stage where I feel that p-zombies, qualia,
the chinese room etc are pointing at a genuine explanatory gap. I know the
more rationally minded think this is hand-waving or closet-mysticism but
considering nobody has satisfactorily defined consciousness in a way that
doesn't simply hide the problem one layer deeper then I'm going to stick to my
gut feeling that the other side of the debate is doing just as much hand-
waving as me.

It's homculi all the way down.

~~~
taneq
> I'm still stuck at the stage where I feel that p-zombies, qualia, the
> chinese room etc are pointing at a genuine explanatory gap.

I can't shake the feeling that a lot of the arguments about qualia are begging
the question. 'Qualia' is a name for something on our _map_ of a human mind.
It doesn't necessarily describe something in the actual _territory_.

~~~
andybak
I did try to acknowledge the question begging in my comment. My point is that
both sides are begging the question.

So neither side (false dichotomy acknowledged) can use question begging as an
argument to dismiss the other.

What's left? Who has more to explain? Surely at this point Descartes kicks in.
The _one_ undeniable piece of knowledge we have - the one that everything else
is built on (precariously).

Isn't the burden of proof on those that want to reduce consciousness to
something else - when consciousness is all we can be certain of?

I'm not dogmatic on this. I am pulled strongly in both directions and I had a
youthful infatuation with Hofstadter/Dennett et al.

On the other hand I know my desire to imbue the self with something
transcendental is driven by romanticism.

So here we sit.

