
Decentralized Reinforcement Learning - MindGods
https://bair.berkeley.edu/blog/2020/07/11/auction/
======
cs702
This looks... _incredibly interesting_. Having a multitude of neural networks
as agents competing with each other to maximize a general objective for the
entire "society of agents" seems very appealing to me as a mechanism for
scaling computation. Call it Minsky's pyrrhic victory, if you will.[a]

One obvious next step is to partition both network and resources within the
this "society of agents," so that no agent competes with all other agents and
no agent competes for all resources, creating sparsity and small-world
dynamics at runtime. The authors are probably already considering this.

Looking forward to digging in :-)

\--

[a] Minsky, who developed the "Society of Mind" theory, is often credited as
having caused a decline in neural net research and a shift to symbolic AI
research in the 1970s and early 1980s, after the publication of his book
"Perceptrons: an introduction to computational geometry," with Seymour Paper.
I find it a bit ironic that the "society of mind" in this new paper consists
of descendants of the perceptron. See
[https://en.wikipedia.org/wiki/Perceptrons_(book)#Reception_a...](https://en.wikipedia.org/wiki/Perceptrons_\(book\)#Reception_and_legacy)
for details.

------
throwawaygo
Yes. Composing archetypal agents in complex adaptive systems is the game I
want to see played. Enter into the realm of the structural topology of the
brain to see the power of NNs.

~~~
dpflan
Would you mind elaborating upon this?: "Enter into the realm of the structural
topology of the brain to see the power of NNs."

~~~
Mekantis
A part of what makes the brain so special and capable is how it's able to
structurally and functionally layer regions upon each other with their own
specialized functions which all funnel their outputs up and down as needed. So
even if every part of the brain is using the same kinds of neurons, they're
all completing very different tasks. This is something (afaik) we haven't seen
much in modern AI and it's probably a missing piece of the puzzle.

~~~
oehtXRwMkIs
Not every part of the brain is using the same kind of neurons. There are many
different cell types, and neurons are especially involved in epigenetic
changes even at the microscopic time scale. It's a common misconception to
assume that the classic artificial neurons are what real neurons are like.
Real neurons are incredibly more complex with many many more moving parts and
a misleading analogue to their artificial counterparts.

~~~
Mekantis
I'm not aware of any neuron that functions differently from other neurons. You
have a soma, dendrites, synapses, electrical impulses and glia. Even if there
are subtypes, they are all based on this basic functionality. Care to
elaborate?

~~~
oehtXRwMkIs
To elaborate on my first point:

In the context of neurobiology, the term "type of neuron" has a specific
meaning referring to the cell type of the neuron. The brain is host to many
many different cell type populations. But I see now that you're not talking
about this at all.

My second point:

I think we're running into a semantic/ambuiguity issue here. I'm not sure what
you mean by "type" or "kind" and "functions".

For example:

> I'm not aware of any neuron that functions differently from other neurons.

Every single neuron functions differently from other neurons, not just in
practice, but in theory (Leibniz's law). When you sequence the RNA present in
the nucleus or the soma (either one, usually not both), you will never get two
neurons with the same profile. In fact, you can "barcode" individual neurons
based on their gene expression profile. This profile is so complex and
ephemeral that it is essentially a fingerprint of the neuron at the time you
sequence it. And like I said, this profile changes very quickly over time.
Just in case the connection between function and RNA sequence is not clear,
look up central dogma. Pre-translational functions are a huge portion of what
neurons do.

> they are all based on this basic functionality

In terms of what you mean by basic functionality, I believe you are mostly
correct. However, one thing to note is that glia are separate from neurons,
and there are two main subtypes of neurons depending on what type of synapse
they have. You can have an electrical synapse, or a chemical one. But usually
when people talk about "neurons" they are talking about the chemical type. The
difference between the two are huge and this leads to my next point:

You can get as basic as you want to argue that all neurons are the same
"kind". You could argue that a binary tree and hashmap are the same "kind". I
could argue that everything in existence is the same kind since everything is
made of elementary particles (mereological nihilism). However, my original
argument is that artificial neurons are misleading when you want to think
about the actual brain. Although you focused on topology (which I'm assuming
you mean physical organization), I don't think you can really ignore the
atomic units of the brain, which are the neurons and glia. Now we are
certainly still very much in the dark about them, we have learned a lot, but
we are still very far from a complete understanding. However, our current
understanding is very far ahead of the "basic" idea of a neuron.

Take learning for example. A ANN learns by backpropagation. Learning is
essentially just gradient descent of the loss function with respect to weights
and biases. This is not at all how neurons "learn". If you focus on long-term
potentiation (LTP), the strengthening of synapses, the reason why this happens
between two artifical neurons is because the larger the weight and bias is in
a synapse (connection between two artifical neurons), the bigger its partial
derivative and impact via chain rule, meaning that backpropogation will
increase its weight proportional to its impact. In biology however, it is
incredibly complex and I don't really see how an ANN is really following the
brain here.

Our current understanding of LTP in biological neurons is that there is short-
term and long-term LTP (yes, long-term long-term potentiation). When an
excitatory neuron releases glutamate (not all neurons do, a key example of how
different cell types behave very differently), it binds to AMPAR and NMDAR and
the receiving cell is depolarized by the influx of Na^+, which allows the Mg^+
plug in the NMDAR to be removed which allows Ca^{2+} to flow in to the
dendritic spine. The Ca^{2+} binds to Calmodulin which binds to CaMKII (one of
many targets of Calmodulin) which undergoes autophosphorylation but also
phosphorylates AMPARS (GluRs) which results in LTP, since they are now
allowing in more positive ions. However this short-term LTP is temporary, and
to maintain it, it must be converted into long-term LTP. Long-term LTP
requires epigenetic changes.

How high-frequency stimulation changes gene expression is not yet fully
understood. But we know that G-protein-coupled receptors (which cascades down
to cAMP), calmodulin, and MAP kinase all directly affect the behavior of CREB,
which is a protein in the nucleus that binds to DNA and regulates
transcription (a transciption factor). So CREB controls the genes that
directly result in long-term LTP. One of these genes makes a protein called
Arc, which travels to the synpases (look into synpatic tagging theory if
curious how) and increases the amount of F-actin (not sure if this is fully
understood these days) which grows the dendritic spines and therefore results
in long-term LTP.

Note that what I just typed out is really just an overview, and there are
entire textbooks that describe LTP. Also, I'm just a student of this topic,
not an expert. And I only talked about LTP which is one of the basic subtopics
of learning and memory.

So how does this compare to ANNs? An artifical neuron does not adjust its
weights solely based on its connections, it must also be done with respect to
a loss function. I don't think biological neurons can be grouped in a way to
be said to be part of a network with a loss-function. Doesn't really make
sense. The only loss-function in biology is natural/artificial selection,
which means that loss is calculated at a population level, not at the
individual level, and certainly not at a neural network level. Furthermore,
evolution works on the phenotypes of random variation. It does not adjust
genotypes to some specific goal. Brains that reproduced relatively more
fertile offspring are able to slightly increase the proportion of their genes
and behavior. Not sure how this translates to artifical neural networks.

What does it mean for an artificial neuron to have short-term LTP? Artificial
neural networks are essentially a collection of matrices and connections. How
does it go through "time"? It changes with each epoch rather than a continuous
flow of time, so I guess you can make the number of epochs the analogy for
time. But this difference between trial-based models and time based models
also results in its own set of differences like the difference between the
Rescorla-Wanger model and the temporal difference model. Being tied to each
epoch and the loss function like how the R-W model is tied to each trial of CS
learning means that it is limited in what it can understand and optimize for.

The way I see it, artifical neurons and networks still very far from "entering
into the realm of structural topology of the brain" which I guess is another
way of saying that we are very far from GAI. Furthermore, I was only talking
about the disparity between biological neurons and artificial neurons in the
narrow context of LTP. I'm sure I could write much more about the disparity
between the organization of the brain and the organization of ANN described in
the article ("topology").

~~~
Nevermark
That was a great intro to a lot of nuances I suspected existed!

My only quibble is equating GAI with human-modeled AI in your last paragraph.

I expect the first successful GAI's won't be modeled on human AI, due to the
advantages of creating new algorithms without the limitations of evolution,
and due to the complexity of our evolved brain as you well explained.

But in any case, GAI is likely to be achievable many different ways.

Even in biology, I wonder if the independently evolved octopus nervous system,
bracketed into a short life time, mentorless childhood, and much higher levels
of decentralization, and higher sensory and motor complexity, might be more
scalable than our brain if it were artificially implemented.

The octopus ability to pull color information out of grayscale sensory
information is just one of many ways their short lived minds exceed us in
interesting ways.

~~~
oehtXRwMkIs
You're definitely correct that GAI is not necessarily going to be human-
modeled. Who knows how that's going to happen if it does. However, I'm betting
that it will be, or at least heavily inspired by it. I think you are
underestimating biomimetics. For example, convolutional neural layers and
residual neural networks were both based on cat/primate visual cortex systems.
And both were huge advances in computer vision.

The most "advanced" GI we know of right now is human intelligence, and I think
it would be wasteful not to extract as much as we can from it. You mention
octopus brains, but I would argue that for GAI we're more concerned about what
humans are good at with their oversized prefrontal cortex, which is higher-
level, abstract, executive cognition. Pulling color information out of
grascale doesn't sound all that impressive to me (especially since that sounds
like an easy unsupervised deep learning task) compared to being able type all
this out.

You mention the limitations of evolution. I think the key limitation of
evolution is that it is largely append-only. Our brain is like a nested doll
where the deeper you go, the further back you go in evolutionary history. The
way evolution plays out means that things are always built on top, there is
never really a large overhaul, since 99.99999...% that just means the zygote
isn't even viable. Do we really care about the lower levels beneath higher
executive function? Do we really care about modeling emotions (e.g.
fear/flight) like the limbic system, or modeling how the brain stem maintains
homeostasis such as monitoring carbonic acid levels in blood? Perhaps. But I
think the prefrontal cortex is what is most interesting. But I guess it
depends on what you want from a GAI.

~~~
Nevermark
We probably have similar views, with some differences.

We definitely have and will pick up inspiration from the brain as you point
out.

That is especially true in terms of the basic topologies of our brains neural
networks. I.e. the painfully simplified artificial neuron model, two layer
networks (which may roughly corrrespind to some single biological neurons that
can have hierarchies of dendrites as I understand), then deep networks,
convolution, recurrence, competitive layers and other topologies.

But I think the freedom of math to produce global algorithms directly allows
faster innovation in a way that incremental change via evolution never had.

I expect GAI will always have some biologically inspired roots, but my
untested (obviously!) opinion is that the first GAI will significantly benefit
from global gradient based algorithms, and incorporate symbolic and database
sides too.

The latter two not looking anything like how our neurons operate.

------
kian
Good to see the Hayek Machine idea being rebooted in the age of deep learning.
I've been wondering when someone would try this out again.

