
Why Artificial Brains Need Sleep - Xplor
https://www.discovermagazine.com/technology/why-artificial-brains-need-sleep
======
seesawtron
The authors inject artificial noise in cycles to their Spiking Neural
Networks. They call this noise injection as Slow Wave Cycles equivalent to
what brain generates during sleep. I do not know what exactly goes during SW
sleep in mammalian brain so it's hard to judge if this analogy by the authors
makes sense.

However, noise injection on its own has been extensively shown to decorrelate
signal from noise so that the networks don't fixate on noise in the input data
while training. This is widely used when using CNNs on image data and is known
to increase performance of network predictions. So I am not sure how much
different this article's findings are from this.

No link to arxiv publication makes it harder to make sense of these findings.

Edit: "Sleep" is not relevant for most ANN architectures used today. Synaptic
pruning however is, similar to dropout.

~~~
sidpatil
This sounds just like dithering.

[https://en.wikipedia.org/wiki/Dither](https://en.wikipedia.org/wiki/Dither)

~~~
seesawtron
Cool, yes we also add similar random (Gaussian) noise to image data before the
networks see it. The decorrelation theory comes originally form signal
processing field like your example shows. Old wisdom in new bottle.

------
burlesona
I’m in no way an expert on this stuff, but I’ve long wondered if AI is really
a hardware problem. That is, the way that conventional computers work is
nothing like the way brains work, and if we’re trying to model brain-like
intelligence, perhaps we need brain-like hardware to do it. It’s fascinating
stumbling across more and more articles describing neuromorphic hardware like
this recently. I guess this idea has occurred to a lot of other people.

~~~
mantap
It's actually the reverse. The hardware we have is _way_ more efficient than
our brains. One TPU can simulate billions of neurons a second in a tiny area.
Computer hardware has a significant time advantage. Electrical impulses in the
brain travel very slowly.

The problem is we don't understand how the brain is architected at anything
other than a very coarse level. That architecture is the product of millions
of years of evolution so we have some catching up to do.

~~~
d_tr
Except that the TPU does not simulate real neurons in any way. It just
performs simple arithmetic using an architecture that is suited to the ANNs we
use today. If you managed to understand how the brain works and tried to run
the model on a cluster of TPUs, you would need way more energy to simulate one
second of brain activity than the 20J a real brain uses in a second. Only by
some magical coincidence would any of the hardware we use today turn out to be
efficient.

~~~
darkerside
Are neurons binary in the way they operate, in that they either fire or they
don't? Or are they analog in nature, where a signal can be either strong,
weak, nonexistent or anywhere in between?

~~~
qubex
For starters, they use a pulse-frequency based signalling system.

And more generally, it’s pretty clear that there is a mixture of context-
sensitive inhibitory and excitatory signals both deriving from the dendrites
(inputs) and from the chemical stew of hormones and so forth they find
themselves immersed in.

There are neurobiologists who claim (controversially, I concede) that a
_single_ neurone is a fully self-contained ‘processor’[1], and not the
equivalent of single logic gate.

Those who somehow confuse our “neural network architectures” with the actual
biology in our skulls are really misguided. Based on the evidence, one can at
best opine that a TPU-based matrix add/multiply unit is approximating a very
clunky and un-nuanced neural network. To assert that they are one and the same
is... indefensible, when one goes beyond even the coarsest detail. A neural
network as we implement it technologically is nothing but a weighted network
of threshold units. That is all.

[1] To clarify: some kind of highly context-aware DSP with both long- and
short-term storage facilities that allow it to react not only to the inputs
that it is being fed, but also to inputs it was fed previously (and possibly
outputs it produced previously as well).

~~~
montebicyclelo
> Those who somehow confuse our “neural network architectures” with the actual
> biology in our skulls are really misguided..

To be fair, many lectures/books/articles/blog posts on ANNs start off by
presenting a biological neuron, and saying that ANNs approximate this.

E.g. [https://youtu.be/uXt8qF2Zzfo?t=385](https://youtu.be/uXt8qF2Zzfo?t=385)
(MIT 6.034 Artificial Intelligence, Fall 2010. 12a: Neural Nets)

~~~
qubex
I do not exempt those who make such statements just because they have
authority. If anything, they should know better.

I started off along much the same path (I bought and read _Neural Networks: A
Comprehensive Foundation_ (1994) by Simon Haykin) when I was in high school,
read it all and thought I knew everything there was to know on the topic, and
then started to read about neurobiology and was stunned to discover I was
living a lie. Then I read _The Computational Beauty of Nature_ (1998) by Gary
William Flake and decided I was going to dedicate myself to understanding the
complexities we brush under the rugs.

I think the Hacker News/Silicon Valley/Machine Learning/Developer crowd really
need to look beyond the virtuosismo of their implementations. It’s a very
introverted and self-referential mindset.

------
waveforms
The late Dr. Gerald M. Edelman (Nobel Prize in Physiology or Medicine) had a
fascinating theory he called "Neural Darwinism" that relied upon "reentrant
mapping" he insisted was not feedback. He had a lab that formulated his ideas
into software and a robot called Darwin 4. Does anyone here know what happened
to that software? Is it described in detail anywhere? The following is an
unflattering but useful discussion of Edelman's robot
[https://blogs.scientificamerican.com/cross-check/my-testy-
en...](https://blogs.scientificamerican.com/cross-check/my-testy-encounter-
with-the-late-great-gerald-edelman/)

~~~
seesawtron
I have not heard of this one but there's a similar robotics implementation of
a network based on the connectome (neural circuit wiring diagram) of C.
Elegans brains(the only complete connectome we have so far). Its still in
preliminary phase of doing cool stuff though but the idea is interesting.

------
js8
I have always felt that there should exist some analogy of the heat engines
(and Carnot cycle) in the information theory. The heat engine increases the
entropy between two heat reservoirs, and uses it to do work, decreasing
entropy elsewhere.

If the purpose of intelligence is to "convert data to information", then
perhaps it is akin to doing some work that decreases information theoretic
entropy?

For example, in regression, we express input data as parameters of the model
and error, so in the heat engine analogy, the input is the incoming heat, the
parameters are the work output, and the error is the waste heat. (The error
has higher entropy because there is less to explain, and you typically throw
it away.)

So perhaps, if somebody would figure out this analogy correctly, we would see
that we always need some cycles to continue the operation, just like in heat
engines. So then we could conclude that any agent might require some kind of
cycles of information creation and destruction in order to have intelligence.

~~~
qubex
The best definition of intelligence I have ever come across (cannot find the
reference sadly, maybe I’m paraphrasing here) is “ _that which you do to keep
your options open when you have no experience or training or prior information
regarding the circumstances you are currently in_ ”. This will manifest,
paradoxically, as a ‘force’.

Thought experiment in order to clarify: how would aliens, staring at our
planet for aeons through a telescope, reach the conclusion that there is or is
not intelligent life here on Earth? Well... they could observe that on
average, historically, about one every hundred million years or so there is a
major asteroid impact. They would eventually observe that from some point
onwards a larger and larger delay would set in from the last impact and the
next, far longer than expected. If they zoomed in they’d notice that somehow
asteroids on impact trajectories would be deflected as if by a physical force
emanating from the planet.

This would be our descendants, using the amazing technology at their disposal,
deflecting asteroids to ensure that the last major extinction level event will
be the one that killed off the dinosaurs 65 million years ago (and counting).
They’d be using their intelligence to keep their options open and avoid being
wiped out. They’re exerting a physical force on their environment and clearing
the sidereal neighbourhood of dangerous rocks and comets.

Conclusion: intelligence is about perception of data but for the sake of
control of the environment. “Data into information”, honestly, is too close to
being a truism to be of any utility.

~~~
TheOtherHobbes
My working definition is that intelligence is about accurate modelling of the
environment, both internal and external.

This includes the ability to make accurate predictions about threats and
opportunities within the environment, and the likely outcomes of planned
actions.

The more intelligent an entity is, the more successfully it can generalise
from past experience and anticipate and influence future events.

This is a completely different definition to plain old IQ test intelligence,
which seems to be closer to raw mental agility. I suspect you can score high
on mental agility - e.g. figure and number series manipulation - and still
have poor applied intelligence. The latter needs the ability to synthesise a
range of diverse stimuli, which is not the same as being able to manipulate
abstracted symbols with no surrounding context.

~~~
qubex
Our definitions seem entirely compatible: a more accurate and flexible model
will enable greater capacity to face circumstances. A deer perceives risks
from predators. We have a totemic fear of predators _but also_ things that are
less immediate, some innate ( _e.g._ heights) and some acquired culturally (
_e.g._ radiation). Our broadened threat model protects us from demise in a
wider set of circumstances: _e.g._ deer flock to to the exclusion zone forests
near Chernobyl because there;s few predators (including few humans), humans
fled from there because they knew or were told by those who knew of the risks
of radiation exposure.

EDIT: Revised preamble.

------
lifeisstillgood
It's fascinating to think that my dreams might just be "noise injection". And
this is why we do science folks - an explanation for sleep that came from
trying to teach silicon to recognise cats. Unabticipated benefits. That's what
we like.

------
1MachineElf
Philip K Dick answered this in 1968 with _Do Androids Dream of Electric
Sheep?_.

------
purplezooey
But do they dream of electric sheep..

