
Artificial brains may need sleep too - jonbaer
https://www.lanl.gov/discover/news-release-archive/2020/June/0608-artificial-brains.php
======
jedberg
Damn that was fascinating. And it makes a ton of sense too.

In case anyone is looking, here is they paper they presented:
[https://drive.google.com/file/d/13u14zNsi7uthLpwCoMrckwhasau...](https://drive.google.com/file/d/13u14zNsi7uthLpwCoMrckwhasauw157I/view)

------
panpanna
Not sure what "stabilization" means in this context.

But isn't what they call sleeping basically regularization? In particular,
mini-batch and drop-out??

~~~
jhanschoo
It's important to note that what is being investigated are neuromorphic
processors. i.e. attempts to model biological brains. A paragraph from the
article directly addresses why conventional regularization isn't used.

> “The issue of how to keep learning systems from becoming unstable really
> only arises when attempting to utilize biologically realistic, spiking
> neuromorphic processors or when trying to understand biology itself,” said
> Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast
> majority of machine learning, deep learning, and AI researchers never
> encounter this issue because in the very artificial systems they study they
> have the luxury of performing global mathematical operations that have the
> effect of regulating the overall dynamical gain of the system.”

~~~
erikerikson
See also:

[https://en.wikipedia.org/wiki/Spiking_neural_network](https://en.wikipedia.org/wiki/Spiking_neural_network)
[https://en.m.wikipedia.org/wiki/Hopfield_network](https://en.m.wikipedia.org/wiki/Hopfield_network)

------
MindGods
A good discussion from May:
[https://news.ycombinator.com/item?id=23366516](https://news.ycombinator.com/item?id=23366516)

------
bachmeier
Questions. Does this explain

\- Why we dream?

\- Why we process information while we sleep?

\- Why lack of sleep is associated with serious health problems?

\- Why the amount of sleep is so long - about a third of the day in the most
common case - but varies a lot from person to person and by age?

A model of a brain as a neural network that's constantly learning, but for
stability purposes needs a restart every so often, doesn't seem convincing in
terms of other things we think we know about sleep.

~~~
Symmetry
1) REM sleep is associated learning new skills. We can teach experimental
subjects a new skill in the laboratory, selectively interrupt their REM sleep,
and see that their skill doesn't increase to the same extent as a control
group who don't have that phase interrupted. The same isn't true with non-REM
sleep. We don't really understand the mechanism though and this potentially
does bear directly on the question.

2) Well, it would get in the way of actually living? If you're talking down
some stairs and then your brain's motor cortex starts working on consolidating
a skill and you fall you'd hurt yourself.

3) I don't think we know that. Possibly we had to sleep for memory related
reasons and our bodies just evolved to use that time for other sorts of
important housekeeping when it wasn't moving around doing stuff?

4) Learning is very important and takes time. In addition to the procedural
memory consolidation in REM sleep we also move long term memory to a more
stable form of storage in other phases. But neural nets don't really have
anything analogous to a brain's explicit memory so this doesn't really bare on
that.

~~~
kthejoker2
#3: Doesn't hibernation suggest the reverse?

Sleep is primarily an energy conserving activity, being able to survive
periods of low energy availability (winter) is an evolutionary advantage.

------
micimize
For a functionalist, this and every other "more general AI might need X also"
is expected. Like, humans need sleep, pain, anxiety, social mediation, etc to
achieve our high level of generality. IMO, our base case should be that a
sufficiently general AI will need them as well.

I'm very excited by the budding neuromorphic phase of generality research for
these kinds of discoveries

~~~
AnHonestComment
Production ML systems already do this: servers used for peak capacity are
repurposed during lulls to retrain the model and do tasks like compact data.
That retrained model is then shared across the system.

So I’d argue that ML systems already “sleep”, it just looks more like dolphin
sleep due to their similar need for continuous operation.

------
Pick-A-Hill2019
(previously submitted by laurex)

[https://news.cornell.edu/stories/2020/03/researchers-
sniff-o...](https://news.cornell.edu/stories/2020/03/researchers-sniff-out-ai-
breakthroughs-mammal-brains)

------
combatentropy
> The best results came when they used waves of so-called Gaussian noise,
> which includes a wide range of frequencies and amplitudes.

Yesterday I was listening again to some videos of rain, to help me relax. I
know very little about what happens when we sleep or why we dream. But if it
is sort of like exposing our brains to soft noise, then that helps me
understand why rain relaxes me.

------
blackboxlogic
Good idea or bad idea: browser profiles where User enters their level of
understanding for a variety of subjects. browser sends the level specific to
each web request. Server customizes content, returning the same article,
styles ranging from explain-it-like-I'm-five to just-show-the-research-paper.

~~~
truculent
That would depend on how it's exposed to the user. Are you empowering the user
to run up and down that scale, or just feeding them whatever your model thinks
is best?

~~~
blackboxlogic
A slider at the top of the page gradually adding detail and vocabulary?

~~~
truculent
That could be cool. Another idea would be to present an executive summary at
the top and more detailed versions as you scroll down. Sort of akin to how
pictures load from more -> less pixelated on a slow internet connection.

------
2OEH8eoCRo0
Why do things need to be unconscious for brain repairs to happen? Wouldn't
evolution find it advantageous for a mind to stay awake while this happens?
Yet no animal has found a way to do so.

~~~
gfody
some animals alternate sleep between each hemisphere, there's a bunch of
examples of exotic sleep patterns in 'why we sleep'

~~~
2OEH8eoCRo0
That's a workaround to ultimately still sleeping.

------
realtalk_sp
Really tiring of this clickbaity nonsense infecting science and math. This
analogy is so tenuous you could snap it with a feather.

~~~
baxtr
_”AI may need food too”

“AI may need friends too”

“AI may need training too!”_

~~~
shahbaby
For ~70 years, the brightest minds on earth have been trying and failing to
solve AI. Many of them refusing to consider that something useful can be
learned from the human brain; the gold standard of what it means to be
intelligent.

Yet today, cutting edge deep learning technology is based on a crude and
increasingly inaccurate model of neurons.

If we're now making discoveries that are revealing artificial processes that
are similar to our own, it's a sign we're headed in the right direction.

~~~
nix23
>the only indisputable example of intelligence in the known universe

I dispute that any time any day with you, just because we define what
intelligence is, do's not mean that we are intelligent.

~~~
shahbaby
Unless we've specifically defined intelligence in a way that reflects
ourselves.

The useful things that we want to get out of an AI system, i.e generalized
learning of abstract concepts, are most clearly demonstrated by the human
brain.

Since we now seem to prefer down votes over discussion, I'll just leave this
with my own speculation that the reason for this strange avoidance of the
brain is that it's a dead end for both academia and industry.

It's much easier and more profitable to expand on already existing machine
learning technologies than to try and find some revolutionary breakthrough in
neuroscience.

~~~
nix23
Well we accepted that animals are intelligent too (no really long ago but
still), even that something like a swarm-intelligence exists, but yeah i know
what you mean.

EDIT: Wow you changed your comment that much that my comment makes nearly no
sense anymore.

------
op03
> spiking neuromorphic processor

what is this pls?

~~~
cinntaile
My (non-expert) take on this...

A neuromorphic processor is a processor that tries to function as the brain
does. I say tries to because the models are still very simplified versions of
how we think that a neuron works. The spiking part comes from one of these
models. A neuron will only send an electrical signal to another neuron once a
certain threshold is met, this is what is meant by a spike. What this allows
us to do is to add a temporal/time component to a neural network. You've
probably heard of "neurons that wire together, fire together". If you have a
lower threshold it means that the neuron you are sending the signal to is more
relevant for whatever thought process is going on right now and vice versa.
New input can change these thresholds.

The biggest promise of spiking neuromorphic computing seems to be a massive
reduction in energy usage while still offering decent accuracy. So for example
you could use it to train a neural network to get to 80% accuracy after which
you'd let another type take over to get to 95%. This field is still in its
infancy though, so expect things to change/improve fast.

~~~
erikerikson
> neurons that wire together, fire together

i.e. Hebbian learning

Usually with a Hopfield network or something similar.

My personal take is that biologically plausible approaches are attempting to
discover (and in lieu of understanding, harness) the algorithms (or mechanisms
that cause emergent intelligence) of our brains.

------
irrational
Have you tried turning it off and turning it back on? It works for all other
computers, why not artificial brains?

~~~
Smaug123
The obvious answer is that if the source of the problems is in state that is
persisted, then an active cleanup process may be required. Turning a database
off and on again will not help reduce its index bloat.

------
tetris11
If lack of sleep is linked to dementia and poor long term learning in humans,
and that every living organism in the world sleeps, I can believe machines
might need it too

~~~
VMG
That is flawed reasoning. Every intelligent organism in the world needs blood,
and machines do not.

Modern airplanes don't flap their wings either.

~~~
sgillen
They don’t need blood but they need energy, planes don’t flap their wings but
they need to generate lift. Whatever the function of sleep is, I think
artificial minds will likely need it too.

~~~
Smaug123
We have a handy analogy in the computing world, in that stop-the-world and
concurrent garbage collection algorithms both exist. If sleep is pure garbage-
collection, then I don't think there's any strong reason to expect it to be
required of all possible mind designs.

However, it seems increasingly likely that the empirical question may be
answered at some point soon!

------
BrissyCoder
Haven't read but I'm betting that there's an analogy to disk defragging...

~~~
Smaug123
How much are you betting, and at what odds? Because I'll take them.

