
Neurogenesis Deep Learning - groar
https://arxiv.org/abs/1612.03770
======
SubiculumCode
They simulate neurogenesis, I guess, but they do not incorporate the most
interesting part of that neurogenesis: That is the new neurons are born into
the dentate gyrus, a region thought to have a particular capacity to
orthoganalize feature representations that are similar (e.g. pattern separate)
allowing distinct memories to be formed for similar events. The dentate gyrus
outputs to a region called Cornu ammonis 3 (CA3) which is heavily recurrent,
and thought to br able to pattern complete a full representation from partial
inputs. That is, CA3 can encode and retrieve the relations between 2 or more
features or objects. For a mathematical model and review one might read: Rolls
(2013)
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3812781/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3812781/)

but many others exist. I'd write more but typing in my phone is driving me to
distraction.

~~~
arkymark
this is really interesting, where/how did you learn this?

I'd like to learn more about these things - brain regions, connections,
functions - and what they might imply about the kinds of computations that are
going on, but my background is mainly on the AI/math side of things.

~~~
SubiculumCode
I'd like to add that our knowledge of the details of hippocampal neuroanatomy
are probably the most advanced of any brain region, which allows the somewhat
informed construction of computational models. I wish I had more time to learn
modeling methods, I have specific developmental hypotheses I'd like to test in
such a model. In the end, I'd probably need to find a knowledgable
collaborator though.

------
jostmey
Neurogensis? How about neural death as a way to prune large neural networks
into more compact ones--now that is a research idea!

~~~
cing
I believe several papers have examined efficiently pruning neural networks,
but neural death would be better branding ;)
([https://arxiv.org/pdf/1506.02626v3.pdf](https://arxiv.org/pdf/1506.02626v3.pdf)
[https://arxiv.org/pdf/1510.00149v5.pdf](https://arxiv.org/pdf/1510.00149v5.pdf))

~~~
simonster
Yann LeCun called a technique for pruning weights "optimal brain damage" back
in 1990:
[http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf](http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf)

------
groar
Basically trying to achieve a certain level of plasticity in deep neural nets
by getting inspiration from
[https://en.wikipedia.org/wiki/Adult_neurogenesis](https://en.wikipedia.org/wiki/Adult_neurogenesis)

~~~
andreyk
To add on to this - they "specifically consider the case of adding new nodes
to pre-train a stacked deep autoencoder", by basically keeping track of when
certain layers cannot reproduce their input and then adding more
nodes+retraining with both new (not reproduced) and old data. It is quite
intuitive, basically the most naive and obvious first attempt at the problem
(not meant in a condescending way, just want to point out it's not that
generalizable and is pretty ad-hoc).

------
argonaut
Sorry if I'm being snobbish, but I do wonder why this paper is only being
submitted to IJCNN, a 2nd tier machine learning conference. I know students
who publish undergrad research at workshops with lower acceptance rates than
IJCNN. I can't think of any important machine learning papers published in
IJCNN in the recent past.

~~~
habitue
It depends on what conclusions you're trying to draw from that information.
What conference a paper was accepted to is a second-order signal of the
noteworthiness. It's probably easier for someone versed in the field to just
read the paper to determine if it's interesting. If you're using the
conference as a quick pass/fail as you skim through the abstracts of hundreds
of papers, ok, but you probably wouldn't make time to comment on HN about it
in that case.

This paper looks like it builds on pretty well-known techniques like stacked
autoencoders, so let's see what first-order noteworthiness data we can gather
from a quick skim of the paper. If I had to guess why it wasn't accepted into
a better conference:

\- It uses stacked autoencoders, which are pretty out of fashion

\- It bothers reporting results on MNIST

\- (more subjectively) It pulls an unfortunately common technique of saying
"here's something the brain does" and then hand-waving that it's a deep reason
why a technique they've come up with is useful, when in fact the relationship
is just "inspired by the general idea of", not "performs the same function as"
the biological mechanism. In this case, I think the tenuous connection of
their technique to research on neurogenesis is pretty flimsy. Clearly
neurogenesis is not how an adult human brain forms new memories or gains
proficiency in new skills (which they acknowledge in the conclusion)

~~~
argonaut
> It's probably easier for someone versed in the field to just read the paper
> to determine if it's interesting.

> If you're using the conference as a quick pass/fail as you skim through the
> abstracts of hundreds of papers, ok

You answered your own statement, I think. Most researchers will skip a paper
in a second tier conference. In fact, most I know won't read an entire paper -
they'll only read some of it and skip stuff.

You're correct that I am not an active researcher (otherwise I would not have
time to be commenting). I merely did some research back in college. But
honestly that little experience gives me a huge leg up on most HN commenters
in understanding research. It's unfortunate that the only reason this paper is
#1 on HN is because it has a cool title.

That being said, MNIST is not really a disqualifier. (Unfortunately) MNIST is
the most popular dataset referenced in NIPS 2016 papers
([https://twitter.com/benhamner/status/805864969065689088](https://twitter.com/benhamner/status/805864969065689088)).
The handwaving is also forgivable; many NIPS papers handwave a lot too.

------
irinarish
A good point was made that a model of neurogenesis must also incorporate
neuronal death besides neuronal birth (since hippocampus and the brain as a
whole have physical constraints, you can't keep growing your network
infinitely :). That's why any model of neurogenesis must incorporate interplay
between birth and death of new (and old) neurons; that's was the main idea of
the paper I mentioned in an earlier post (this year ICLR submission
[https://openreview.net/forum?id=HyecJGP5ge](https://openreview.net/forum?id=HyecJGP5ge))
Note that just adding nodes to networks was proposed before, eg. the classical
work on cascade correlations.

------
irinarish
For a model that incorporates both neuronal birth and death, see ICLR
submission at OpenReview:

[https://openreview.net/revisions?id=HyecJGP5ge](https://openreview.net/revisions?id=HyecJGP5ge)

NEUROGENESIS-INSPIRED DICTIONARY LEARNING: ONLINE MODEL ADAPTION IN A CHANGING
WORLD Sahil Garg, Irina Rish, Guillermo Cecchi, Aurelie Lozano

------
gallerdude
Very wishful thinking on my part, but I think we're far closer to a general
intelligence than most expect.

~~~
empath75
I think what we might see is a kind of autonomous corporation that is
nominally under the control of shareholders, a CEO or a board, but which makes
decisions without very much or any human input, and which gains some amount of
legal rights through corporate personhood.

It won't be a 'general ai', though. More like a set of loosely connected
systems that operate 'in the best interests of the shareholders', however
that's defined.

It's pretty much the end state of the trend of pushing decision making to
algorithms to remove moral and legal culpability from individuals.

~~~
BeingIncubated
I'm hoping that eventually translates to the state.

------
iverjo
How does this relate to Progressive Neural Networks [0]? That technique is
also about accumulating knowledge (while not forgetting existing knowledge)

[0] [https://arxiv.org/abs/1606.04671](https://arxiv.org/abs/1606.04671)

------
joantune
It never ceases to amaze me that the best steps towards achieving AI is to
look at how we perceive that a Neuron works and simulate it.

And the thing is, we aren't exactly sure why exactly that is.. it's amazing.

Sometimes the best thing we can do is imitate nature

~~~
throwaway287391
I completely blame my own community, rather than you, for writing this, but as
an AI researcher, your comment is terribly painful to read. We have little to
no idea how actual neurons (let alone entire brains) really work. The things
that are often called "(artificial) neural networks" really shouldn't be
called that. I strongly prefer terms like "computational networks" or (where
applicable) "recurrent/convolutional networks".

~~~
hyperbovine
> We have little to no idea how actual neurons (let alone entire brains)
> really work.

I think that slights neuroscience, which has devoted the past 60 years to
answering this question, to a fair degree. But I agree that the biomimetic
motivations offered up for various flavors of neural net feel pretty bogus. It
seems to me like, among the major old-school researchers in the field, only
Geoff Hinton still does this.

~~~
throwaway287391
Fair. I was definitely unnecessarily harsh on neuroscience; my quibble is only
with my own community's claims that what we're doing is anything like how the
brain works. Thanks to you and sxg for correcting the record.

------
m3kw9
Very nice coin term for just another deep learning method

------
hmate9
Slightly off topic, but I hate how publications are written. It seems like
authors are purposely using big words and sentences that are often 5-6 lines
long in order to make it seem more clever.

I find myself often having to reread a sentence in order to understand it.

These algorithms are often very simple and can be easily explained. Don't over
complicate them.

~~~
amelius
Then here's a challenge: could you write the abstract of the article in
"simple English", without changing the meaning?

~~~
unfamiliar
Here's my attempt; not a huge number of changes because it was not too bad to
begin with, but with slightly less self-indulgent language, and a lot of the
jargon has to stay (partly because I don't know the field):

Neural machine learning methods, such as deep neural networks (DNN), have
achieved remarkable success in a number of complex data processing tasks.
These methods have arguably had their strongest impact on tasks such as image
and audio processing - areas where humans have always performed better than
conventional algorithms. In contrast to biological neural systems, which are
capable of learning continuously, deep artificial networks have a limited
ability for incorporating new information after a network has been trained. As
a result, continuous learning methods could be very helpful in allowing deep
networks to handle data sets which change over time. Here, inspired by the
process of adult neurogenesis in the hippocampus, we investigate how adding
new neurons to artificial neural networks can allow them to acquire new
information, while preserving what they have already learned. Our results on
the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes
lower and upper case letters and digits, show that neurogenesis looks like a
good approach for tackling the "stability-plasticity dilemma" that has been a
problem for adaptive machine learning algorithms for some time.

As an academic, I tend to agree that we frequently feel compelled to apply
more verbosity than is strictly required in order to communicate the intended
semantic constructs.

~~~
SubiculumCode
Academic writing avoids using periods. Unfortunately.

------
yahyaheee
Seems a bit like a GAN

------
paulsutter
TL/DR:

\- "We specifically consider the case of...a stacked deep autoencoder (AE),
which is a type of neural network designed to encode a set of data samples
such that they can be decoded to produce data sample reconstructions with
minimal error

\- "The first step of the NDL algorithm occurs when a set of new data points
fail to be appropriately reconstructed by the trained network...When a data
sample’s RE is too high, the assumption is that the AE level under examination
does not contain a rich enough set of features to accurately reconstruct the
sample.

\- "The second step of the NDL algorithm is adding and training a new node,
which occurs when a critical number of input data samples (outliers) fail to
achieve adequate representation at some level of the network.

\- "The final step of the NDL algorithm is intended to stabilize the network’s
previous representations in the presence of newly added nodes. It involves
training all the nodes in a level with both new data and replayed samples from
previously seen classes on which the network has been trained.

