
Adding layers to the middle of trained network without invalidating the weights - Tenoke
https://svilentodorov.xyz/blog/add-layers
======
simonster
There's at least one existing paper about this idea
([https://arxiv.org/abs/1511.05641](https://arxiv.org/abs/1511.05641)). Also,
it is possible to initialize a convolutional layer so that it passes through
its input, but initializing the weights properly requires a little more work
than calling tf.keras.initializers

------
dontreact
It would be interesting to see neural networks that can grow layers as needed
using this technique. Perhaps you could have the network occasionally monitor
its own progress on a tune set somehow and adjust the model capacity based on
the amount of over/underfitting observed.

~~~
thrax
Exactly my thoughts on this.. if you could detect when the network was getting
saturated or over fitted, spawn new layers. Maybe this would allow starting
with much simpler networks, operating on lower rez inputs, and then scaling
then up as you feed them more data, instead of burning cycles slowing training
a large network from scratch.

------
wodenokoto
Author jumps straight into how to implement this in code before giving any
clues as to what the strategy or purpose is.

The idea is funny, but it would be nice if there were some arguments as to why
it might be useful, and even better with an example of a model being improved
by this.

~~~
phowon
The Hug of Death seems to have killed the page so I can't really tell what
it's about, but here's an example of inserting new trainable layers in BERT:
[https://arxiv.org/abs/1902.00751](https://arxiv.org/abs/1902.00751)

~~~
Tenoke
I don't think it's Hug of Death even since it's not actually that much traffic
and it's not down for me. Cloudflare + Github Pages just seems to be a much
less reliable combination than I believed before I moved to it a year ago (it
sometimes goes down even with almost no traffic, and then is up a second
later).

This paper is definitely relevant though!

