
Deep Learning without Backpropagation - williamtrask
https://iamtrask.github.io/2017/03/21/synthetic-gradients/?hn=3
======
etiam
The Hinton reference you were trying to find was likely joint work with
Timothy Lillicrap. Geoff refers to it, among other places, in the (recorded)
lecure from the reminescence symposium for David MacKay.
[http://divf.eng.cam.ac.uk/djcms2016/#hinton](http://divf.eng.cam.ac.uk/djcms2016/#hinton)

I don't know that there is an article out for that work yet, but the initial
finding by Lilllicrap et al. has been published as a preprint:
[https://arxiv.org/abs/1411.0247](https://arxiv.org/abs/1411.0247)

~~~
williamtrask
That's the symposium!!! Thank you so much! I was really hoping someone would
remember that part.

~~~
etiam
Happy to be of help. I think it's very interesting work and I'm glad to see
you recognized it in your post.

------
ilaksh
Is this a little similar to the paired autoencoder/decoders in this one
[https://arxiv.org/pdf/1609.03971.pdf](https://arxiv.org/pdf/1609.03971.pdf)?

~~~
cing
Nope, that paper uses backpropagation, and appears to be aimed at something
entirely different.

------
visarga
I loved the synthetic gradients idea when it came out, but the deafening
silence in the following months has been a letdown. I was hoping synthetic
gradients would be great for parallelizing models on multiple GPUs or
improving RNNs.

~~~
williamtrask
you know what's funny though...i don't think it's that hard to implement...
best i can tell, every framework already has what it needs.

there's just no boilerplate code for people to start with yet.

------
deepnotderp
Has anyone actually managed to get these to work on an imagenet scale?

~~~
ar15saveslives
I don't think it's usable IRL (in its present state), as, according to
figures, it doesn't work well even on cifar/mnist. Correct me if I'm wrong,
but the value of this paper is that you can decouple a model and train layers
asynchronously/independently, just first steps to distributed NN training.

~~~
curuinor
Well, HogWild works well enough. It's another one for the giant "weird shit
you can get away with in neural network land" bucket, I think

