
What is backpropagation and what is it doing? [video] - adamnemecek
https://www.youtube.com/watch?v=Ilg3gGewQ5U
======
SonOfLilit
We need more people teaching math through visual intuition. Life a friend of
mine said, "if you want to do computation fast, phrase it as a problem for
your GPU, er, visual cortex".

Here is a tool you can play with to visualize this:
[http://playground.tensorflow.org](http://playground.tensorflow.org)

If you liked this video, try this different visual intuition of what a neural
network does that I find even better:

[http://colah.github.io/posts/2014-03-NN-Manifolds-
Topology/](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/)

Also, remember that back propagation is a very general algorithm, it works not
only on linear transformation weights but on any direct acyclic computation
graph that is differentiable in its weights.

~~~
phreeza
This can also go wrong, for example visualising probability distributions in
low dimensions leads to very wrong intuitions about the behavior of high-
dimensional dimensional distributions.

~~~
kahoon
I think this is a very good point. Years ago people were worried about
gradient descent getting stuck at a local minima, plausibly because this
problem is very obvious in a 3 dimensional space. In higher dimensions however
this problem seems to go away more or less and a lot of worrying about the
issue seems to be the result of lower dimensional intuitions wrongly
extrapolated to higher dimensions.

~~~
Eliezer
Stuff got stuck in local minima for years before we learned about stuff like
momentum and dropout and dropped a ton of GPU power on it.

~~~
llamaz
When I was implementing a neural network for a university assignment (2 years
ago so my memory might fail me), we had to run our algorithm multiple time
with different starting positions, then take the minimum of those local
minima.

I'm not sure what momentum and dropout are, but I agree with Eleizer, without
these things (which I didn't use) local minima are a problem.

~~~
eat_veggies
Dropout is where you randomly remove neurons from your network during
training, which prevents them from depending too much on specific neurons
(making the output more generalizable). It was developed in 2014 so it would
have been brand new tech back when you were in your class.

------
adamnemecek
There needs to be some sort of organized push for visualization tools. I know,
I might be bringing the proverbial owls to the proverbial Athens with saying
that here, but I really do feel that if done right this could impact the
course of the world like nothing else. This could be as important as idk,
invention of book press or smth. Make computer "the visualization machine".

I think that one of the fundamental problems is that to be a visualization
machine, you need to have easy access of the GPU and OpenGL is provides
anything but. I think that shadertoy (shadertoy.com) is the thing that comes
the closest but the learning curve is kinda steep.

I know that people like Alan Kay, Bret Victor or Michael Nielsen (his post was
on the fp the other day
[https://news.ycombinator.com/item?id=15616637](https://news.ycombinator.com/item?id=15616637))
share these sentiments but this is a task bigger than a single people.

Idk what I really mean by "organized push". I'm not sure if the problem is
well defined too

~~~
minimaxir
In deep learning, TensorBoard
([https://www.tensorflow.org/get_started/summaries_and_tensorb...](https://www.tensorflow.org/get_started/summaries_and_tensorboard))
works with TensorFlow and Keras to show what the model is doing. However, it
ends up being more complicated/unintuitive than a YouTube video, so it's not
as useful.

~~~
adamnemecek
The problem is that this is an ad hoc solution. What I'm talking about would
be some formalization of visualization (I guess kinda like grammar of graphics
without the statistical aspect) so you can visualize just about anything.

~~~
minimaxir
“Visualizing just about anything” isn’t helpful if you want to _learn_ from
the visualization, though. (c.f the /r/dataisbeautiful subreddit nowadays:
[https://www.reddit.com/r/dataisbeautiful/top/?sort=top&t=mon...](https://www.reddit.com/r/dataisbeautiful/top/?sort=top&t=month))

That’s not to say that a purely artistic data visualization has no _value_ ,
but it’s not academic. (I admit I am guilty of that at times)

~~~
adamnemecek
Data visualization is only a part of it. I'm talking about visualizing
concepts.

~~~
posterboy
Structure _is_ data

------
kharms
In my opinion this author produces the best math videos on youtube.

If you can afford it and enjoyed this video, consider supporting him on
Patreon.
[https://www.patreon.com/3blue1brown](https://www.patreon.com/3blue1brown)

~~~
aidos
Someone on here (I think) recommended his videos on linear algebra a while
back and I've since watched them all, several times.

A couple of hours of watching time built an intuition and understanding of
linear algebra and the broader maths around it that 4 years of university
training didn't give me. That's a little unfair, because I obviously learnt a
lot on the courses that make these videos easier to understand, but man,
they're so well done.

~~~
NuclearFishin
Totally agree. Had exactly the same experience. Became a patron as a way to
offer my thanks!

------
nouveaux
This was very timely for me and for anyone else learning, here are the first
few videos of the series:

[https://www.youtube.com/watch?v=aircAruvnKk](https://www.youtube.com/watch?v=aircAruvnKk)

[https://www.youtube.com/watch?v=IHZwWFHWa-w](https://www.youtube.com/watch?v=IHZwWFHWa-w)

[https://www.youtube.com/watch?v=Ilg3gGewQ5U](https://www.youtube.com/watch?v=Ilg3gGewQ5U)
(Original video)

[https://www.youtube.com/watch?v=tIeHLnjs5U8](https://www.youtube.com/watch?v=tIeHLnjs5U8)

------
quotemstr
The entire YouTube channel is fantastic. 3Blue1Brown's series on linear
algebra is the best I've seen anywhere.

~~~
wybiral
Agreed. Pretty much every video on that channel is just as good as this one.

------
afarrell
A bit of a side-note, but I think it is an interesting piece of marketing that
Amplify Partners decided to sponsor[1] the previous video in this series. I
wonder (and hope) we'll see more VCs sponsoring open educational content
relevant to their focus.

[1]
[https://www.youtube.com/watch?v=IHZwWFHWa-w&t=1205](https://www.youtube.com/watch?v=IHZwWFHWa-w&t=1205)

------
timonoko
In 1988 Teuvo Kohonen had an "animation" with rotating disks how the
perceptron learns
[https://youtu.be/Qy3h7kT3P5I?t=42m24s](https://youtu.be/Qy3h7kT3P5I?t=42m24s).
Did not help comprehension much.

------
phkahler
What tools does a person use to make a video like this? I've been wanting to
do the same on my topic of expertise for a while now.

~~~
edanm
He uses custom tools.

However, he actually recommends _against_ using his tools. He suggests a
better option is to use traditional animation tools.

I'm actually not sure what one would use for more traditional animations of
his style though. I mean, theoretically you can use blender/etc for most 3d
things, but how easy would it be to make something math-based there? Hopefully
someone with some real animation experience can chime in.

~~~
mcintyre1994
On the Manim Github he has some suggestions: "For 9/10 math animation needs,
you'd probably be better off using a more well-maintained tool, like
matplotlib, mathematica or even going a non-programatic route with something
like After Effects. I also happen to think the program "Grapher" built into
osx is really great, and surprisingly versatile for many needs."

~~~
raverbashing
> I also happen to think the program "Grapher" built into osx is really great

I didn't know about this, and it's a nice find

~~~
pls2halp
There's a really cool story about how the software wasn't supposed to ship
with macOS, but the devs got it on anyway:
[http://www.pacifict.com/Story/](http://www.pacifict.com/Story/)

~~~
Micoloth
Great story, thank you for sharing! :D

------
perfmode
Each one of these videos consists of thousands of lines of code. The attention
to detail is impressive.

~~~
posterboy
have you looked at visualizations done in mathematica? LoC is not a good
measure here.
[http://community.wolfram.com/content?curTag=graphics%20and%2...](http://community.wolfram.com/content?curTag=graphics%20and%20visualization)

------
samueloph
oh my god, as soon as i saw this video was from 3Blue1Brown i immediately
thought "this gonna be good!". I didn't realize he was posting a Deep Learning
series.

------
crusso
I played around with ML a few years back. I took the Andrew Ng course on
Coursera and spent some time with some python notebooks - but I never did
anything with it beyond just proving that I could follow the examples and
implement my own ML solutions for some simple training sets.

Now I have some problems I'd like to solve with ML. So assuming that I
understand the basic concepts, what's the HN recommendation for a good
library/system to get started with on doing some practical ML with neural
nets?

Would TensorFlow be the best way to get into it?

~~~
andreyk
Keras is probably the best library to get started - Ternsorflow is mainly
needed if you need to mess with lower-level details, which for many things is
not really necessary.

~~~
crusso
Would you still recommend Keras for non-image work?

------
oliv__
I dove into this not knowing anything about neural networks, but the feeling I
came out of it with was incredible: I love it when something blurry and
obscure slowly morphs into a sharper picture in your mind, it's so empowering.

------
mlamat
Thank you very much. I must code a neural network with backpropagation for my
AI class. Can anyone recommend a book?

~~~
Yreval
This book is a good practical introduction that walks you through the basic
ideas as you develop some basic functionality.
[http://neuralnetworksanddeeplearning.com/](http://neuralnetworksanddeeplearning.com/)

I'm often pretty skeptical of e-books and self publications, but the above
link is pretty good (and the video series linked here references it as well.)
The Goodfellow book that another commenter mentioned is a high-quality survey
of the field and a nice, high-level overview of different research directions
in deep learning, but isn't as pragmatic as an introduction.

------
alexasmyths
This video series is amazing and I wish it existed long ago.

------
cmatt01
There needs to be some sort of intense advocation for visualization tools.

