
From scratch: reverse-mode automatic differentiation (in Python) - montebicyclelo
https://sidsite.com/posts/autodiff/
======
melvinroest
I'm just an enthusiast. With that said, I think the author's first image image
really captures the core of forward propagation and back propagation. For
people who want to get more of a feel on autodiffing, I find these 2 sources
amazing [1, 2]. They don't use the word autodiffing, but looking at it, I can
see it's the same thing.

I once made an attempt at animating feed forward propagation [3]. If I'd
animate back propagation, I'd probably make an animation on autodiffing as
well as that is the most intuitive way I know on how to explain back
propagation right now. It's still on my big to do list ;-)

[1]
[http://karpathy.github.io/neuralnets/](http://karpathy.github.io/neuralnets/)

[2] 3Blue1Brown does in episode 4 of his Deep Learning series:
[https://www.youtube.com/watch?v=tIeHLnjs5U8](https://www.youtube.com/watch?v=tIeHLnjs5U8)

[3] [https://melvinroest.github.io/Interactive-Neural-
Networks/](https://melvinroest.github.io/Interactive-Neural-Networks/)

------
bpesquet
Nice introductory resource.

On the same topic, see also:

\- Andrej Karpathy's elegant micrograd library:
[https://github.com/karpathy/micrograd](https://github.com/karpathy/micrograd)

\- This tiny neural networks API inspired by it:
[https://github.com/bpesquet/pyfit](https://github.com/bpesquet/pyfit)

------
beagle3
This is a fantastic hands-on introduction to practical autodiff. (And the
blog's other articles are worth reading too if you find their subjects
interesting).

------
adamnemecek
I wonder if automatic integration is possible. And yes I'm aware LS.

~~~
martincmartin
There's a straight forward algorithm for automatic differentiation. You can
think of it like long division: tedious to do, but you can easily program a
computer to do it.

Integration is a whole different ball game. It's essentially a puzzle: what
function, when differentiated, produces this result? There are various
heuristics you can apply, e.g. integration by parts. And there are some
databases of common integrals. But in the end, there's no algorithm,
integration requires ingenuity to solve.

~~~
Someone
It’s slightly better.
[https://en.wikipedia.org/wiki/Risch_algorithm](https://en.wikipedia.org/wiki/Risch_algorithm):
_“The Risch algorithm is used to integrate elementary functions. These are
functions obtained by composing exponentials, logarithms, radicals,
trigonometric functions, and the four arithmetic operations (+ − × ÷)”_

That algorithm isn’t easily programmed, though, and will fail in the many
cases where the integral cannot be expressed in elementary functions.

Worse, according to that page, it needs the ability to check whether
expressions are zero
([https://en.wikipedia.org/wiki/Constant_problem](https://en.wikipedia.org/wiki/Constant_problem)),
and that, in general, is undecidable if I understand that page correctly.

[https://mathworld.wolfram.com/RischAlgorithm.html](https://mathworld.wolfram.com/RischAlgorithm.html)
seems a tiny bit more optimistic about the limitations of this approach, but
[https://link.springer.com/chapter/10.1007/978-3-319-00894-3_...](https://link.springer.com/chapter/10.1007/978-3-319-00894-3_10)
seems, to me, in line with what Wikipedia says.

~~~
pkaye
There is also an open source rule based integrator that I came across.

[https://rulebasedintegration.org/](https://rulebasedintegration.org/)

