
Introduction to Automatic Differentiation - noelwelsh
http://alexey.radul.name/ideas/2013/introduction-to-automatic-differentiation/
======
scentoni
Q: "What’s the derivative of a data structure anyway?" A: The Derivative of a
Regular Type is its Type of One-Hole Contexts
[http://strictlypositive.org/diff.pdf](http://strictlypositive.org/diff.pdf)

~~~
agumonkey
Amazing, I was looking for such a paper for a long time.

------
noelwelsh
Automatic differentiation computes with so-called dual numbers, which store an
infinitesimal (e) alongside the result of the computation. The infinitesimal
is the derivative. Working with infinitesimals is how calculus started out but
was abandoned when calculus was formalized using epsilon-delta limits
([http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition...](http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition_of_limit))

The is a newer formalization of calculus, called non-standard analysis, that
works with infinitesimals, called the hyperreals. As you can imagine, there is
a strong relationship between dual numbers and hyperreals. I found this
exchange interesting:

[http://math.stackexchange.com/questions/341535/is-the-
theory...](http://math.stackexchange.com/questions/341535/is-the-theory-of-
dual-numbers-strong-enough-to-develop-real-analysis-and-does-i)

It's kinda cool that non-standard analysis slips in through the back door when
using dual numbers. I always found working (informally!) with infinitesimals
quite simple.

------
TTPrograms
So there's a new name for taylor expansions now?

~~~
jwmerrill
AD is a way (actually ways) of organizing a computation. There are many
different ways to organize the computation and evaluation of a Taylor series
at a point.

As an exercise, try to explain the difference between symbolic
differentiation, forward mode AD, and reverse mode AD. You will find that
there is more content here than just a new name.

