
Automatic Differentiation in Machine Learning: A Survey (2018) [pdf] - lambdamore
http://www.jmlr.org/papers/volume18/17-468/17-468.pdf
======
cs702
I'm _trembling with excitement_ at the prospect that easy-to-use high-
performance automatic differentiation looks likely to become a "must have"
capability for more and more computer languages. It's going to become easier
and easier to specify objective functions and have the computer optimize
programs for a wider range of application.

~~~
thearn4
I work on non-linear optimization problems for most of my work projects, and I
know exactly what you mean. One of these years we'll be freed from remembering
calculus I :)

~~~
killjoywashere
right after the year of the Linux desktop, eh?

------
adamnemecek
The idea of duality is insane. Its present in linear logic, automatic diff,
constructive mathematics, probability, quantum theory, game design, discrete
optimization, etc.

I wonder if it's related to chirality.

~~~
ssivark
For those of us who are not aware of duality in auto differentiation (and
haven't had a chance to read the above review), could you introduce the idea?
Are you talking about forward mode vs reverse mode -- since I haven't
pondered, what's interesting/deep about that?

~~~
adamnemecek
AD relies on dual numbers. Dual numbers are more suited for doing calculus.

Structurally, dual numbers are numbers of the form a + b * e (where e is
epsilon s.t. e^2 = 0 but e != 0. Think of it as the imaginary constant but
instead of i^2 = -1, you have e^2=0).

For example, multiplication of two dual numbers (a + b * e)(c + d * e) = (ac +
ad * e + bc * e +bd * e^2). Since e^2 = 0, you end up with (ac + (ad + bc) *
e).

Here comes the magic. Dual numbers let you evaluate a function and get the
derivative at that point by just evaluating the function. In the above,
example, if this was the result of a function, ac is the value of the function
and ad + bc is the derivative of that function at that value.

[https://blog.demofox.org/2014/12/30/dual-numbers-
automatic-d...](https://blog.demofox.org/2014/12/30/dual-numbers-automatic-
differentiation/)

~~~
selimthegrim
Before looking at the wiki I thought, huh, Grassmann numbers on HN?

~~~
adamnemecek
You are not wrong.

