
Operational Calculus for Differentiable Programming - bmc7505
https://arxiv.org/abs/1610.07690
======
whatshisface
One thing that is not entirely clear to me about constructions like these is
the behavior of array indexing. I understand how to differentiate the
assignment of x+y into the variable z, but if I use X as an index into memory
and then write Y at that location, the proper result of differentiation looks
a lot less clear. One reason for that is my memory locations are discrete
points, but X must be treated as if it was continuous, if functions over it
are to be differentiable.

~~~
radarsat1
My understanding is that the trick to differentiable key-value stores is to
treat the key as a probability over locations, and to treat the value as a
weighted sum using these probabilities. From (pdf)
[https://aclweb.org/anthology/D16-1147](https://aclweb.org/anthology/D16-1147)

------
amuresan
Could someone with a better understanding of the topic give a brief example of
how this can be used in practice?

~~~
ivan_ah
This is a theory paper so no immediate practical applications. Potential
applications would be generalizing derivative calculations used in currently
ML libraries, see
[https://en.wikipedia.org/wiki/Automatic_differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)
and [http://www.cs.nuim.ie/~gunes/files/Baydin-MSR-
Slides-2016020...](http://www.cs.nuim.ie/~gunes/files/Baydin-MSR-
Slides-20160201.pdf#page=6)

Useful quote from paper: "We have been inspired by the development of
differentiable programming to formalize a theoretical model, that encompasses
the ideas underlying differentiable programming and provides a more general
setting for investigations of differentiable programs. The presented
theoretical model enables analytic investigations of differentiable programs
through algebraic tools, that are closer to the field of programming; i.e. the
presented operators can take the same role as higher order functions in
functional programming."

------
keithalewis
It's funny/not funny nobody can give a concrete example of how to apply this.
AFAICT the last (only?) example in the paper is about copying memory. I know
an easier way to do that.

