
Differentiable Programming - efm
http://edge.org/response-detail/26794
======
subnaught
This seems to be heavily influenced by (if not outright plagiarized from)
this[0], which, needless to say, does a much more thorough job of explaining
things.

[0]: [http://colah.github.io/posts/2015-09-NN-Types-
FP/](http://colah.github.io/posts/2015-09-NN-Types-FP/)

~~~
wrong_variable
yeah, I got angry halfway reading OP since I remember colah's article.

upvote for visibility and making sure colah gets credit for coming up with the
idea first.

colah goes deeper and talks about how it relates to algebraic types from
functional programming.

~~~
davidad_
I submitted the OP essay with an attribution ("see Chris Olah's blog for a
fuller treatment") but it was removed by the editors of Edge before
publication (apparently their policy is never to send readers away from Edge).
I have asked them to at least re-insert Chris' name and I hope they do so
soon.

~~~
cwyers
> apparently their policy is never to send readers away from Edge

Holy fucking wow. Do they know what an Internet is?

------
im2w1l
I previously made a toy VM where a program consisted of a matrix, rows
corresponded to instruction slot, and then the values in that row were
probabilities for different instructions. This means probability of emitting a
specific output is differentiable in the probabilities, which can then be
learned.

Didn't get very good results though.

~~~
_pmf_
There's also [0], which I found to be almost eerie, where someone has tried to
use evolutionary programming for an FPGA, with the result that the winning
generation exploited the analogue characteristics of the concrete FPGA chip
being used (i.e. it did not work on another FPGA chip of identical type).

Maybe I'm a bit peculiar, but this article sent chills down my spine.

[0] [http://www.damninteresting.com/on-the-origin-of-
circuits/](http://www.damninteresting.com/on-the-origin-of-circuits/)

------
thomasahle
So what's up with that differentiable stack? Is that similar to what the
Facebook people were doing on learning addition?

~~~
davidad_
Here's the differentiable stack/queue/deque paper:
[http://arxiv.org/abs/1506.02516](http://arxiv.org/abs/1506.02516)

It is indeed similar to some work from FAIR, where one of the tasks was binary
addition: [http://arxiv.org/abs/1503.01007](http://arxiv.org/abs/1503.01007)

------
tluyben2
Some code on the subject:

[https://github.com/zenna/Arrows.jl](https://github.com/zenna/Arrows.jl)
[https://github.com/wojzaremba/algorithm-
learning](https://github.com/wojzaremba/algorithm-learning)

------
dnautics
> This newly dominant approach, originally known as "neural networks," is now
> branded "deep learning,"

Is this true? My impression is that "deep learning" is a series of
mathematical tricks and design and implementation concepts that get you to
solve neural networks of depth greater than 3-4, which becomes mathematically
challenging.

~~~
visarga
One other reason machine learning has acquired the "Deep" label was the rise
of the computing power. It's not just a bunch of tricks, it's also plain old
hardware speed that is driving the revolution.

Also, we now have access to huge datasets to try our algorithms on. Progress
in ML depends a lot on the training data that is available.

