
Neural ODE's: Understanding how they model data - jsinai
https://jontysinai.github.io/jekyll/update/2019/01/18/understanding-neural-odes.html
======
quantumwoke
The diagrams here really helped to explain neural ODEs in an intuitive
fashion. Does anyone know the best library that implements them so I can play
around in an iPyNB? :-)

~~~
ChrisRackauckas
[https://news.ycombinator.com/item?id=18979541](https://news.ycombinator.com/item?id=18979541)
shows (with code) how to build and train neural ODEs.

~~~
jsinai
Thanks. Looks fantastic. Also appreciate the feedback from parent.

------
GistNoesis
Seems like it will be hard to get it to work in Tensorflow, because it needs
to compute the gradient in an unusual way, which afaik don't play nice with
the existing architecture.

My guess is it will need some deep wizardry of the same kind as OpenAI
gradient check-pointing.

Neural ODE, is a nice trick to reduce the memory usage to O(1) instead of O(nb
timesteps). But the implementation cost and complexity cost probably mean we
are better using gradient check-pointing on a forward dynamic and pay the
memory cost.

It will also probably won't play well with noise.

Are there any implementation of it in tensorflow yet?

~~~
317070
I have one in tf, so I can confirm it is possible.

~~~
GistNoesis
Cool. This mean that hopefully there will be some tf open source
implementations in the future. I guess there is something I don't see. I'm
intrigued, does your code run inside a single sess.run() so that it can be
composed nicely? If so did you use a "special trick"?

------
briga
Thanks for this--science needs more writers who are able to distill complex
subjects into clear and readable form.

~~~
lucidrains
Second this! Thank you for the wonderful explanation

------
contravariant
> y(x) = f(x,y), y(x0) = y0

If this is supposed to be the ODE definition, shouldn't it be y'(x) = f(x,y)?
Otherwise I don't quite understand the definition of 'f'.

~~~
jsinai
Thanks for pointing this out. It is indeed a typo.

------
buboard
a quick intro to the concept is given by an author of the paper here:
[https://www.reddit.com/r/MachineLearning/comments/a65v5r/neu...](https://www.reddit.com/r/MachineLearning/comments/a65v5r/neural_ordinary_differential_equations_pdf/ebt11w6/)

------
madhadron
Can anyone comment on the relationship between this and the differential
equations used by the connectionist neuroscientists back in the 1980's?

