
The Essential Tools of Scientific Machine Learning - ChrisRackauckas
http://www.stochasticlifestyle.com/the-essential-tools-of-scientific-machine-learning-scientific-ml/
======
jofer
This is an aside to the main point of the article, but I have to take issue
with a couple of quotes:

> For a long time scientific computing has kept machine learning at an arm's
> length because its lack of interpretability and structure mean that, despite
> it’s tremendous predicitve success, it is almost useless for answering these
> kinds of central scientific questions.

> However, the recent trend has been to merge the two disciplines, allowing
> explainable models that are data-driven, require less data than traditional
> machine learning, and utilize the knowledge encapsulated in centuries of
> scientific literature.

Recent trend?? This is the norm in every scientific field I've even slightly
worked in... This quote just described what we call "inverse theory" in the
geosciences. It's the root of pretty much everything we do and has been for
many decades. Those of in the scientific computing world have hardly been
keeping machine learning at arm's length... A lot of machine learning methods
come from scientific fields (e.g. gaussian processes).

There's more to machine learning than CNNs, and many fields have been using
machine learning much longer terms like "data science" and "AI" have been
around. We just trend towards parametric approaches or approaches that can be
reasoned about more clearly. (E.g. lots of convex optimization problems and
interpolation/"super-resolution" problems.) Those are machine learning methods
as well.

All that said, most of what they're talking about in this article is about
approximating solutions to differential equations with purely non-parametric
approaches. That is somewhat new and is a much narrower topic, which this
article does a nice job of describing. (I don't really take objection to the
article as a whole, but a few sentences in it strike me as presumptuous.)

~~~
ChrisRackauckas
Solving inverse problems has been around forever, agreed. Not just in
geosciences but also aerospace, biology, etc. But, some of the ways that
people are beginning to use universal function approximators inside of
scientific simulators is new. The techniques are very similar to traditional
inverse problems though: use adjoint equations for gradients and update
parameters with some optimizer. It's just now they models include more non-
mechanistic terms to fit. There were hints of it before with latent force
models, but I am not sure of anyone using this idea to successfully solve 100
dimensional PDEs until 2 years ago.

>There's more to machine learning than CNNs.

Most definitely agreed. I think there's a lot to do with utilizing all of the
methods of machine learning neural networks and beyond, automatically within
scientific simulation codes.

~~~
jofer
Yeah, using non-parametric methods to approximate PDEs is really what this
article is about, and it is a recent and very fruitful trend. I got a bit
unnecessarily up-in-arms about phrasing.

(edit: And I just realized you're the author -- excellent work by the way!)

------
haddr
Very much Julia oriented. Nice that Julia is catching up with other. This one
however is missing comparison with R tools. I guess R would also be quite
green on the "Quick Summary Table".

~~~
ChrisRackauckas
Do you have an example of an R automatic differentiation package which has an
ecosystem of compatible tools like PDE solvers, automatic sparsity detection,
convolutional neural networks, and global sensitivity analysis? If you have
one I'll add it to the list. But the ones that I know are things like madness
[1], radx [2], and TMB [3], all of which are good AD systems in their own
right (the latter most having some good parallelism), but are missing most of
the relevant features for a discussion of scientific ML (but would be great
for standard ML). I could also point to autodiffr [4], but that's using Julia
to autodiff R.

But if there's one I missed I'd love to hear about it and start tracking the
project!

[1]
[https://cran.r-project.org/web/packages/madness/vignettes/in...](https://cran.r-project.org/web/packages/madness/vignettes/introducing_madness.pdf)

[2] [https://rdrr.io/github/quantumelixir/radx/man/radx-
package.h...](https://rdrr.io/github/quantumelixir/radx/man/radx-package.html)

[3]
[https://www.jstatsoft.org/article/view/v070i05/v70i05.pdf](https://www.jstatsoft.org/article/view/v070i05/v70i05.pdf)

[4] [https://non-contradiction.github.io/autodiffr/index.html](https://non-
contradiction.github.io/autodiffr/index.html)

~~~
haddr
I don't, but maybe a disclaimer on R in this article could bring those who
could share their experience with R?

------
xor0110
Good thing I did not ever "Give Up On Julia"™! So happy to see numerical
programming breaking new grounds like this

~~~
taigi100
You and me both ^_^

------
mrguyorama
How do I, with a very nice AMD GPU, and running on Windows, play around with
any sort of GPU accelerated learning for anything? Last I knew, pretty much
nothing worked with either of those caveats.

~~~
apl
Running Windows is perfectly fine; the major libraries for GPU-accelerated
autodiff and networks (CUDNN with Pytorch or Tensorflow) have great support
nowadays. It's the AMD GPU that remains essentially useless, as of 2019. If
you want to get into the game, I'd recommend buying a middle-of-the-road
NVIDIA GPU like the RTX2060.

For toying with autodiff and basic CNNs, CPU works just fine by the way...

~~~
mrguyorama
>It's the AMD GPU that remains essentially useless, as of 2019

I guess more important question... _Whyyyyyyyyyyyyy_

~~~
jawilson2
Probably support for CUDA over OpenCL.

~~~
ChrisRackauckas
CuBLAS and CuDNN are just better libraries than what exist on OpenCL right
now. Until that changes it'll be hard to switch for numerical work.

------
peter303
I was hoping 'scientific' meant publishing the training data and resulting AI
model in order that others could either use it or reproduce it. A lot of
neural net A.I. isnt really scientific in this sense.

