
Differential Equations with Transformers: Deep Learning for Symbolic Math - liao1729
https://medium.com/analytics-vidhya/solving-differential-equations-with-transformers-21648d3a1695
======
thomasahle
Post by the researchers (Facebook AI): [https://ai.facebook.com/blog/using-
neural-networks-to-solve-...](https://ai.facebook.com/blog/using-neural-
networks-to-solve-advanced-mathematics-equations/)

Arxiv paper:
[https://arxiv.org/pdf/1912.01412.pdf](https://arxiv.org/pdf/1912.01412.pdf)

------
whatshisface
I remember there being a lot of skepticism about the performance metrics used
in that paper last time this topic came up. Vaguely, I think they were
comparing the runtime of Wolfram's perfect answer search to their own model
which was not required to get the answer exactly right. I think there were
also some other shenanigans.

~~~
throwlaplace
I was a critic when this was first posted (and was promptly downvoted). I
don't understand why this is getting traction (or even a research direction
they decided to pursue). does anyone really believe this thing is doing some
kind of symbolic reasoning? it's doing nothing but memorizing training
examples! that it has nonzero performance in test is owing to the wonky
metric.

~~~
whatshisface
On one hand you have 1980s symbolic AI that can produce true, coherent
statements, and on the other you have GPT-2 which can produce really nifty
gibberish. If you could put the two together maybe that would be some
progress. I can see that getting transformers to emulate symbolic math might
be a step in that direction.

~~~
throwlaplace
that's not reasoning though. it's not even an expert system or prolog (because
adversarial examples exist). it's very close to a million monkeys typing.

~~~
monocasa
Where does a 100B neurons firing fit on to that spectrum?

------
program_whiz
Raises a tough question about mathematics generally. Is math anything other
than memorizing a set of rules (and being able to regurgitate them when shown
a problem)? Obviously for anything like arithmetic, multiplication, etc. the
answer is "yes this is just memorized".

Then for more complex math, can't it be argued that this is just recombination
of memorized theories and algos, but using some kind of random search?

I'm not sure math involves as much symbolic reasoning as we like to think, but
I could be wrong. The same goes for programming - mostly just regurgitating
known algos (which break into smaller memorized sub-algos). When presented
with something novel, generally just use random approaches until one fits. A
few people in history who had to come up with "the first" such programs had
some novelty, but even those were mostly just applications of known /
memorized solutions in a random way (e.g. quicksort).

~~~
whatshisface
For at least a hundred years (as long as symbolic logic has existed, however
long that really is) it has been widely known that you can iterate through
every single theorem without any special creativity or thinking. The problem
with that strategy is that you will spend most of your time on theorems like
(not not X => X), (not not not not X => X), (not not not not not not X => X)
and so on. What mathematicians actually do is use their human intuition and
aesthetic sense to pilot the mechanical formalism towards goals that are worth
achieving.

~~~
vladf
That’s not a really fair way of putting it. Unless you’re suggesting that
there’s something deeply magical about our human “aesthetic” intuition why
can’t that itself be coded into a set of syntactic indicators (e.g., favor
short equations, don’t repeat yourself, connecting theorems otherwise far from
each other in some distance metric is valuable) to be used as a search
heuristic?

~~~
whatshisface
The human aesthetic can't practically be programmed as a classical algorithm.
The human ability to identify cats can't even be programmed as a classical
algorithm, that's why neural networks were invented. Maybe a neural network
could predict how interesting mathematicians would find a theorem but by that
point there's no point for anybody to do anything, even artists.

------
manthideaal
Related: Poor man integrator (1), a 99 lines of maple symbolic integrator that
uses an heuristic (parallel integrator).

[http://www-sop.inria.fr/cafe/Manuel.Bronstein/pmint/index.ht...](http://www-
sop.inria.fr/cafe/Manuel.Bronstein/pmint/index.html)

Clearly, if your generator for equations is limited the system can learn the
dictionary lhs (integral) => rhs (derivative) to go rhs => lhs. (integrate)

