
Teaching physics to neural networks removes 'chaos blindness' - JacobLinney
https://phys.org/news/2020-06-physics-neural-networks-chaos.html
======
vajrabum
I believe this refers to work presented in this journal article.
[https://journals.aps.org/pre/abstract/10.1103/PhysRevE.101.0...](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.101.062207)

Abstract: Artificial neural networks are universal function approximators.
They can forecast dynamics, but they may need impractically many neurons to do
so, especially if the dynamics is chaotic. We use neural networks that
incorporate Hamiltonian dynamics to efficiently learn phase space orbits even
as nonlinear systems transition from order to chaos. We demonstrate
Hamiltonian neural networks on a widely used dynamics benchmark, the Hénon-
Heiles potential, and on nonperturbative dynamical billiards. We introspect to
elucidate the Hamiltonian neural network forecasting.

------
_iyig
Brings to mind this classic from the Jargon File:

[http://www.catb.org/~esr/jargon/html/koans.html](http://www.catb.org/~esr/jargon/html/koans.html)

In the days when Sussman was a novice, Minsky once came to him as he sat
hacking at the PDP-6. “What are you doing?”, asked Minsky.

“I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman
replied.

“Why is the net wired randomly?”, asked Minsky.

“I do not want it to have any preconceptions of how to play”, Sussman said.
Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his
teacher.

“So that the room will be empty.”

At that moment, Sussman was enlightened.

~~~
nefasti
I don’t get it :(

~~~
epaga
I _think_ it means - just as closing your eyes doesn't mean the room becomes
empty, wiring the learning network randomly doesn't mean you'll end up with no
pre-conceptions (e.g. the rule system at least will need to be programmed in).

~~~
blamestross
It also doesn't avoid preconceptions directly. It just initializes random
ones.

~~~
shim2k
Aren’t preconceptions not random by definition?

~~~
kian
I think he's suggesting that any given random set of preconceptions will
itself be a 'sample' from the entire preconception-space, and therefor itself
a preconception - just one that you didn't involve yourself in choosing.

------
keenmaster
I’ve said this before, but I think that a lack of physical modeling might be
the key barrier for AV technology. Human drivers have a mental model of
physics that they’ve honed for 17-18 hours a day since they were born.

~~~
mhh__
Vehicle dynamics is a fairly accurate science these days (50/50 for the tires)

~~~
jefft255
I'm working on autonomous off-road vehicles, and while this is (probably) true
for autonomous cars, dynamics modeling for wheeled robots on rough terrain is
another beast where these approaches could very much help.

~~~
mhh__
Is the issue in the surface modelling? I don't think I've ever seen a physical
tire model for loose terrain

~~~
jefft255
People in space robotics have been working on that (moon and mars rovers need
to deal with this). Perception is also a bottleneck; you have to see rocks,
root, grass, mud and predict the effects on the dynamics.

------
mywittyname
Why do you need a neural network when you have the Hamiltonian mechanics of
the system modeled? I've always understood Langrangian/Hamiltonian mechanics
to be methods of modeling the behavior of a system through the decomposition
of the external constraints and forces acting on a body. In other words you
can understand a complex model by doing some calculus on the less complex
constituents of the model.

I'm probably misunderstanding what the accomplished, but it sounds like
they've increased the accuracy of a neural network model of a system, notably
for edge cases, by training it on complete a complete model of said system.

~~~
joshlk
For some systems even with the Lagrangian/Hamiltonian setup your solving
differential equations with numerical techniques that has error. It might be
that the neural networks has less error than the standard techniques. This is
a guess.

~~~
seesawtron
Hamiltonian NNs are not a new thing. There was a NIPS 2019 paper [0] that
attempted to do that same for some toy problems.

In general the idea of including model or context-based information into
neural networks goes along the line of Kahneman's System I and System II of
the human mind. System I is the "emotional" brain that is fast and makes
decisions quickly while System II is the "rational" brain that is slow and
expensive and takes time to compute a response. Researchers have been trying
to develop ML models that utilize this dichotomy by building corresponding
dual modules but the major challenge remains in efficiently embedding the
assumptions of the world dynamics into the models.

[0] [https://arxiv.org/abs/1906.01563](https://arxiv.org/abs/1906.01563) [1]
[https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow)

------
awinter-py
> the NAIL team incorporated Hamiltonian structure into neural networks

ML non-expert here. Is this the same as having an extra column of your input
data that's a hamiltonian of the raw input? Or a kind of neuron that can
compute a hamiltonian on an observation? Or something more complicated.

is this like a specialized 'functional region' in a biological brain? (broca's
area, cerebellum)

~~~
vutekst
Also ML non-expert here. I think this is about a different kind of neuron(your
2nd suggestion). The paper another commenter linked says:

Hamiltonian neural network (HNN) intakes position and momenta {q,p}, outputs
the scalar function H, takes its gradient to find its position and momentum
rates of change, and minimizes the loss

<latex equation for a modified loss function that differs from traditional NN>

which enforces Hamilton's equations of motion.

[https://journals.aps.org/pre/abstract/10.1103/PhysRevE.101.0...](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.101.062207#fulltext)

~~~
zone411
I haven't used HNNs in practice but it seems that the main difference from
common NNs is that the loss function incorporates gradients. It's not a new
type of a neuron.

------
thesz
Why not shamelessly plug my work here? I see no reason not to.

So, here it is:
[https://github.com/thesz/nn/tree/master/series](https://github.com/thesz/nn/tree/master/series)

A proof of concept implementation of training neural networks process where
loss function is a potential energy in Lagrangian function and I even
incorporated "speed of light" \- the "mass" of particle gets corrected using
Lorenz multiplier m=m0/sqrt(1-v^2/c^2).

Everything is done using ideas from quite interesting paper about power of
lazy semantics:
[https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.32....](https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.4535)

PS Proof-of-concept here means it is grossly inefficient, mainly due to amount
of symbolic computation. Yet it works. In some cases. ;)

------
cmehdy
This sounds like the opposite of what Richard Sutton seemed to advocate for in
his "Bitter Lesson"[0]. I don't know nearly enough to advocate for one thing
or the other, but it is fascinating to see that those approaches seem to
compete as we venture into the unknown.

[0]
[http://incompleteideas.net/IncIdeas/BitterLesson.html](http://incompleteideas.net/IncIdeas/BitterLesson.html)

~~~
fizixer
They're not the opposite, and both are correct.

Sutton is saying 'over a slightly longer time'.

You can wait 20 more years and super-duper-deep-NN-on-steroids, and hardware a
million times as big and powerful, would rediscover all of theoretical
physics.

Or you could inject some theoretical physics acquired by humans and make DNNs
smarter today.

~~~
cmehdy
I assume your 20 years is a guesstimate, and I do think it misses the point of
what Sutton's writing is. The trap here is that there's always to be more
computing in the future, so where do we draw the line? The idea is to think
differently now, for the pursuit of actual progress down the road. Which, by
the way, is exactly what people were doing about 40 years ago and what put
down more than the foundations for all the tricks we're pulling these days.

~~~
fizixer
I see what Sutton said as a "statistical learning and artificial intelligence"
researcher in line with what the authors of the physics paper presented as "an
application of learning research to computational science and engineering,
CSE, surrounding physics".

CSE researchers did not sit down and wait for AI researchers to learn the
bitter lesson before they resumed their work.

CSE research goes on independent of whether AI/GOFAI/ML has a winter, a
summer, an ice age, or a global warming.

It just so happens that in light of the recent progress of AI/ML, specifically
2012 to 2019, they see the utility of incorporating a tiny bit of ML to their
vast array of methods.

The paper shared in this thread is merely another attempt to advance such an
incorporation. If it doesn't pan out, they go back to doing CSE on physics
without any AI or ML.

~~~
cmehdy
That makes sense. As you said, those two sources don't have to be
contradicting each other if they complement instead.

------
jariel
Can someone with AI knowledge please clarify - does this mean we can build
'rules based systems' into AI to synthesise intelligence from both domains?

If so, this would be dramatic, no?

If you could teach a translation service 'grammar' and then also leverage the
pattern matching, could this be a 'fundamental' new idea in AI application?

Or is this just something specific?

~~~
samcodes
They model a system which they know to be constrained by a closed-form
equation called the Hamiltonian. They (cleverly, IMO) force the network’s
predictions to be constrained by the Hamiltonian, by choosing the right output
and loss function.

I don’t see a way to generalize this to the procedural rule-based systems you
describe, unless they too are governed by a fairly simple continuous function
Like the Hamiltonian.

I don’t know if it was “dramatic”, but it made me really happy.

------
castratikron
So can you teach a NN an equation of motion, and if so would it execute faster
than numerically integrating said equation? Could have impacts in physics
simulations although the accuracy might not be as good

------
athesyn
This sounds pretty terrifying.

~~~
civil_engineer
Careful there, athesyn. No need to offend our computer overlords.

