
Foundations Built for a General Theory of Neural Networks - LogicRiver
https://www.quantamagazine.org/foundations-built-for-a-general-theory-of-neural-networks-20190131/
======
nora4
Deep Learning is currently an empirical science guided by intuition of
practitioners. A main principle in experimental sciences is that a theory
without predictive power is not considered a full-fledged theory. As such,
unless they are interesting predictions coming from their theory (rather than
only barely justifying existing empirically observed phenomena), this is just
speculative theory that I would not use the phrase "Foundations Built" for.

As an example of this general litmus test for a theory see e.g. Eddington's
confirmation of GR:
[https://en.wikipedia.org/wiki/Tests_of_general_relativity#De...](https://en.wikipedia.org/wiki/Tests_of_general_relativity#Deflection_of_light_by_the_Sun)
. If there are hitherto unknown phenomena in DL predicted by this theory then
I'd stand corrected and concede that there may be something to these theories.

~~~
fizx
Read this as "some foundations built". Even that is much-needed progress.

~~~
hyperpallium
Yeah, it's not foundations like elementary or axiomatic.

An actual theory of ANN and then of NNN will be of more momentous import than
of any previous phenomena, will usher in a fundamentally transformative age
where we understand ourselves, and will surely require a fundamental
breakthrough in pure mathematics... probably a great many.

------
zazen
Here are the two papers linked towards the end of the slightly rambling
article:

[https://arxiv.org/abs/1705.05502](https://arxiv.org/abs/1705.05502)
[https://arxiv.org/abs/1810.00393](https://arxiv.org/abs/1810.00393)

~~~
andbberger
See also:

[https://arxiv.org/abs/1606.05340](https://arxiv.org/abs/1606.05340)

------
geuis
The current state of neural networks is somewhat analogous to how cathedrals
were built in the medieval era.

Master architects had a good understanding of then-ancient geometry and
applied that knowledge plus knowledge of arches from Roman times to create the
great cathedrals found throughout Europe.

The architects didn't have a deep understanding of material science (stone)
and other forces that are taken into account in the modern era, but they had
enough knowledge from experience and experimentation combined with relatively
simple geometry to create strong structures that have stood the test of
centuries.

The analogy between that and modern practical AI/neural network
implementations is that experts in the field have good basic knowledge of how
to construct neural networks but there isn't yet an advanced theoretical
knowledge of the science that allows for precise predictions and modeling.
i.e. Experts are still at the level of experimentation and noting the results
and getting useful results without a super-deep understanding of how neural
networks produce useful results.

Hypotheses have been/are being proposed and tested right now. Expect that as a
deeper level of experimental results are achieved we'll soon have a sound
theoretical model that will lead to profound advances in the field.

~~~
bra-ket
I think the current state of the art in AI is more comparable to the first mud
huts, when they just started throwing mud on a wall to see if it sticks

~~~
geuis
I dunno. I think we’re a bit further along than that. Baked bricks at least.

------
the_grue
I think a important advance is being able to build neural networks in a
modular fashion by constructing a toolkit of components and techniques one can
reasonably expect, when combined, to produce certain result. As the field
advances, this toolbox will become richer and better defined and delineated,
forming basic building blocks. Just like functions, loops and virtual methods
came out of the chaos of assembly.

------
superpermutat0r
I love the fact that I'm seeing how stuff works. Before theory, people built
huge buildings, cars, airplanes, magical electrical circuits etc.

Neural networks are the same thing. Bunch of heuristics that will later be
proven in theory. Yet still work incredibly well. Like cathedrals did without
any mathematics.

Says a lot about how tinkering with heuristics is powerful.

~~~
throwawaymath
But there was a stunning amount of mathematics used for all the feats of
engineering you're talking about, even early on.

~~~
superpermutat0r
Well, I'm pretty sure there was not even Euclid's geometry thousand years
after he invented it in any architecture recipe book, yet they built brutally
large structures.

We have 13th century books from architects that built cathedrals filled with
recipes with 0 mathematics, all heuristics, rules.

~~~
throwawaymath
Are you sure? For example, the Florence Cathedral (Il Duomo) required
nontrivial, novel mathematics. Brunelleschi didn't succeed in that effort
simply through iterative engineering.

I think in general it would be more appropriate to say that fundamental theory
tends to predate its applications, and then the theory and applications evolve
as one informs the other in a positive feedback cycle. Likewise a lot of
improvements to rocket science are due to iterative engineering, but we
couldn't get off the ground (literally) without calculus.

~~~
zazen
Brunelleschi was working at the height of the Renaissance, in the early 1400s.
They weren't short of book learning in renaissance Florence, but that's why
it's called the renaissance. I imagine medieval cathedral building, e.g. Notre
Dame starting more than 200 years previously, might have looked very
different. They couldn't build a dome like Brunelleschi back then.

------
rdlecler1
I've beaten this drum before but another area to look at is network
connectivity. When visualizing a neural network we typically see a fully
connected network from one layer to the next, but if you set the value of many
weights to zero you'll find that they have absolutely no effect on the
function of the network. So basically we're looking at a lot of spurious
interactions and this just clouds our thinking. In fact the network topology
is what's driving network function here but we're not doing a good job
exposing that intuitively. Computational research on Gene Regulatory Networks
--which are also modeled with the same NN math--have shown how network
topology is really the key driver of function.

For anyone who is interested, I published a paper during my PhD that has about
195 citations. It shows that under an evolutionary process that Artificial
Gene Regulatory Networks select out spurious network interactions, leaving you
with what's functionally necessary (Also see the supplementary). I came from
AI before starting my PhD and believe that this could be important for
building a foundation for NN as well.

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/)

------
lotaezenwa
Proof of the universal approximation theorem [0] for those interested

[0]
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf)

