
Neural networks as non-leaky mathematical abstraction - george3d6
https://blog.cerebralab.com/Neural_networks_as_non-leaky_mathematical_abstraction
======
jordigh
> I find it rather weird that mathematics is usually taught to people on a
> historical basis.

It very much is not. If you tried to learn calculus the way Newton learned it
(with fluxions, or worse, awkwardly stated in the style of Euclid’s elements)
or the way Leibniz learned it (ill-defined infinitesimals), you’re gonna be in
for a bad time. We synthesise, reformulate, and reteach all the time. There’s
a lot more that has changed since Euclid’s time than merely the notation.

I skimmed the rest of the article, but the philosophical mathematical preface
is questionable.

~~~
dr_zoidberg
I'd like you to elaborate a bit more on the ill-defined inifinitesimals,
because that sounds a bit like the way I was taught calculus in 2006. But
maybe it's just that I studied engineering and we got the "light" part and not
too rigurous. Or maybe we were taught in a horrid way and didn't know better.

~~~
galimaufry
They are well-defined, but the usual framework that explains them ("non-
standard analysis") is extremely abstract.

If you are dealing with polynomials/algebraic geometry only then there is a
very simple, totally rigorous approach using nilpotents. And there is a newer
approach called "alpha calculus" that I know nothing about but here is an
example:
[https://arxiv.org/pdf/0807.3477.pdf](https://arxiv.org/pdf/0807.3477.pdf)

~~~
jordigh
They are well-defined _now_ , but when Leibniz and Cauchy were working with
them, they were not, and used carelessly could lead to errors. Their
"solution" was to just pretend those errors didn't happen ("don't do that,
then") instead of trying to come up with a consistent system of rules for
infinitesimals.

------
madhadron
Putting aside the weird preface about observation and mathematics, which
misses the point that science builds a theory, which is a formal system that
recapitulates the relations among observations, the neural network stuff shows
very deep lack of understanding. For example:

> an autoencoder (AE) is basically one of the easiest ones me to explain,
> implement and understand

Then a list describing it. But that list does not constitute an explanation
unless you have a pretty good intuitive understanding of linear algebra and
function composition.

> So essentially, neural networks become a sort of mathematical abstraction
> that isn’t very leaky.

I think what the author really means by "leaky" is "not opaque." Mathematical
structures that you can analyze in a well defined way (not opaque/"leaky")
like integrals vs mathematical structures that the tools to dig into in a deep
way don't exist yet (opaque/"not leaky").

Except that such understanding of neural nets is a major current research
program.

So to all the youngsters who might be looking at this, this article is the
ravings of a crank who doesn't know what he's talking about.

~~~
gravypod
I took this more to mean the author liked NNs because they were not
transparent. To model something with an NN you don't need to know all of the
mathematical constructs that go into the NN.

------
FlyingAvatar
This entire concept seems backwards to me.

The reasoning for author's premise that sigma summation and integrals are
leaky abstractions is unclear.

The two are their own independent abstractions; while perhaps complicated to
fully understand, they will work exactly as designed and, provably so, from a
logical standpoint. If they were leaky, they would display some kind of
shortcoming that didn't allow them to convey parts of the core idea they were
created to represent.

On the other hand, an NN implementation of any sufficiently complex concept is
almost guaranteed to be leaky as it is too complicated to be provably correct,
and will have too great a test surface to verify exhaustively. There are
likely to exist edge cases where the NN fails outright that will be never
discovered until used in a specific scenario. That seems leaky.

Can someone set me straight on what the author was trying to convey?

~~~
hansvm
The neural network counterpoint feels odd, but with respect to summations and
integrals as leaky abstractions I think the author's point is that deploying
them effectively as tools typically depends on a mountain of hierarchical
knowledge on top of which elementary calculus is built. I don't know that this
is necessarily a requirement -- one could conceive of a world where integrals
are taught purely in terms of methods for moving in and out of that concept
domain (analogous to the train/predict interface common in ML), and where
people are simply taught algebraic rules for manipulating integrals entirely
in the realm of integrals without relying on their pedagogical grounding in
limits (sort of like how we have high-level descriptions of layers, drop-out,
and other concepts that are at the same level of abstraction as neural
networks themselves). I think the author claims that this latter approach is
rare in mathematics, and it seems they make the stronger claim that
mathematical concepts aren't generally amenable to that kind of strategy with
integrals and summations as specific examples of where it's impossible. FWIW,
I disagree with that assertion, but nevertheless it seems to be what the
author is trying to say.

~~~
kevinventullo
_one could conceive of a world where integrals are taught purely in terms of
... algebraic rules for manipulating integrals entirely in the realm of
integrals without relying on their pedagogical grounding in limits_

This is absolutely how I was first taught integral calculus.

    
    
      ∫x^n = x^{n+1}/(n+1) + C
      ∫e^x = e^x + C 
      ∫(f+g) = ∫f + ∫g
    

etc. Only later did I really learn the details of limits and the rigorous
epsilon-delta underpinnings in real analysis.

I used to think this was a bad thing, but now I’m not so sure. The word
“calculus” does mean a system for calculation.

------
andrepd
_> Tell a well educated ancient Greek: “ 95233345745213 times 4353614555235239
is always equal to 414609280180109235394973160907, this must be fundamentally
true within our system of mathematics”… and he will look at you like you’re a
complete nut._

What is this supposed to mean? I honestly don't understand. They most
certainly realized that although tedious to compute it is just as well-defined
as 2*3=6. What is he implying here?

~~~
kryptiskt
Archimedes dealt with much bigger numbers in The Sand Reckoner (he worked
through his numbering system up to 10^(8*10^16)). He certainly wouldn't flinch
at those.

------
pwdisswordfish2
What a load of nonsense.

When I prove, say, Cauchy's theorem about groups, it doesn't matter which
concrete group I'm operating on to prove it. If I write a sigma to sum some
vectors, it doesn't matter if the vectors are coordinates in 3D space,
functions defined on an interval, or just numbers. Mathematical abstractions
are pretty much the only ones that _don 't_ leak.

What, that you have to understand summation to understand what the sigma
refers to? Well, duh. That doesn't make the sigma a leaky abstraction, because
the sigma is not an abstraction at all! Not even a trivial one. The sigma is
_notation_. And it's hardly news that you have to understand the concepts
behind a notation to understand notation itself.

The author seems to misunderstand what an abstraction actually is.

~~~
countertointel
This comment is counter-intelligence.

All signs refer directly to abstractions in the mind that can be compiled into
a variety of useful forms. Yours does not take precedent in this case, as it
is plain to see that sigma is under specified due to the huge range of
concepts it’s practical usage covers. Your attempt to prescribe that which
should be prescribed is nothing but authoritarianism.

~~~
MrEldritch
What on Earth are you talking about?

------
bonoboTP
A lot of fields in math expose a kind of API that abstracts away details quite
well.

You can use SVMs without fully understanding the specific QP optimization
algorithm underneath, you can understand the kernel trick without all RKHS
math etc.

Or in classical vision, you can use keypoint extraction without caring about
the details.

Or in linear algebra you can understand what SVD is and does without knowing
the detailed steps of computing it.

I don't see neural nets as very special in this.

------
sudoscript
So abstract mathematics are leaky, and applied mathematics are not (you don’t
have to know how they work to apply them, e.g. regression).

------
incompleteness
Gödels incompleteness theorem guarantees all abstractions are leaky.

Try explaining the intrinsic meaning of written language in written language.
You can't close the loop without making the result trivial.

