
Machine learning is not just glorified statistics - xTWOz
https://towardsdatascience.com/no-machine-learning-is-not-just-glorified-statistics-26d3952234e3
======
wodenokoto
I've been meaning to ask if "townrdsdatascience" is a serious website, but I
guess this answers it.

It seems in order for the argument here to make sense, you have to say that
predictive modelling is outside the field of statistics, and that "a class of
computational algorithms" (presumably including classic ML algorithms such as
decision trees, random forest and support vector machines) are also not
statistical algorithms.

The author gets close to having a point when saying that in reinforcement
learning you may not even have a dataset, so what is there to make statistics
on? Well, you make statistics on the generated data.

I think he is right once he starts talking about approach and not knowing
"variance of a population, or to define marginal probability" is not necessary
to perform ML. I mean, you can perform ML without knowing what variance is,
just as well as you can perform psychology experiments without knowing the
variance, but I get his point:

Classical statistics is very much focused on explaining a dataset, whereas ML
is very much focused on making future predictions. And you can combine and
build a lot of predictive models, without the knowledge of the other half of
statistics, and vice versa. But this argument is like saying topology isn't
math because it is about shapes and not numbers. Or that NLP is not neither
machine learning or statistics, because it is about language.

~~~
Buttons840
> Classical statistics is very much focused on explaining a dataset, whereas
> ML is very much focused on making future predictions.

True enough. Could you slightly change this and say statistics is about
understanding the past and ML is about predicting the future? The only way to
predict the future is to understand the past, or be very lucky.

~~~
Spivak
_One branch of statistics_ is about understanding the past. Statisticians are
very much interested in predicting the future and have methods that look
nothing like ML.

------
pathseeker
Mostly just arguing that it's special and not 'statistics' because it's
statistics being calculated at a much larger scale in automated fashions.
Seems like unnecessary gatekeeping just for the purpose of inflating egos.

~~~
stonogo
His bio indicates he does 'ML @ Harvard'. I would imagine there's more than
just ego riding on this.

~~~
typomatic
Your suggestion is that someone who did a degree specifically in machine
learning at Harvard would be less likely to be defending their discipline out
of a sense of ego?

~~~
stonogo
Not merely a suggestion; I'm stating outright that when someone's career (both
the value of an acquired scholarly degree and the earned income within the
field) depends on a field being regarded as a valid specialization, that
person is going to put effort into reinforcing that perceived validity.

~~~
typomatic
Oh okay! I thought you were suggesting the opposite (which is nonsense).
Thanks!

------
mrfredward
I would consider machine learning to be an application of statistics, in much
the same way that mechanical engineering is an application of physics. The
foundations of mechanical engineering are rooted in physical concepts, but
mechanical engineers have formulas for all sorts of things, like calculating
the fatigue life of gears, that you can't get from pure physics because they
are empirically derived curve fits, rather than natural laws. The science is
more pure, but the application of it is what's powerful in the real world.

So in a sense I agree that "machine learning is not statistics," but I
strongly disagree with the tone of the article, which is "we're better than
statistics." Don't shit on the shoulders you're standing on.

~~~
tabtab
It may not be an "application of statistics" in the direct sense, but that
depends on how one defines statistics, which brings one back to the original
problem.

Statistics and ML often have similar goals, but ML emphasizes computational
efficiency over trace-able accuracy. Thus, I view each field as having
different weights on the same sub-goals.

------
chubot
Here's one way I think of it: in statistics vs. machine learning, there's a
difference in your goals, which is reflected in a difference in your models.

\- In statistics, the goal is to explain something. The models have few
variables, and each variable should mean something, like the influence of the
person's age or sex on the outcome.

\- In machine learning, the goal is to make something work. This is apparently
better done with millions of variables (neural network weights), and each
variable is opaque and means nothing by itself.

He hints at this distinction in the blog post but it's not entirely clear.

~~~
sideshowb
That's the difference between inference and prediction. Both statistics and
machine learning can do either of those things.

~~~
chubot
Can you give some examples of both?

~~~
Spivak
Statistics being used for prediction:
[https://fivethirtyeight.com/features/how-our-march-
madness-p...](https://fivethirtyeight.com/features/how-our-march-madness-
predictions-work/)

Statistics being used for inference:
[https://fivethirtyeight.com/features/stephen-curry-is-the-
re...](https://fivethirtyeight.com/features/stephen-curry-is-the-revolution/)

~~~
lottin
As others have pointed out, in statistics the goal is understanding something.
Once you have understood it you can predict its behaviour. However the reverse
is not true.

~~~
thousandautumns
This isn't true though. There are ton's of uninterpretable model methodologies
in classical statistics that have little to no ability to allow for
understanding but are aimed entirely at accurate predictions. Where is this
narrative that statistics is only interested in understanding coming from?

~~~
srean
> Where is this narrative that statistics is only interested in understanding
> coming from?

You may have heard of this guy called Fisher he might like to have a few words
with you. He says he fathered modern Stats, that he connected what was a bag
of recipes to math. Many seem to agree [0] despite the fact he does not seem
to be the most pleasant bloke around. The British queen seemed awfully
impressed with him though, YMMV.

[0] "a genius who almost single handedly created the foundations for modern
statistical science" \-- Hald, Anders, A History of Mathematical Statistics.

~~~
thousandautumns
I'm sure you were attempting to make a point, but you failed. But if you are
trying to claim that R.A. Fisher only cared claims that statistics is only
about inference, the point is moot because:

1\. R.A. Fisher isn't the Almighty Statistical God just because he did laid a
lot of foundations in early statistics.

2\. Fisher has a long history of seriously stupid personal-beliefs including
but not limited to: refuting anything and everything relating to Bayesian
statistics, attempting to discredit the studies done showing a link between
smoking and lung cancer, and advocating for eugenics and the idea of
superiority/inferiority between races.

So just because Fisher may have claimed something doesn't make it so.

------
cwyers
Machine Learning is not just statistics. David Donoho spells out the history
of the whole thing in his 50 Years Of Data Science:

[https://courses.csail.mit.edu/18.337/2015/docs/50YearsDataSc...](https://courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf)

The clearest statement of the difference I've found is Leo Breiman's
"Statistical Modeling: The Two Cultures:"

[https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...](https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726)

The abstract has a succinct explanation:

> Abstract. There are two cultures in the use of statistical modeling to reach
> conclusions from data. One assumes that the data are generated by a given
> stochastic data model. The other uses algorithmic models and treats the data
> mechanism as unknown. The statistical community has been committed to the
> almost exclusive use of data models. This commitment has led to irrelevant
> theory, questionable conclusions, and has kept statisticians from working on
> a large range of interesting current problems. Algorithmic modeling, both in
> theory and practice, has developed rapidly in fields outside statistics. It
> can be used both on large complex data sets and as a more accurate and
> informative alternative to data modeling on smaller data sets. If our goal
> as a field is to use data to solve problems, then we need to move away from
> exclusive dependence on data models and adopt a more diverse set of tools.

The "stochastic data model" camp covers most of what people think of when they
think of traditional stats -- everything from OLS through to more
sophisticated techniques. Things like neural networks and random forests are
algorithmic techniques that make no assumptions about or conclusions about the
distribution of the underlying data.

~~~
thousandautumns
The premise of the abstract is flawed though. Who says statisticians aren't
using algorithmic models? Perhaps the author doesn't but random forests,
clustering, PCA, gaussian processes, and even neural networks are standard
fare for many statisticians.

~~~
cwyers
I mean, the author certainly used random forests, seeing as he invented them:

[https://www.berkeley.edu/news/media/releases/2005/07/07_brei...](https://www.berkeley.edu/news/media/releases/2005/07/07_breiman.shtml)

~~~
thousandautumns
I can't help but notice he died in 2005. Perhaps the premise of the two
cultures was more relevant a few decades ago, but in my more recent
experience, its absolutely not the case.

~~~
cwyers
There is certainly cross-pollination between the two, and practitioners these
days often adopt both. But the snide "machine learning is just statistics
practiced by people who don't know what they're doing" comments you tend to
see on HN ignore that machine learning was something that sprung up (largely
in CS departments) to address challenges that statistics departments weren't
addressing.

------
geodel
"..I get it — it’s not fashionable to be part of the overly enthusiastic,
hype-drunk crowd of deep learning evangelists."

I think it is very fashionable. The people who are not part of this crowd are
perhaps older and boring like me.

------
gaius
_Yet, if you had asked me, or most of the students in that class, how to
calculate the variance of a population, or to define marginal probability, you
likely would have gotten blank stares. That seems a bit inconsistent with the
claim that AI is just a rebranding of age-old statistical techniques._

So the author is asserting that... ML is statistics for people who don’t know
statistics? I wouldn’t necessarily argue with that, but I wouldn’t brag about
it either...

------
thousandautumns
I don't love the term 'mansplaining', but if there is a term that describes
essentially the same idea but in the context of academic fields, its exactly
how I would describe the central thesis of this blog post and a trend I've
encountered frequently in the last couple years. There is a rising tide of CS
people who have just latched onto the hype of data science, and now go around
letting statisticians know that no, they aren't _actually_ interested
prediction, they don't _actually_ know how to work with large data, and don't
_actually_ work with non-parametric methods. It certainly comes as a shock to
all of the statisticians in the world who have indeed been working on these
types problems for a long time now.

~~~
srean
> It certainly comes as a shock to all of the statisticians in the world who
> have indeed been working on these types problems for a long time now

Yup on high dimensional data of dimension as fantastic as 12. I feel bad for
them though but they have only themselves to blame -- got too comfortable
within their small world and lost touch of what the next set of interesting
problems are.

Its only after getting kicked in the nuts that I see a course correction and
that's enriching both ML as well as Stats.

If one computes stats on 600 data points with 10 dimensions and feels king of
the hill, they can continue, but there is likelihood that some one else will
be eating your lunch and you will be left behind. Sadly enough, this has
already happened and is quite evident if one steps out of the stats bubble.
Statistics could have been what machine learning and datamining is now, been
the main driving force, the owner of initiative. On the contrary other
communities are using statistics and probability motivated approaches but
engineering them well to grab (funding) attention, well deserved in my
opinion. It is them who got the ball rolling again.

[https://news.ycombinator.com/item?id=17687303](https://news.ycombinator.com/item?id=17687303)

~~~
thousandautumns
Your response is so entirely off base I don't know where to begin.

> Yup on high dimensional data of dimension as fantastic as 12

Who says that has been the limit of classical statistics.

> If one computes stats on 600 data points with 10 dimensions and feels king
> of the hill, they can continue, but there is likelihood that some one else
> will be eating your lunch and you will be left behind.

Again, why do you have this impression? You clearly have no experience in the
field if this is what you think statistics is. Unless your intent is to simply
construct strawman arguments.

> Statistics could have been what machine learning and datamining is now, been
> the main driving force, the owner of initiative.

This is entirely based on the assumption that machine learning and datamining
and statistics are distinct and separate, which isn't the case and is my
entire point.

> On the contrary other communities are using statistics and probability
> motivated approaches but engineering them well to grab (funding) attention,
> well deserved in my opinion. It is them who got the ball rolling again.

Seriously, wtf are you talking about?

~~~
threatofrain
You basically went line by line to take too many words to say “you’re wrong”.
That's how much content is left over after you filter out the insults.

------
wrs
“The world’s frontier of technological progress and innovation.” What does
that even mean? All other progress and innovation is somehow trailing behind
ML? What?

------
azinman2
What a silly/confused article. To say it’s not stats because someone can
implement a GAN without deep stats knowledge is ignoring everything that’s
lead to that point. It’s like arguing there’s no assembly in programming
because I can write and understand hello world in Python. Your fancy GAN is
not coming from a vacuum. It’s been built on top of a ton of math+stats, and
then you end up implementing it in tensorflow which itself is a bunch of
abstractions done for you.

------
jawjay
By definition machine learning is Statistics + Decision Theory. Statistics
simply tells you what information you have about the world, not what to do
with that information.

------
olliej
Yes it is. Having a bigger model state makes it more complex. It doesn’t make
it not statistics.

------
phonebucket
Quote:

“One of our assigned projects was to implement and train a Wasserstein GAN in
TensorFlow. At this point, I had taken only an introductory statistics class
that was a required general elective, and then promptly forgotten most of it.
Needless to say, my statistical skills were not very strong. Yet, I was able
to read and understand a paper on a state-of-the-art generative machine
learning model...Yet, if you had asked me, or most of the students in that
class, how to calculate the variance of a population, or to define marginal
probability, you likely would have gotten blank stares. ”

I agree that the GAN might be implementable without much stats knowledge, but
I would be very surprised if someone who did not know population variances and
marginal probabilities would be able to follow the Wasserstein GAN paper. Just
see for yourself:
[https://arxiv.org/abs/1701.07875](https://arxiv.org/abs/1701.07875)

------
deehouie
This is a terrible post, coming from someone who doesn't know statistics to
claim something is not statistics.

~~~
gaius
_his is a terrible post, coming from someone who doesn 't know statistics to
claim something is not statistics._

But the author went to Harvard, doncha know. _Harvard_.

------
RedComet
Yes, it is.

~~~
tempodox
No, it's worse. It's statistics on steroids used by people who don't have a
clue about statistics.

~~~
basch
>At this point, I had taken only an introductory statistics class that was a
required general elective, and then promptly forgotten most of it.

------
lottin
This sentence sums it up perfectly:

> In many cases, these algorithms are completely useless in aiding with the
> understanding of data and assist only in certain types of uninterpretable
> predictive modeling.

Statistics is a crucial component of the scientific method. It is the tool
with which scientists check whether their theories agree with empirical
evidence.

Machine learning is about building mathematical models that are apparently
"right", in the sense that they have predictive power, but that don't
necessarily improve our understanding of the data-generating processes
involved.

------
myth_buster
I was listening to Lex Fridman's AGI Podcast where Vladimir Vapnik came to
talk about Statistical Learning[0] and his take on deep learning as compared
to statistics/mathematics was dismissive. To paraphrase from memory, the
problems deep learning is solving is not hard enough.

There has been a rift between the statistical community and ML community for a
while and I see it similar to the arguments one makes when it comes to
sciences vs engineering.

[0] [https://lexfridman.com/ai/](https://lexfridman.com/ai/)

------
justfor1comment
I have to frequently work with Data Scientists to learn their ML models and
scale them in our production environments. A lot of times the models are not
even statistics, it's just linear algebra. They have a tendency to go for
approximate algorithms when an exact algorithm could be written. I largely
blame the company for this and not the data scientists. Every company wants to
be in the ML game but may not have the volume and variety of data to warrant a
use case. When your favorite hammer is ML then every data problem is a nail.

------
srean
The post makes two main claims

"Machine Learning Does Not Require An Advanced Knowledge of Statistics"

Lets take it at its face value, even then it does not preclude ML from being
glorified stats. Depending on how advanced the 'advanced' is in that statement
I would agree with it. To use ML tools well you do need some familiarity with
understanding how uncertainty effects the results and that is as up one can be
statistics alley. I say this although I am firmly in the ML side of the tribe.
BTW I would even claim this

"[Practicing] Machine Learning Does Not Require An Advanced Knowledge of
Machine Learning"

The other claim that the post makes is

"Machine Learning = Representation + Evaluation + Optimization"

Whoah! big blind spot there. Dealing with and reasoning about uncertainty,
generalization is a big deal in ML. Sure, it helps to argue that ML and stats
is different if one ignores that bit.

I do think ML and Stats is different but not for those reasons.

Yeah ML brings to bear some tools that card holding statisticians have
traditionally not used in anger before, for example, advanced and large scale
optimization, algorithms, data structures.

Using new tools to address the same question does not qualify as a deep
difference in my books. For example, even the fields of optimization and
algorithms themselves use tools that are different from what the tools were 50
years ago. I think the main difference is in the questions that ML and Stats
wants to answer, and here there indeed are differences.

Stats (barring edge cases) is primarily interested in going from a sample to
making claims about the population, or making claims about something via a
claim about the population. ML is primarily interested in going from a sample
to another sample. (Do note sample is a collective noun.) Now, there have been
statisticians (fewer in number) and a body of statistics literature that has
focused on prediction as opposed to parameter recovery, but that's not main
stream in Stats.

And finally in pseudo-quotes "I managed to train a model without knowing what
variance is. I know variance is statistics. ML is not Statistics ... QED ". Ah
I see, nevertheless, not what I would call a brilliant case of logical
deduction".

------
pdimitar
If you start your article with the assumption that ML is being a subject to
memes because "it's not fashionable to like it" \-- which also includes
"liking" it, as if it's some kind of aesthetics art... Then your argument
falls apart from the get go.

Perhaps the author should use ML itself to find out why people started mocking
it!

