
The revolution of machine learning has been exaggerated - benryon
http://nautil.us/issue/78/atmospheres/are-neural-networks-about-to-reinvent-physics
======
FiberBundle
"The widespread misimpression that data + neural networks is a universal
formula has real consequences: [...] in what students choose to study, and in
what universities choose to teach."

I think this is a problem that deserves more attention. A large portion of CS
students nowadays choose to focus on Machine Learning, many brilliant CS
students decide to get a PhD in ML in hope of contributing to a field that
will continue to develop at a pace similar to the progress we have seen in the
last 7 years. This view, I think, is not justified at all. The challenges that
AI faces such as incorporating a world model into the techniques that work
well will be significantly harder than what most researchers openly admit,
probably to the detriment of other CS research areas.

~~~
thewarrior
It may not be ideal for fundamental research but demand for ML practitioners
is only likely to increase over time.

ML is pure alchemy for a business operating at scale. It’s as if a coal plant
could turn its waste emissions into profit. You have this “data exhaust” from
all the activity happening within your system that can be used to optimize
your system Atleast a few percentage points beyond what is other wise
possible. A team of 5 ML engineers can improve an ad targeting system by 5 %
and if the company is google that’s billions.

ML creates feedback loops of improvements in product that improve usage that
lead to more data which further strengthens the moat a business has.

It totally makes sense to jump on this train. It won’t solve AI but will make
a lot of people wealthy optimizing systems.

~~~
heavenlyblue
>> A team of 5 ML engineers can improve an ad targeting system by 5 % and if
the company is google that’s billions.

Not going to pick up on your arbitrary 5% constant here, but please elaborate
how these ML engineers are any different from anyone with a quantitative
background?

~~~
bumby
I think that's the key here. It's not ML magic that's driving success, it data
literacy. Steve Levitt (of Freakanomics fame) said his advice to students
would be to ensure they have base knowledge in understanding data analytics
regardless of their domain. ML is just the sexy subset that gets all the
attention

~~~
ghaff
And ML is a tiny part of the whole data science toolchain and process. Getting
the data to the point where you can use it for something interesting is
probably the hardest part.

~~~
thewarrior
My point was that cutting edge ML can add a bit more on top of what data
literacy can achieve. At scale that’s worth a lot.

------
personjerry
That's because the media hype is completely different from the industry hype.
This article is a form of media hype.

The pragmatic, already realized effects of machine learning are mostly in
improving existing systems, especially classification systems, and have
increased metrics by big percentages at big data companies, and more
applications are being found every day. _That_ is why real demand and interest
is so high for ML.

These generative, speculative models are interesting experiments by
researchers but inconsistent and far overstudied by the media, because "AI
invents new physics" is more understandable and sexier than "AI improves lead
generation by 6.8%"

~~~
spamizbad
I would say the industry definitely has its own hype. And honestly, I think
the news media is a trailing indicator on that hype.

For example, there's already an ongoing replication crisis that's been ongoing
for several years with ML papers in computer science. I've spoken with people
in the industry who have even speculated that there's straight-up academic
fraud taking place. If you get a splashy, sexy finding you can get scooped up
by a FAANG-like company with a fat salary before anyone gives your work a
second look.

~~~
omicron1
From working with many ML researchers, it's not speculation. Some aren't very
proud, but if you want to get ahead and stay competitive, you have to publish
your un-replicable results. The results aren't usually completely fake, but
they depend on enough dishonest sleight of hand that they might as well be.

------
buboard
These articles (and a lot of what g. marcus writes) are attacking strawmen. I
ve never heard no one claiming that NNs will invent new theories and i dont
think that 2008 article is widely read. But, for things that are hard both
computationally and theoretically, like protein folding, NNs may really be
revolutionizing the field even if we don't know how they do it. Scientists do
not buy-in foolishly into every AI hype. The problems lie with VCs and funding
bodies, which are indeed swayed by adding "AI" to every kind of proposal.
That's a separate problem, though

~~~
skywhopper
That's the whole point... at the very beginning the post cites articles that
suggest or even outright say that AI is coming up with new theories (or in the
CS world, see all the articles and bloviators who predict AI will replace
programming as a job). Regardless of what the scientists in the field would
say one-on-one, the outright-wrong portrayal of ML in the press, egged on by
VCs out to flip their investments, is causing problems for anyone who wants to
actually make progress in their field, or recruit students honestly.

~~~
buboard
> articles that suggest or even outright say that AI is coming up with new
> theories

I would contest that there are widely read articles saying those things. Yes,
surely someone publishes those , but they are generally dismissed. the field
is conscious of its hype (e.g. check r/machinelearning), and i don't think
students are stupid either

------
deminature
This is a perfect example of the 'AI effect' in action:

>"It's part of the history of the field of artificial intelligence that every
time somebody figured out how to make a computer do something—play good
checkers, solve simple but relatively informal problems—there was a chorus of
critics to say, 'that's not thinking'." AIS researcher Rodney Brooks
complains: "Every time we figure out a piece of it, it stops being magical; we
say, 'Oh, that's just a computation.'"

[https://en.wikipedia.org/wiki/AI_effect](https://en.wikipedia.org/wiki/AI_effect)

~~~
feral
We really need another name, which refers to people uninformedly quoting 'the
AI effect'.

How about "the AI effect effect"?

This happens all the time:

\- X suggests that success at a particular task is a good proxy to general
intelligence.

\- Years later Y solves that task with a clever hack and lots of computation,
but no general intelligence.

\- The media breathlessly suggests we've made progress on general
intelligence.

\- Z points out we haven't really.

\- Someone says "Oh that's the AI effect!", by which they mean the goalposts
have been unfairly moved on AI.

No, they have not. There's something called general intelligence, humans can
do it, and Deep Blue cannot, and it's fine to say that Deep Blue beating
humans at Chess was not thinking.

It just turned out that Chess wasn't as good a proxy to thinking as we
initially thought, and mentioning "The AI effect" just clouds the discussion.

~~~
visarga
> There's something called general intelligence, humans can do it, and Deep
> Blue cannot

There's no such thing. General is a strong word. Humans can't for example
handle more than 7 objects in working memory. We have an inbuilt limitation to
how much complexity we can grasp. Programmers know what I mean.

General intelligence requires a general (infinitely complex and challenging)
environment. Without it it will always be a specialised intelligence. Human
intelligence is specialised in human survival, as individuals and part of
society.

~~~
feral
As you appeal to Programmers: do you believe quicksort is a general sorting
algorithm?

Would it still be a general sorting algorithm if someone had just developed it
and used it solely to sort numbers between 1 and 100?

It would be.

Why do you think you need to a general environment to develop a general
algorithm?

No one is arguing human rationality is unbounded. But our intelligence
generalizes, in a practical sense, to a great many more tasks than any
computer algorithm I'm aware of.

Mainstream psychologists believe in general intelligence. They even attempt to
measure components of it, with good prediction of performance on unseen tasks.

I think there's a big burden if you want to argue it doesn't exist.

~~~
visarga
> Why do you think you need to a general environment to develop a general
> algorithm?

Because intelligence is not intrinsic to the agent but the result of the agent
trying to maximise rewards in an environment. In the end the driving force is
survival - the agent needs only to survive, any method would do. So as soon as
it overcomes a challenge it stops evolving and turns to exploiting. And the
list of challenges that threaten survival does not scale to infinity. Anytime
you think about intelligence you need to also think about the environment and
the task otherwise it is meaningless.

~~~
omnicognate
You state this as if it's fact but it's a point of view, not necessarily wrong
but not proven either. More to the point, it's circular: "intelligence is not
intrinsic to the agent but the result of the agent trying to maximise rewards
in an environment" is a statement that the human mind operates on the same
principles as ML algorithms. If you assume this, then not surprisingly it
follows that ML algorithms can in principle do anything the human mind can do.
Not everybody agrees and the question has not been empirically settled.

------
dfan
I assume from the title that this is either written by Gary Marcus or quotes
him extensively. (I just checked and was correct.) It would be nice to hear
this sort of critique from more than one person.

~~~
ycombinete
Is he a good source of critique? Sincere question, I don’t know anything about
the field.

~~~
high_derivative
He is rather controversial in machine learning circles. I would say he is
essentially right about the fact that a lot of ml work is crap. Like most of
anything is crap, so that is not a groundbreaking insight and it comes off as
somewhat grandstanding to keep saying that.

He has however really turned this into a personal brand which many suspect he
is using to advertise his startups. Many people feel he is sort of
disingenuous about the actual progress being made in computer vision and NLP,
even if there is many ways to go.

To answer your question: You can read a single piece of his and maybe get some
value from that, but he keeps rehashing the same points to sh*t over different
new work, and there is not much to be learned from that after a while.

Disclaimer: I did not read the above article in full, I am just familiar with
Gary Marcus + work in ML industry.

Edit: Keep expanding this but it got me thinking. Earlier in my career, I was
also eager for the 'gotchas', for being able to point out why something is
crap, exaggerated, cannot do what it promises, and so forth. As I outgrew this
impulse to focus more on what is good about some work, I began spotting it on
new students a lot. They would come into seminars and smugly crap all over
recent papers.

I have since come to the conclusion that it is useful to have realistic
internal assessment of progress in your field, but that sharing your harshest
criticisms is not necessarily a good way to look smart or make progress.

It often turns out that while feeling clever about realizing limitations of
some work, the field just moves forward. The people you just criticized
leapfrogged while you were busy thinking about how to one-up them.

~~~
t_serpico
It's easy to critique a paper but hard to pinpoint its value in the broader
context of a field.

------
geoalchimista
A recipe for the typical media hype "AI learns physics" discovery

1\. Find a well-known classical problem with a battle-tested deterministic
model representation

2\. Generate synthetic data using the model

3\. Train a 1-layer neural network to score 99% in predicting the synthetic
data (even if the deterministic model can score 99.999% but let's ignore the
minor difference)

4\. Claim AI "learns" the underlying physics and does a better job than
classical models

5\. World dominance

~~~
knzhou
This is exactly it. Earlier this year there was a hyped up paper [0] going
around claiming that they could derive physics with AI. On inspection I found
that what they actually did was this:

1\. Take one simple physics equation, such as F = ma

2\. Generate 10^6 tuples exactly satisfying this equation (F = 6, m = 2, a =
3), with absolutely no redundant, erroneous, or extraneous information

3\. Feed these to a primitive "AI" system which brute-force tries the simplest
relations that could relate the three provided variables (m = Fa, a = Fm, F =
ma, ...)

4\. Recover F = ma, declare physicists obsolete

It's such a poor imitation of the real process of discovery in science that
it's frankly insulting. When people hear AI is involved with something they
turn their brains off.

0: [https://arxiv.org/abs/1905.11481](https://arxiv.org/abs/1905.11481)

------
lettergram
In this particular case I think the headline of the title is probably better
than the one on HN.

Yes, I’m many fields ML has been exaggerated, but it’s just a matter of time.
Most companies aren’t using ML effectively yet.

But I know for a fact when you improve NLP, you _can_ improve almost all white
collar jobs. NLP the past two years has been crazy, and we haven’t seen
anything yet.

~~~
dmreedy
I agree that there has been some impressive progress on NLP lately; am curious
what you think the corresponding improvements in white collar jobs have been,
though? Genuinely asking! Or do you think they are mostly yet to come?

~~~
lettergram
From my own work:

* NER (name-entity recognition) is becoming insanely good (99+%), this can help with everything from data security to automated form filling, to automated medical billing, etc. Multiple billion dollar industries will be shook up by this.

* My own work on data generation is pointing to reduced surface area of security purposes and improved access to quasi-data for data science and application testing[1]

* Translation from images to text descriptions are also a major thing, which when combined with other systems can do things like make suggested diagnosis.

* Text generation systems are impacting how consumers interact. The chat bots these days are getting to the point where it's very difficult to identify if it's a human or not. I can't go into too many details, but see[2]

* There's also my side business which is a search engine for people who would know an answer to a questions: [https://insideropinion.com/](https://insideropinion.com/)

In general, NLP is probably going to have a larger impact that driverless cars
on our day-to-day lives. We're seeing insanely good story generation (e.g.
GPT-2), chat bots, billing, etc. I also think WAY WAY more is yet to come. The
transformer (which is spuring most of this) is only a couple years old. We'll
see lots more applications as time goes on.

[1] [https://medium.com/capital-one-tech/why-you-dont-
necessarily...](https://medium.com/capital-one-tech/why-you-dont-necessarily-
need-data-for-data-science-48d7bf503074)

[2]
[https://arxiv.org/pdf/1908.01841.pdf](https://arxiv.org/pdf/1908.01841.pdf)

------
orasis
From a professional perspective, if you think ML is a fad you are completely
missing the train.

~~~
semiotagonal
From a professional perspective, does it make sense to get on a train that is
so crowded already? Step 0 is probably to take Andrew Ng's on Coursera, but as
of right now, you'd be among "2,647,287 already enrolled!" [0]

[0] [https://www.coursera.org/learn/machine-
learning](https://www.coursera.org/learn/machine-learning)

~~~
orasis
You have to be joking. Demand for ML skills, especially scaling in production,
is off the charts.

~~~
username90
Scaling ML in production doesn't take much ML skills though.

~~~
omicron1
The bulk of industry ML work isn't actually suited to ML phds without
engineering skills. I work in FAANG and this is a huge problem where the ML
phds have poor communication with skilled engineers and a lack of engineering
experience. They often even look down upon people who don't have fancy
credentials. Unfortunately they just end up creating a money wasting disaster
of a system.

------
sam0x17
A problem is that the current mode of input for neural networks (one floating
point value per input node) makes neural networks problematic for tackling
problems with variable input size (read: most problems). My current research
looks into how we can maybe fix that in certain cases, but in general it's a
big limitation. Recurrent networks fix this to a certain extent, but they have
not had the revolution that CNN's have enjoyed just yet. We can also do things
like represent things in binary, but this adds complexity, and often greatly
inhibits learning.

------
jokoon
Once you realize there's "artificial" in AI, it quickly dispel the interest.

Science cannot even agree on a practical definition of intelligence. Unless
you let biologists, psychologists and neurologists work with computer
scientists to advance the discussion, I don't think it's worth it to explore
machine learning techniques to pursue intelligence.

Don't call it ML or AI, call it "improved statistical prediction". The buzz
will quickly fade.

------
andbberger
In my experience companies just aren't willing to invest, and get turned off
when they realize just how long the timelines on bringing DL to production
are.

It certainly works, but magic it ain't. You need an experienced practitioner
and lots of patience.

------
jhrmnn
This year, I’ve been working on representing electronic wave functions with
neural networks:
[https://arxiv.org/abs/1909.08423](https://arxiv.org/abs/1909.08423) I come
from a physics background, and my impression is that NNs are a cool
computational tool that every computational physicist probably should have in
their toolbox (at least having a sense of their capability). But so far I
haven’t seen any truly revolutionary advance with NNs acrose the whole of
materials science.

------
jowdones
When I started with computers back in 1992, Eastern Europe, noone around me
had the slightest idea what computers can do. I sort of imagined (and hoped)
they will (magically) solve math problems for me so I won't need to actually
make the hard effort to understand.

27 years later, not just 99.99% of the humans in general but of my "fellow"
programmers, if I can call them so, are still stuck at this petty ignorance
level.

~~~
le3dh4x0r
Is that really so? I think this field has been hyped pretty hard and lots of
people want a piece of that deep neural cake.

------
hinkley
Question for the collective wisdom:

What, in your estimation, are some current or near-future trends that
represent an objective improvement to the state of the art?

------
egocodedinsol
"You can see the computer age everywhere but in the productivity statistics."
This was true for a few years, until it wasn't.

------
hinkley
I would like to humbly submit, from my nose-bleed seat, that ML would serve us
best by adding it to our arsenal of fuzzing techniques.

Train a model and then set it loose to do exploratory testing, looking for
valid-looking inputs that cause the wrong answer to be returned from our
entirely pedestrian Plain Old Business Logic. The AI proposes a scenario, but
the human is the final arbiter.

------
tus88
Deep fakes are pretty good. It's kindof interesting....ML has allowed
computers to generate realistic images that humans cannot discern....while
image recognition remains woefully hopeless compared to human perception.

So rather than competing with the human brain...AI will try and trick the
human brain instead.

~~~
EpicEng
>ML has allowed computers to generate realistic images that humans cannot
discern

I've yet to see a deep fake that's even remotely indiscernible from the real
thing.

~~~
afterburner
You probably haven't been noticing them in movies though. Such as when they
stuck Margot Robbie's entire head on an actual skater for I, Tonya.

[https://www.youtube.com/watch?v=bqgD6lHrQ_8](https://www.youtube.com/watch?v=bqgD6lHrQ_8)

~~~
EpicEng
Was that a DF though?

~~~
BubRoss
That video shows a CG head and says CG face replacement. Typically sequences
like this are done on a shot by shot basis and end up being a very healthy mix
of compositing, cg face replacements, CG head replacements, etc. These types
of effects have been used for literally decades across a wide range of movies
and have nothing to do with the recent trend and techniques of 'deep fakes'.

~~~
EpicEng
Right, so not a DF, which is what we're talking about here.

~~~
BubRoss
I was answering your question.

------
thecleaner
Most ML will eventually be commoditized so it is unlikely that specializing
only in ML will make things better for people who dont come from permier
schools. If you have a so called MiML from any premier school you may not know
jackshit about production Ml and still earn hundreds of thousands.

------
dgudkov
>tendency among science journalists ... to overstate the significance of new
advances in AI and machine learning

It's very hard to not overstate when overstating brings you clicks, likes,
retweets, and dopamine spikes.

------
Lapalux
Machine learning = stats + computing power.

The key thing here is that the core ingenuity of ML is statistics - the limits
of which are well known.

ML hype is false promises based on uninformed wild extrapolation.

------
yters
Not if we measure the revolution in games and kitten pictures! There has been
tremendous progress in those domains!

~~~
xamuel
What revolution in games do you have in mind? Is there any particular video
game that is making really interesting use of ML?

~~~
friendlybus
All the production tools for making games/video are cramming ml tools in. It's
one of the first places to be swimming in it.

The two minute papers guy made a material predictor to cut down on preview
render time. Quixel uses ml for search. Substance and Houdini are gaining
tools soon. Blender has used it behind the scenes for its new neblua tool.
Mocap & animation are bathed in it. There's probably a lot more I can't recall
right now.

------
Gatsky
Still waiting for a revolution which has the optimal amount of hype. I wonder
if this is an application for GANs.

------
justicezyx
The ratio of ML interested or majored job candidates I saw nowadays is
staggering, appears >70%.

------
blankdebut
People doing research in ML have been saying something along these lines for
years.

------
KaoruAoiShiho
I'm not expert but I have a feeling this article will age poorly.

------
m3kw9
DeepFake issues has definitely been exaggerated

