
AI winter is well on its way - wei_jok
https://blog.piekniewski.info/2018/05/28/ai-winter-is-well-on-its-way/
======
bitL
I was recently "playing" with some radiology data. I had no chance to identify
diagnoses myself with untrained eyes, something that probably takes years for
a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up
with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this
classifier beat best human performing radiologists. Now the very same model
with no other change than adjusting number of categories could be used in any
image classification task, likely with state-of-art results. I was surprised
when I applied it to another, completely unrelated dataset and got >92%
accuracy right away.

If you think this is a symptom of AI winter, then you are probably wasting
time on outdated/dysfunctional models or models that aren't suited for what
you want to accomplish. Looking e.g. at Google Duplex (better voice
synchronization than Vocaloid I use for making music), this pushed state-of-
art to unbelievable levels in hard-to-address domains. I believe the whole SW
industry will be living next 10 years from gradual addition of these concepts
into production.

If you think Deep (Reinforcement) Learning is going to solve AGI, you are out
of luck. If you however think it's useless and won't bring us anywhere, you
are guaranteed to be wrong. Frankly, if you are daily working with Deep
Learning, you are probably not seeing the big picture (i.e. how horrible
methods used in real-life are and how you can easily get very economical 5%
benefit of just plugging in Deep Learning somewhere in the pipeline; this
might seem little but managers would kill for 5% of extra profit).

~~~
khawkins
AI winters are a result of a massive disparity between the expectations of the
general public and the reality of where the technology currently sits. Just
like an asset bubble, the value of the industry as a whole pops as people
collectively realize that AI, while not being worthless, is worth
significantly less than they thought.

Understand that in pop-sci circles over the past several years the general
public is being exposed to stories warning about the singularity by well
respected people like Stephen Hawking and Elon Musk
([http://time.com/3614349/artificial-intelligence-
singularity-...](http://time.com/3614349/artificial-intelligence-singularity-
stephen-hawking-elon-musk/)). Autonomous vehicles are on the roads and Boston
Dynamics is showing very real robot demonstrations. Deep learning is breaking
records in what we thought was possible with machine learning. All of this
progress has excited an irrational exuberance in the general public.

But people don't have a good concept of what these technologies can't do,
mainly because researchers, business people, and journalists don't want to
tell them--they want the money and attention. But eventually the general
public wises up to the unfulfillment of expectations, and drives their
attention elsewhere. Here we have the AI winter.

~~~
Florin_Andrei
> _AI winters are a result of a massive disparity between the expectations of
> the general public and the reality of where the technology currently sits._

I think they also happen when the best ideas in the field run into the brick
wall of insufficiently developed computer technology. I remember writing code
for a perceptron in the '90s on an 8 bit system, 64 k RAM - it's laughable.

But right now compute power and data storage seem plentiful, so rumors of the
current wave's demise appear exaggerated.

~~~
bitL
I wish GPUs were 1000x faster... Then I could do some crazy magic with Deep
Learning instead of waiting weeks for training to be finished...

~~~
jacquesm
That's more a matter of budget than anything else. If you problem is valuable
enough spending the money in a short time-frame rather than waiting for weeks
can be well worth the investment.

~~~
bitL
I cannot fit a cluster of GPUs into a phone where I could make magic happen
real-time though :(

~~~
jacquesm
Hm. Offload the job to a remote cluster? Or is comms then the limiting factor?

~~~
bitL
It won't give us that snappy feeling; imagine learning things in milliseconds
and immediately displaying them on your phone.

~~~
Florin_Andrei
Jeez. That would be faster than protein-and-water-based systems, which up
until now are still the faster learners.

------
joe_the_user
This is a deep, significant post (pardon pun etc).

The author is clearly informed and takes a strong, historical view of the
situation. Looking at what the really smart people who brought us this
innovation have said and done lately is a good start imo (just one datum of
course, but there are others in this interesting survey).

 _Deepmind hasn 't shown anything breathtaking since their Alpha Go zero._

Another thing to consider about Alpha Go and Alpha Go Zero is the vast, vast
amount of computing firepower that this application mobilized. While it was
often repeated that ordinary Go program weren't making progress, this wasn't
true - the best, amateur programs had gotten to about 2 Dan amateur using
Makov Tree Search. Alpha Go added CNNs for it's weighting function and
petabytes of power for it's process and got effectiveness up to best in the
world, 9 Dan professional, (maybe 11 Dan amateur for pure comparison). [1]

Alpha Go Zero was supposedly even more powerful, learned without human
intervention. BUT it cost petabytes and petabytes of flops, expensive enough
that they released a total of ten or twenty Alpha Go Zero game to the world,
labeled "A great gift".

The author convenniently reproduces the chart of power versus results. Look at
it, consider it. Consider the chart in the context of Moore's Law retreating.
The problems of Alpha Zero generalizes as described in the article.

The author could also have dived into the troubling question as of "AI as
ordinary computer application" (what does testing, debugging, interface
design, etc mean when the app is automatically generated in an ad-hoc fashion)
or "explainability". But when you can paint a troubling picture without these
gnawing problems appearing, you've done well.

[1]
[https://en.wikipedia.org/wiki/Go_ranks_and_ratings](https://en.wikipedia.org/wiki/Go_ranks_and_ratings)

~~~
tim333
>Deepmind hasn't shown anything breathtaking since their Alpha Go zero

They went on to make AlphaZero, a generalised version that could learn chess,
shogi or any similar game. The chess version beat a leading conventional chess
program 28 wins, 0 losses, and 72 draws.

That seemed impressive to me.

Also they used loads of compute during the training but not so much during
play.(5000 TPUs, 4TPUs).

Also it got better than humans in those games from scratch in about 4 hours
whereas humans have had 2000 years to study them so you can forgive it some
resource usage.

~~~
felippee
It's not like humanity really needs another chess playing program 20 years
after IBM solved that problem (but now utilizing 1000x more compute power). I
just find all these game playing contraptions really uninteresting. There are
plenty real world problems to be solved of much higher practicality. Moravec's
paradox in full glow.

~~~
batmansmk
I guess there are reasons why researchers build chess programs: it is easy to
compare performance between algorithms. When you can solve chess, you can
solve a whole class of decision-making problems. Consider it as the perfect
lab.

~~~
EliRivers
What is that class of decision-making problems? It's nice to have a machine
really good at playing chess, but it's not something I'd pay for. What
decision-making problems are there, in the same class, that I'd pay for?

 _Consider it as the perfect lab._

Seems like a lab so simplified that I'm unconvinced of its general
applicability. Perfect knowledge of the situation and a very limited set of
valid moves at any one time.

~~~
jcelerier
> What decision-making problems are there, in the same class, that I'd pay
> for?

an awful lot of graph and optimization problems. See for instance some
examples in
[https://en.wikipedia.org/wiki/A*_search_algorithm](https://en.wikipedia.org/wiki/A*_search_algorithm)

~~~
AstralStorm
Perfect information problem solving is not interesting anymore.

Did they manage to extend it to games with hidden and imperfect information?

(Say, chess with fog of war also known as Dark Chess. Phantom Go. Pathfinding
equivalent would be an incremental search.)

Edit: I see they are working on it, predictive state memory paper (MERLIN) is
promising but not there yet.

~~~
dgacmu
Strongly disagree. There are a lot of approximation algorithms and heuristics
in wide use - to the tune of trillions of dollars, in fact, when you consider
transportation and logistics, things like asic place & route, etc. These are
all intractable perfect info problems that are so widespread and commercially
important that they amplify the effect of even modest improvements.

(You said problems, not games...)

~~~
AstralStorm
Indeed, there are a few problems where even with perfect information you will
be hard pressed to solve them. But that is only a question of computational
power or the issue when the algorithm does not allow efficient approximation
(not in APX space or co-APX).

The thing is, an algorithm that can work with fewer samples and robustly
tolerating mistakes in datasets (also known as imperfect information) will be
vastly cheaper and easier to operate. Less tedious sample data collection and
labelling.

Working with lacking and erroneous information (without known error value) is
necessarily a crucial step towards AGI; as is extracting structure from such
data.

This is the difference between an engineering problem and research problem.

~~~
dgacmu
Perhaps a unifying way of saying this is: it's a research problem to figure
out how to get ML techniques to the point they outperform existing heuristics
on "hard" problems. Doing so will result in engineering improvements to the
specific systems that need approximate solutions to those problems.

I completely agree about the importance of imperfect information problems. In
practice, many techniques handle some label noise, but not optimally. Even
MNIST is much easier to solve if you remove the one incorrectly-labeled
training example. (one! Which is barely noise. Though as a reassuring example
from the classification domain, JFT is noisy and still results in better real
world performance than just training on imagenet.)

------
nopinsight
A different take by Google’s cofounder, Sergey Brin, in his most recent
Founders’ Letter to investors:

“The new spring in artificial intelligence is the most significant development
in computing in my lifetime.”

He listed many examples below the quote.

“understand images in Google Photos;

enable Waymo cars to recognize and distinguish objects safely;

significantly improve sound and camera quality in our hardware;

understand and produce speech for Google Home;

translate over 100 languages in Google Translate;

caption over a billion videos in 10 languages on YouTube;

improve the efficiency of our data centers;

help doctors diagnose diseases, such as diabetic retinopathy;

discover new planetary systems; ...”

[https://abc.xyz/investor/founders-
letters/2017/index.html](https://abc.xyz/investor/founders-
letters/2017/index.html)

An example from another continent:

“To build the database, the hospital said it spent nearly two years to study
more than 100,000 of its digital medical records spanning 12 years. The
hospital also trained the AI tool using data from over 300 million medical
records (link in Chinese) dating back to the 1990s from other hospitals in
China. The tool has an accuracy rate of over 90% for diagnoses for more than
200 diseases, it said.“

[https://qz.com/1244410/faced-with-a-doctor-shortage-a-
chines...](https://qz.com/1244410/faced-with-a-doctor-shortage-a-chinese-
hospital-is-betting-big-on-artificial-intelligence-to-treat-patients/)

~~~
felippee
Hi, author here:

Well first off: letters to investors are among the most biased pieces of
writing in existence.

Second: I'm not saying connectionism did not succeed in many areas! I'm a
connectionist by heart! I love connectionism! But that being said there is
disconnect between the expectations and reality. And it is huge. And it is
particularly visible in autonomous driving. And it is not limited to media or
CEO's, but it made its way into top researchers. And that is a dangerous sign,
which historically preceded a winter event...

~~~
nopinsight
I agree that self-driving had/have been overhyped over the previous few years.
The problem is harder than many people realize.

The difference between the current AI renaissance and the past pre-winter AI
ecosystems is the level of economic gain realized by the technology.

The late 80s-early 90s AI winter, for example, resulted from the limitations
of expert systems which were useful but only in niche markets and their
development and maintenance costs were quite high relative to alternatives.

The current AI systems do something that alternatives, like Mechanical Turks,
can only accomplish with much greater costs and may not even have the scale
necessary for global massive services like Google Photos or Youtube
autocaptioning.

The spread of computing infrastructure and connectivity into the hands of
billions of global population is a key contributing factor.

~~~
felippee
> The difference between the current AI renaissance and the past pre-winter AI
> ecosystems is the level of economic gain realized by the technology

I would argue this is well discounted by level of investment made against the
future. I don't think the winter depends on the amount that somebody makes
today on AI, rather on how much people are expecting to make in the future. If
these don't match, there will be a winter. My take is that there is a huge bet
against the future. And if DL ends up bringing just as much profit as it does
today, interest will die very, very quickly.

~~~
nopinsight
Because there is a dearth of experts and a lack of deep technical knowledge
among many business people, there are still a great many companies that have
not yet started investing in deep learning or AI despite potential profits
based on _current_ technology. Non-tech sectors of the economy are probably
underinvesting at the moment.

This is analogous to the way electricity took decades to realize productivity
gains in the broad economy.

That said, the hype will dial down. I am just not sure the investment will
decrease soon.

~~~
sanxiyn
While I agree there is underinvestment in non-tech sectors, I don't see why
that would change and they will use deep learning. There are lots of
profitable things in non-tech sectors that can be done with linear regression
but not done.

~~~
cm2187
There are lots of things in the non-tech sector that can be automated with
simple vanilla software but isn't. To use AI instead, you need to have 1)
sophisticated devs in place, 2) a management that gets the value added, 3)
lots of data in a usable format, 4) willingness to invest & experiment. Lots
of non-tech businesses are lacking one if not all of these.

------
dekhn
I'm a scientist from a field outside ML who knows that ML can contribute to
science. But I'm also really sad to see false claims in papers. For example, a
good scientist can read an ML paper, see claims of 99% accuracy, and then
probe further to figure out what the claims really mean. I do that a lot, and
I find that accuracy inflation and careless mismanagement of data mars most
"sexy" ML papers. To me, that's what's going to lead to a new AI winter.

~~~
mtgx
You hear Facebook _all the time_ saying how it "automatically blocks 99% of
the _terrorist content_ " with AI to the public and governments.

Nobody thought to ask: "How do you _know_ all of that content is terrorist
content? Does anyone check every video afterwards to ensure that all the
blocked content was indeed terrorist content?" (assuming they even have an
exact definition for it).

~~~
shmageggy
Also, how do they know how much terrorist content they aren't blocking (the
1%), since they by definition haven't found it yet?

------
imh
FYI This post is about deep learning. It could be the case that neural
networks stop getting so much hype soon, but the biggest driver of the current
"AI" (ugh I hate the term) boom is the fact that everything happens on
computers now, and that isn't changing any time soon.

We log everything and are even starting to automate decisions. Statistics,
machine learning, and econometrics are booming fields. To talk about two
topics dear to my heart, we're getting way better at modeling uncertainty
(bayesianism is cool now, and resampling-esque procedures aged really well
with a few decades of cheaper compute) and we're better at not only talking
about what causes what (causal inference), but what causes what when
(heterogeneous treatment effect estimation, e.g. giving you aspirin right now
does something different from giving me aspirin now). We're learning to learn
those things super efficiently (contextual bandits and active learning). The
current data science boom goes far far far far beyond deep learning, and most
of the field is doing great. Maybe those bits will even get better faster if
deep learning stops hogging the glory. More likely, we'll learn to combine
these things in cool ways (as is happening now).

~~~
digitalzombie
Bayesian can be seen as a subset of deep learning or hell a superset.

AI is a superset and Machine learning is a subset of AI and most funding is in
deep learning. Once Deep Learning hit the limit I believe there will be an AI
winter.

Maybe there will be hype around statistic (cross fingers) which will lead to
Bayesian and such.

~~~
eli_gottlieb
>Bayesian can be seen as a subset of deep learning or hell a superset.

 _eh-hem_

DIE, HERETIC!

 _eh-hem_

Ok, with that out of my system, no, Bayesian methods are definitely _not_ a
subset of deep learning, in any way. Hierarchical Bayes could be labeled "deep
Bayesian methods" if we're marketing jerks, but Bayesian methods mostly do not
involve neural networks with >3 hidden layers. It's just a different paradigm
of statistics.

~~~
digitalzombie
My mentor was very very adamant about Bayesian network and hierarchical as
being deep learning.

He sees the latent layer in the hierarchical model as the hidden layer and the
Bayesian just have a strict restrictions/assumptions to the network where as
the deep learning is more dumb and less assuming. A few of my professor thinks
that PGM, probability graphical model is a super set of deep learning/neural
network.

This is where my thinking come from.

IIRC, a paper have shown that gradient descent seems to exhibit MCMCs (blog
with paper link inside that led to this conclusion of mine:
[http://www.inference.vc/everything-that-works-works-
because-...](http://www.inference.vc/everything-that-works-works-because-its-
bayesian-2/)).

But I am not an expert in Neural Network nor know the topic well enough to say
such a thing. Other than was deferring to opinions of some one that's better
than myself. So I'll keep this in mind and hopefully one day have the time to
do more research into this topic.

Thank you.

~~~
eli_gottlieb
I think your link, and your mentor, are somewhat fundamentalist about their
Bayesianism.

------
MichaelMoser123
Forget about self driving cars - the real killer application of deep learning
is mass surveillance - there are big customer for that (advertising, policing,
political technology - we better get used to the term) and its the only
technique that can get the job done.

I sometimes think that there really was no AI winter as we got other
technologies that implemented the ideas: SQL Databases can be seen as an
application of many ideas in classical AI - for example its a declarative
language for defining relations among tables; you can have rules in the form
of SQL stored procedures; actually it was a big break (paradigm shift is the
term) in how you deal with data - the database engine has to do some real
behind the scenes optimization work in order to get a workable representation
of the data definition (that is certainly bordering on classical AI in
complexity).

these boring CRUD applications are light years ahead in how data was handled
back in the beginning.

~~~
dx034
The BBC recently requested information about the use of facial recognition
from UK police forces. Those that use facial recognition reported false
positive rates of >95%. That led some to abandon the systems, others just use
it as one form of pre-screening. Mass surveillance with facial recognition is
nowhere near levels where it can be used unsupervised. And that's even before
people actively try to deceive it.

For advertising, I'm also not sure if there's been a lot of progress. Maybe
it's because I opted out too much but I have the feeling that ad targeting
hasn't become more intelligent, rather the opposite. It's been a long time
that I've been surprised at the accuracy of a model tracking me. Sure,
targeted ads for political purposes can work very well but are nothing new and
don't need any deep learning nor any other "new" technologies.

Where I really see progress is data visualisation. Often dismissed it can be
surprisingly hard to get right and tools around that (esp for enterprise use)
have developed a lot over recent years. And that's what companies need. No
one's looking for a black-box algorithm to replace marketing, they just want
to make some sense of their data and understand what's going on.

~~~
nmca
Aha, yeah I saw this in the news - pretty classic use of people horribly
misunderstanding statistics and/or misrepresenting the facts. Let's say one
person out of 60 million has sauron's ring. I use my DeepMagicNet to classify
everyone, and get 100 positive results. Only one is the ringbearer, so I have
a 99% error rate. Best abandon ship.

~~~
dx034
I do think of myself that I can read statistics. They didn't mention it as a
false positive rate, that's what I interpreted. I don't have the article here
but they said that the system gave out ~250 alerts, of which ~230 turned out
to be incorrect. It didn't specify at all how big the database of potential
suspects was. The number of scanned faces was ~50-100k each time (stadium).
Nevertheless, the 230/250 is the issue here, simply because it destroys trust
in the system. If the operator already knows that the chance of this being a
false alarm is ~5%, will they really follow up all alerts?

------
rch
> Deepmind hasn't shown anything breathtaking since their Alpha Go zero.

Didn't this _just_ happen? Maybe my timescales are off, but I've been thinking
about AI and Go since the late 90s, and plenty of real work was happening
before then.

Outside a handful of specialists, I'd expect another 8-10 years before the
current state of the art is generally understood, much less effectively
applied elsewhere.

~~~
ehsankia
Also why does every single result has to be breathtaking? Here's a quick
example, at IO they announced that their work on Android improved battery life
by up to 30%. That's pretty damn impressive.

~~~
felippee
> Also why does every single result has to be breathtaking?

If you build the hype like say Andrew Ng it better be. Also if you consume
more money per month than all the CS departments of a mid sized country, it
better be.

~~~
raducu
In terms of hype you may be right, but it doesn't mean that if something
doesn't live up to the hype of Andrew Ng or Elon Musk it won't still be pretty
good.

For instance: even if Elon Musk doesn't colonize Mars but instead just builds
the BFR, that would still be amazing; even if BFR is never build but falcon 9
becomes fully reusable that would be great; even if falcon 9 won't be fully
reusable, the fact that it cut the launching cost to space is still pretty
good.

Even if we don't achieve any great breakthroughs with AGI, the fact that we
started to use transfer learning to diagnose human disseases is pretty
amazing; the fact that a japanese guy used tensorflow on a raspbery pi to
categorize real cucumbers by shape is amazing.

All of this stuff won't go away; people will not say "hey, let's just forget
about this deep learning thing and put it in some dusty shelf, it's useless
for now". Maybe it will take 20 or 50 more years, maybe it's a slow thaw, but
how could this be a winter?

~~~
rm_-rf_slash
Honestly I think the raspberry pi indicates the (short-term) future of AI.
Most of the “easy” problems have been solved (image classification, game
playing), but the hard ones like NLP are orders of magnitude more complex and
therefore elusive.

I’m happy to leave the hard problems for the PhDs and the big tech
researchers. Go nuts, folks.

In the meantime, the applications for small-scale, pre-trained neural networks
seem limitless. Manufacturing, agriculture, retail, pretty much any industry
could make use of portable neural networks.

~~~
raducu
I feel exactly the same way. Just wait 2-3 years before someone launches an
embedded TPU and the sky will be the limit.

------
mastrsushi
Warning 23 year old CS grad angst ridden post:

I'm very sick of the AI hype train. I took a PR class for my last year of
college, and they couldn't help but mention it. LG Smart TV ads mention it,
Microsoft commercials, my 60 year old tech illiterate Dad. Do any end users
really know what it's about? Probably not, nor should that matter, but it's
very triggering to see something that was once a big part of CS turned into a
marketable buzzword.

I get triggered when I can't even skim through the news without hearing Elon
Musk and Steven Hawking ignorantly claim AI could potentially takeover
humanity. People believe them because of their credentials, when professors
who actually teach AI will say otherwise. I'll admit, I've never taken any
course in the subject myself. An instructor I've had who teaches the course
argues it doesn't even exist, it's merely a sequence of given instructions,
much like any other computer program. But hey, people love conspiracies, so
let their imagination run wild.

AI is today what Big Data was about 4 years ago. I do not look highly on any
programmer that jumps bandwagons, especially for marketability. Not only is it
impure in intention, it's foolish when their are 1000 idiots just like them
over-saturating the market. Stick with what you love, even if it's esoteric.
Then you won't have to worry about your career value.

~~~
akvadrako
AI taking over is the biggest threat facing humanity, but I don't think
Hawking ever claimed it was imminent; it's likely 1000+ years away.

It should be obvious why a superior intelligence is something dangerous.

~~~
pfisch
"it's likely 1000+ years away."

That seems like a pretty high number when you consider the exponential rate of
technological advancement.

1000 years in the future is probably going to be completely unrecognizable to
us given the current rate of change in society/tech.

~~~
Scea91
True exponentials exist very rarely. What we are used to call exponential is
often at most quadratic and even then the trend usually exists only for some
limited time window.

I wouldn't dare to make prediction about mankind 1000 years from now based on
a relatively short time window of technological growth. There are many huge
obstacles in the way.

~~~
pfisch
Even the present would be mostly unrecognizable to someone who lived 1000
years ago.

1000 years from now will certainly be even stranger to us.

------
oh-kumudo
Author's reasons:

1.Hype dies down (which is really good! Meaning the chance of burst, is
actually lower!)

2.Doesn't scale is false claim. DL methods have scaled MUCH better than any
other ML algorithms in recent history (scale SVM is no small task). Scaling
for DL methods are much either as comparing to other traditional ML
algorithms, where it can be naturally distributed and aggregated.

3\. Partially true. But self-driving is a sophisticated area by itself, DL is
part of it, it can't really put full claim on its potential future success or
ultimate downfall.

4\. Gary Marcus isn't an established figure in DL research.

AI winter will ultimately come. But it is because people will become more
informed about DL's strengths and limits, thus becoming smarter to tell what
is BS what is not. AGI is likely not going to happen just with DL, but that is
no way meaning it is a winter. DL has revolutionized the paradigm of Machine
Learning itself, the shift has now complete, it will stay for a very very long
time, and the successor is likely to build upon it not subvert it completely
as well.

~~~
felippee
Author here: I'm using deep learning daily so I have a bit of an idea on what
I'm talking about.

1) Not my point. Hype is doing very well. But narrative begins to crack,
actually indicative of a burst... 2) DL does not scale very well. It does
scale better than other ML algorithm because those did not scale at all. If
you want to know what scales very well, look at CFD (computational fluid
dynamics). DL in nowhere near that ease in scaling. 3) self driving is the
poster child of current "AI-revolution". And it is where by far most money is
allocated. So if that falls, rest of DL does not matter. 4) Not that this
matters, does it?

~~~
evrydayhustling
The scaling argument in the article doesn't make any sense. There are
rhetorical queries like "does this model with 1000x as many parameters work
1000x as well?" but what it means to scale or perform are not clearly or
consistently defined - let alone defined in a way that would make your point
about the utility of the advances.

OpenAI's graph shows new architectures being used with more parameters because
people are innovating on architecture and scale at the same time. Arguing that
old methods "failed to scale" is like arguing that processor development was a
failure because Intel had to develop a 486 instead of making a 386 work with
more transistors (or more _something_ ).

And what does CFD have to do with anything, except maybe an odd attempt to
argue from authority? Can you formalize from CFD a notion of "scaling well"
well that anyone else agrees is useful for measuring AI research?

~~~
prewett
CFD was merely used as an example of something that does scale well. I'm not
sure it was the best example, since CFD isn't very common. But basically you
have a volume mesh and each cell iterates on the Navier-Stokes equation. So if
you have N processor cores, you break the mesh in N pieces, each of which get
processed in parallel. Doubling the number of cores allows you process double
the amount in the same time, minus communication loses (each section of the
mesh needs to communicate the results on its boundary to its neighbors).

I don't fully understand the graph, but it looks like his point is that Alpha
Go Zero uses 1e5 times as many resources than AlexNet, but does not produce
anywhere near 10,000 times better results. We saw that with CFDt 1e5 more
cores resulted in 1e5 better results (= scales). The assertion is that DL's
results are much less than 1e5 better, hence it does not scale.

Basically the argument is:

1\. CFD produces N times better results given N times more resources [this is
implied, requires a knowledge of CFD]. That is, f(a _x) = a_ f(x). Or, f(a _x)
= 1_ a * f(x).

2\. Empirically, we see that DL has used 1e5 more resources, but is not
producing 1e5 times better results. [No quantitative analysis of how much
better the results are is given]

3\. Since DL has f(a * x) = b * a * f(x), where b < 1, DL does not scale.
[Presumably b << 1 but the article did not give any specific results]

This isn't a very rigorous argument and the article left out half the
argument, but it is suggestive.

~~~
felippee
Thanks for that, that is essentially my point. Agree it is not very rigorous,
but it gets the idea across. By scalable we'd typically think "you throw more
gpu's at it and it works better by some measure". Deep learning does that only
in extremely specific domains, e.g. games and self play as in alpha go. For
majority of other applications it is architecture bound or data bound. You
can't throw more layers, more basic DL primitives and expect better results.
You need more data, and more phd students to tweak the architecture. That is
not scalable.

~~~
evrydayhustling
More compute -> more precision is just one field's definition of scalable...
Saying that DNNs can't get better just by adding GPUs is like complaining that
an apple isn't very orange.

To generalize notions of scaling, you need to look at the economics of
consumed resources and generated utility, and you haven't begun to make the
argument that data acquisition and PhD student time hasn't created ROI, or
that ROI on those activities hasn't grown over time.

Data acquisition and labeling is getting cheaper all the time for many
applications. Plus, new architectures give ways to do transfer learning or
encode domain bias that let you specialize a model with less new data. There
is substantial progress and already good returns on these types of scalability
which (unlike returns on more GPUs) influence ML economics.

~~~
felippee
OK, the definition of scalable is crucial here and it causes lots of trouble
(this is also response to several other posts so forgive me if I don't address
your points exactly).

Let me try once again: an algorithm is scalable if it can process bigger
instances by adding more compute power.

E.g. I take a small perceptron and train it on pentium 100, and then take a
perceptron with 10x parameters on Core I7 and get better output by some
monotonic function of increase in instance size (it is typically a sub linear
function but it is OK as long as it is not logarithmic).

DL does not have that property. It requires modifying the algorithm, modifying
the task at hand and so on. And it is not that it requires some tiny tweaking.
It requires quite a bit of tweaking. I mean if you need a scientific paper to
make a bigger instance of your algorithm this algorithm is not scalable.

What many people here are talking about is whether an instance of the
algorithm can be created (by a great human effort) in a very specific domain
to saturate a given large compute resource. And yes, in that sense deep
learning can show some success in very limited domains. Domains where there
happens to be a boatload of data, particularly labeled data.

But you see there is a subtle difference here, similar in some sense to
difference between Amdahl's law and Gustafson's law (though not literal).

The way many people (including investors) understand deep learning is that:
you build a model A, show it a bunch of pictures and it understands something
out of them. Then you buy 10x more GPU's, build model B that is 10x bigger,
show it those same pictures and it understands 10x more from them. Look I, and
many people here understand this is totally naive. But believe me, I talked to
many people with big $ that have exactly that level of understanding.

~~~
evrydayhustling
I appreciate the engagement in making this argument more concrete. I
understand that you are talking about returns on compute power.

However, your last paragraph about how investors view deep learning does not
describe anyone in the community of academics, practitioners and investors
that I know. People understand that the limiting inputs to improved
performance are data, followed closely by PhD labor. Compute power is relevant
mainly because it shortens the feedback loop on that PhD labor, making it more
efficient.

Folks investing in AI believe the returns are worth it due to the potential to
scale deployment, not (primarily) training. They may be wrong, but this is a
straw man definition of scalability that doesn't contribute to that thesis.

------
jarym
I’m not sure how the hype wagon started but I for one am glad it’s about to
pop.

I am working (founded) a startup and while we have AI on the roadmap for about
a years time, it isn’t something that’s central to our product. (We already
use some ML techniques but I’d not confidently boast its the same thing as
AI).

Cue an informal lunch with a VC guy who takes a look, says we’re cool and
tells us just to plaster the word AI in more places - he was sure we could
raise a stupendous sum of cash doing that.

As an AI enthusiast I was bothered by this. We have everyone and their mother
hyping AI into areas it’s not even relevant in, let alone effective at.

A toning down would be healthy. We could then focus on developing the right
technology slowly and without all the lofty expectations to live up to.

~~~
flohofwoe
> ...just to plaster the word AI in more places...

I bet that's how the new "AI-assisted" Intellisense in Visual Studio got
greenlit:

[https://blogs.msdn.microsoft.com/visualstudio/2018/05/07/int...](https://blogs.msdn.microsoft.com/visualstudio/2018/05/07/introducing-
visual-studio-intellicode/)

If an AI-infested text editor isn't a sure sign that the bubble is going to
pop soon then I don't know ;)

~~~
annabellish
I think part of this ridiculousness may just be that the common lexicon
doesn't have enough words for AI. "AI-assisted Intellisense" at the moment
seems to boil down to a marginally novel way of ordering autocomplete
suggestions, and like...

It's not wrong? It's neat, it's potentially useful, and it's powered by
something under the umbrella of "AI". The problem is that that umbrella is
gigantic, and covers everything from the AI system providing routes for the AI
system in a self-driving truck on AI-provided schedules for a hypothetical
mostly-automated shipping business, to a script I whipped up in ten minutes to
teach something 2+2 and literally nothing else.

So we get to the nonsense position where there isn't a better way to describe
a minor improvement to what is essentially the ordering of a drop-down list
except by comparison to the former example.

------
solomatov
AI winter is not on its way. We constantly get new breakthroughs and there's
no end in the view. For example, in the last year a number of improvements in
GANs were introduced. This is really huge, since GANs are able to learn a
dataset structure without explicit labels, and this is a large bottleneck in
applying ML more widely.

IMO, we are far away from AGI, but even current technologies applied widely
will lead to many interesting things.

~~~
felippee
I sure agree there are many interesting things going on, there is no question
about that. Also most of them are toy problems focused in some restricted
domains, while a huge bag of equally interesting real world problems is
sitting untouched. And let me tell you, all those VC's that put in probably
way north of $10B are not looking forward to more NIPS papers or yet another
style transfer algorithm.

~~~
solomatov
>Also most of them are toy problems focused in some restricted domains, while
a huge bag of equally interesting real world problems is sitting untouched.

It always starts with toy problems. Recognizing pictures from imagenet was
also a toy problem back then.

------
xedarius
"Deepmind hasn't shown anything breathtaking since their Alpha Go zero"

... what about when the Google assistant near perfectly mimicked a human
making a restaurant reservation .... the voice work was done at DeepMind.

All the problems in AI haven't been solved yet? Well no, of course not.
Limitations exist and our solutions need to be evolved.

I think perhaps the biggest constraint is requiring huge amounts of training
data so solve problem X. Humans simply don't need that, which must be some
indication that what we're doing isn't quite right.

~~~
gaius
_what about when the Google assistant near perfectly mimicked a human making a
restaurant reservation_

Any sufficiently advanced technology is indistinguishable from a rigged demo

------
dhab
Disclaimer: I am a lay technical person and don't know much about AI.

I find this article somewhat condescending. I look at all the current
development as stepping stones to progress, not an overnight success that does
everything flawlessly. I imagine the future might be some combination of
different solutions, and what the author proposes may or may not play a part
in it.

~~~
bra-ket
it's not a stepping stone, if you look closely it's a dead end

~~~
biswaroop
I don't see how systematically accurate image classifiers and facial
recognition systems built on deep learning is a 'dead end'. Products are
products. If deep learning has led to actual profits in actual companies, it's
not a dead end. As to whether this leads to AGI is a completely different
question.

~~~
jimothywales
The point is that the profits may not be as grand as the current level of hype
may indicate

Edit: additionally it could be a dead end because the hype tends to narrow the
directions we explore with ML. If everyone is obsessing about DL, we could be
infuriatingly ignoring other research directions right under our noses.

------
zerostar07
An AI "winter" is a long period in which [edit: funding is cut because...]
researchers _are in disbelief_ about having a path to real intelligence. I
think that is not the case at this time, because we have (or approaching)
adequate tools to rationally dismiss that disbelief. The current AI "spring"
has brought back the belief that connectionism may by the ultimate tool to
explain the human brain. I mean you can't deny that DL models of vision look
eerily like the early stages in visual processing in the brain (which is a
very large part of it). Even if DL researchers lose their path in search for
"true AI", the neuroscientists can keep probing the blueprint to find new
clues to its intelligence. Even AI companies are starting to create plausible
models that link to biology. So at this time, it's unlikely that progress will
be abandoned any time soon.

E.g. [https://arxiv.org/abs/1610.00161](https://arxiv.org/abs/1610.00161)
[https://arxiv.org/abs/1706.04698](https://arxiv.org/abs/1706.04698)
[https://www.ncbi.nlm.nih.gov/pubmed/28095195](https://www.ncbi.nlm.nih.gov/pubmed/28095195)

~~~
dekhn
No, AI winter was when the AI people oversold the tech, then failed to
deliver, and lost their funding. This is well documented in histories of the
field.

~~~
zerostar07
I think the scientific pessimism preceded the funding cuts:

[https://en.wikipedia.org/wiki/AI_winter](https://en.wikipedia.org/wiki/AI_winter)

------
felippee
Author here: seriously I'm here at the front page for the second day in the a
row!?

The sheer viral popularity of this post, which really was just a bunch of
relatively loose thoughts indicates that there is something in the air
regarding AI winter. Maybe people are really sick of all that hype pumping...

Just a note: I'm a bit overwhelmed so I can't address all the criticism. One
thing I would like to state however, is that I'm actually a fan of
connectionism. I think we are doing it naively though and instead of focusing
on the right problem we inflate a hype bubble. There are applications where DL
really shines and there is no question about that. But in case of autonomy and
robotics we have not even defined the problems well enough, not to mention
solving anything. But unfortunately, those are the areas where most
best/expectations sit, therefore I'm worried about the winter.

~~~
s-shellfish
Do you think there's a balance to be struck between connectionism and
computationalism?

------
skybrian
The argument is that self-driving won't work because Uber and Tesla had well-
publicized crashes. But I don't see how this tells us anything about other,
apparently more cautious companies like Waymo. There seem to be significant
differences in technology.

More generally, machine learning is a broad area and there's no reason to
believe that different applications of it will all succeed or all fail for
similar reasons. It seems more likely there will be more winners along with
many failed attempts.

~~~
Animats
_The argument is that self-driving won 't work because Uber and Tesla had
well-publicized crashes. But I don't see how this tells us anything about
other, apparently more cautious companies like Waymo. There seem to be
significant differences in technology._

Yes. I've been saying this for a while. Waymo's approach is about 80%
geometry, 20% AI. Profile the terrain, and only drive where it's flat. The AI
part is for trying to identify other road users and guess what they will do.
When in doubt, assume worst case and stay far away from them.

I was amazed that anyone would try self-driving without profiling the road.
Everybody in the DARPA Grand Challenge had to do that, including us, because
it was off-road driving and you were not guaranteed a flat road. The
Google/Waymo people understood this. Some of the others just tried dumping the
raw sensor data into a deep learning system and getting out a steering wheel
angle. Not good.

~~~
BoomWav
A lot of companies fear the ship's leaving without them. They try to rush
ahead without thinking and then crashes. That's what it feels like every time
a car company or another tech company say they're going to build the next
self-driving car. I don't know why but I always feel like waymo is already
10,000 miles ahead.

------
madmax108
Honestly, I think this is a good thing for both AI researchers as well as AI
practitioners. One mans AI-winter is another mans stable platform.

While the number of world-shattering discoveries using DL may be on the
decline (ImageNet, Playing Atari, Artistic Style Transfer, CycleGAN,
DeepFakes, Pix2Pix etc), now both AI researchers and practitioners can work in
relative peace to fix the problem of the last 10%, which is where Deep
Learning has usually sucked. 90% accuracy is great for demos and papers, but
not even close to useful in real life (as the Uber fiasco is showing).

As an AI practitioner, it was difficult to simply keep up with the latest
game-changing paper (I have friends who call 2017 the Year of the GAN!), only
to later discover new shortcomings of each. Of course, you may say, why bother
keeping up? And the answer is simply that when we are investing time to build
something that will be in use 5-10 years from now, we want to ensure the
foundation is built upon the latest research, and the way most papers talk
about their results makes you believe they are best suited for all use cases,
which is rarely the case. But when the foundation itself keeps moving so fast,
there is no stability to build upon at all.

That and what jarym said is perfectly true as well.

The revolution is done, now it's time to evolution of these core ideas for
actual value generation , and I for one am glad about that.

------
afpx
AI winter? Hardly. Current methods have only been applied to a very tiny
fraction of problems that they can help solve. And, this trend will only
accelerate until computing resources become too expensive.

As long as there is ROI, AI projects will continue to be financed, top
thinkers around the world will be paid to do more research, and engineers will
implement the most recent techniques into their products and services to stay
competitive. This is a classic feedback system that results in exponential
progress.

------
tim333
This seems over negative. Just the opening argument, that companies were
saying "that fully self driving car was very close" but "this narrative begins
to crack"

Yet here they are self driving
[https://www.youtube.com/watch?v=QqRMTWqhwzM&feature=youtu.be](https://www.youtube.com/watch?v=QqRMTWqhwzM&feature=youtu.be)
and you should be able to hail one as a cab this year
[https://www.theregister.co.uk/2018/05/09/self_driving_taxis_...](https://www.theregister.co.uk/2018/05/09/self_driving_taxis_waymo/)

~~~
mastrsushi
You sure about that?
[https://www.youtube.com/watch?v=8IqpUK5teGM](https://www.youtube.com/watch?v=8IqpUK5teGM)

Tesla self driving cars have crashed too. Arrogant people like Elon Musk are
making a bad name for the hardworking AI developers who are actually trying to
make self driving cars faulty proof.

~~~
tim333
That vid of the Uber crash - just because one company has a crap product
doesn't mean they are all bad. Waymo is about the only one that seems about
ready to go and I think partly because they don't just count on deep learning.

~~~
mastrsushi
Like I said, Uber and Tesla have both been reported. If two companies of that
status with their funding abilities are potentially dangerous, I wouldn't
doubt any smaller company would.

------
daveguy
I thought there would be more of a backlash / winter onset when people realize
that Alexa is so annoying to deal with (and you basically have to learn a set
of commands) because AI isn't that clever yet. Also, when people realize that
autocorrect took a dive for making edits when Google started putting a neural
net in charge. (No! Stop deleting random words and squishing spaces during
edits).

In other words I figured it would be the annoyances at what "should be easy by
now" that would get Joe CEO to start thinking "Hm. Maybe this isn't such a
good investment." When measurements are made and reliable algorithmic results
attract and keep more users than narrowly trained kind of finicky AIs.

I don't want there to be an AI winter, and it won't be as bad as before. There
are a lot of applications for limited scope image recognition, and other tasks
that we couldn't do before. Unfortunately,I do agree with the post that winter
is on its way.

------
sytelus
The OP is obviously not keeping up with the field and has lot to learn about
scientific approach. He basically uses the count of tweets from AndrewNg and
crashes from risk-taking companies as indicator of "AI winter". He should have
tried to look in to metrics such as number of papers, number of people getting
in to field, number of dollars in VC money, number of commercial products
using DL/RL etc. But you see, that's a lot of work and your conclusion might
not align with whatever funky title you had in mind. Being an armchair opinion
guy throwing link bait titles is much more easier.

~~~
felippee
I'll happily read your next post where you will include all of those. In fact
amount of VC money spent in that field would only support my claim. And the
number of papers is irrelevant. There were thousands of papers about Hopfield
network in the 90's and where are all of them now? You see, all the things you
point out is the surface. What really matters is that self driving cars crash
and kill people, and no one has any idea how to fix it.

------
didymospl
I think the most important question is what 'winter' really means in this
context. The new concepts in AI tend to follow the hype cycle so the
disillusionment will certainly come. One thing is the general public see the
amazing things Tesla or Google do with deep learning and extrapolate this
thinking we're on the brink of creating artificial general intelligence. The
disappointment will be even bigger if DL fails to deliver its promises like
self-driving cars.

Of course the situation now is different than 30 years ago because AI has
proved to be effective in many areas so the research won't just stop. The way
I understand this 'AI winter' is that deep learning might be the current local
maximum of AI techniques and will soon reach the dead end where tweaking
neural networks won't lead to any real progress.

------
visarga
AI winter is not "on its way". There is AI hype and anti-AI hype, and then
there is actual practice. This article is anti-AI hype, just as bad as its
opposite. In practice there are tons of useful applications. We haven't even
begun to apply ML and DL to all the problems laying around us, some of which
are quite accessible and impactful.

The hype cycle will pass with time, when we learn to align our expectations
with reality.

~~~
meh2frdf
Give us a list then, because what I’m seeing is shoehorning “AI” into
everything but not with significant results.

~~~
visarga
It's not my job to supplement the press or to do paper reading for you. If
you're really interested in AI and don't just read the clickbait press, then
open Arxiv and look there.

[http://www.arxiv-sanity.com/](http://www.arxiv-sanity.com/)

If, on the other hand, your opinion that AI is in a winter has been already
decided without reading the latest scientific papers, then there's nothing I
can say to you that will change your mind.

------
zitterbewegung
I think that we will have AI Winter once we see the true limitations that face
us having a level 5 fully autonomous self driving car. The other thing we will
see happen is the deflation of the AdTech bubble. Once we see both of these
events occurring that should start the AI Winter.

~~~
jimothywales
I agree. The AdTech sphere is keeping the current hype alive more than
anything else. There's some obvious imbalances in AdTech that should lead it
to a damning end soon enough.

------
tananaev
AI and machine learning is a tool. Like any other tool it's perfect for some
problems and doesn't work well for other. Pick the right tools for the problem
that you are working on. Don't follow the hype and don't use AI/ML just for
sake of using it.

------
matiasz
Judea Pearl sees a way out of the winter.

[https://www.theatlantic.com/technology/archive/2018/05/machi...](https://www.theatlantic.com/technology/archive/2018/05/machine-
learning-is-stuck-on-asking-why/560675/)

~~~
AlexCoventry
I think a lot of GOFAI approaches ought to be revisited to see whether they
benefit from the new perceptual and decision capabilities of Deep Learning
systems. Alex Graves's papers are particularly good at this.

Things like this reinforcement learner for theorem proving are pretty exciting
possibilities.
[https://arxiv.org/pdf/1805.07563v1.pdf](https://arxiv.org/pdf/1805.07563v1.pdf)

------
OliverJones
There's a lot of good stuff coming from research in AI these days. Still, I
think the author's right.

As with the onset of the previous AI winter a generation ago, the problem is
this: Once a problem gets solved (be it OCR or Bayesian recommendation engines
or speech recognition or autocomplete or whatever) it stops being AI and
starts being software.

As for self-driving cars: I recently took a highway trip in my Tesla Model S.
I love adaptive cruise control and steering assistance: they reduce driver
workload greatly. But, even in the lab-like environment of summertime limited
access highways, driverless cars are not close. Autosteer once misread the
lane markings and started to steer the car into the side of a class 8 truck.
For me to sit in the back seat and let the car do all the work, that kind of
thing must happen never.

Courtesy is an issue. I like to exit truck blindspots very soon after I enter
them, for example. Autosteer isn't yet capable of shifting slightly to the
left or right so a driver ahead can see the car in a mirror. Maybe when
everything is autonomous that won't be an issue. But how do we get there.

Construction zones are problems too: lane markings are confusing and sometimes
just plain wrong, and the margin for error is much less. Maybe the Mobileye
rig in my car can detect orange barrels, but it certainly doesn't detect
orange temporary speed limit signs.

This author is right. AI is hype-prone. The fruits of AI generally function as
they were designed, though, once people stop overselling them.

------
rossdavidh
While I basically agree, really it ought to be called "AI autumn is well on
its way", since I'm not sure we're into actual winter (i.e. dramatic reduction
in $$ available for research) quite yet. But, probably soon.

~~~
felippee
Author here, yeah, it is the autumn. But I guess not many people would
recognize the meaning, winter on the other hand is not ambiguous...

~~~
rossdavidh
True.

------
epicmellon
"it is striking that the system spent long seconds trying to decide what
exactly is sees in front (whether that be a pedestrian, bike, vehicle or
whatever else) rather than making the only logical decision in these
circumstances, which was to make sure not to hit it."

That is striking. It always sort of bothered me that AI is really a big
conglomeration of many different concepts. What people are working on is deep
learning _for machines_ , but we think that means "replicating human
skill/behavior". It's not. Machines will be good at what they are good at, and
humans good at what they're good at. It's an uphill battle if your expectation
is for a machine that processes like a human, because the human brain does not
process things like computer architectures do.

Now, if some aspiring scientist wanted to skip all that and _really_ try to
replicate (in a machine) how the human brain does things, I think such a
person would be starting from a very different perspective than even modern AI
computing.

~~~
goatlover
That's why Augmented Intelligence is a better term. It doesn't scare up
visions of Skynet or Hal 9000 run amok. Nor does it promise utopian
singularity right around the corner.

It just means better tools to increase human capacity. But it's not nearly as
good at getting headlines in the media.

------
ozy
We call it deep learning, but it is deep pattern matching. Extremely useful,
but don't expect it to result in thinking machines.

~~~
halflings
Are our brains magic? If they aren't then surely they must be doing something
that we can reproduce. We've built so many things that we considered "thinking
machines" in the recent past (realistic speech synthesis, image recognition
and captioning, human-level translation, elaborate recommender systems, robust
question answering) on "deep pattern recognition".

~~~
ollin
Brains are not magic, and will be reproduced eventually, but DNNs are a
fundamentally weaker architecture and won't be enough. Neural nets can solve
some problems that brains can solve easily and lots of other ML methods
couldn't solve, which is great. But the space of problems that brains can
solve and neural nets can't is still rich, and will remain so until better
methods are developed.

------
dontreact
The discussion on radiology is extremely sloppy.

Andrew Ng claimed human level performance on one radiology task (pneumonia).
This claim seems to hold up pretty well as far as I can tell. Then the person
criticizing him on twitter posts results on a completely different set of
tasks which are just baseline results in order to launch a competition. These
results are already close to human level performance, and after the
competition it's very possible they will exceed human level performance.

Yes it's true that doing well at only Pneumonia doesn't mean that the nets are
ready to replace radiologists. However, it does mean that we now have reason
to think that all of the other tasks can be conquered in a reasonably short
time frame such that someone going into the field should at least consider how
AI is going to shape the field going forward.

------
cs702
Well, the breathless hype around deep learning (with and without reinforcement
learning) is bound to subside sooner or later, and attendance to staid
academic conferences like NIPS sooner or later will revert back to a smaller
group of academics and intellectuals who are truly interested in the subject
over the long term.[a] That much is certain.

But we're still in the early stages of a _gigantic wave of investment_ over
the next decade or two, as organizations of all sizes find ways to use deep
learning in a growing number of applications. Most small businesses, large
corporations, nonprofits, and governments are not using deep learning for
anything yet.

[a]
[https://twitter.com/lxbrun/status/908712249379966977](https://twitter.com/lxbrun/status/908712249379966977)

------
joejerryronnie
Well, now that the cat's out of the bag in regards to AI/ML, we can all get in
on the ground floor of the next hype wave - quantum computing!

~~~
zitterbewegung
IMHO Quantum computing is as well hyped as Cold Fusion and shares some of its
properties. Until "quantum supremacy" occurs or something that will show a
real speedup we won't hear that much from it.

~~~
AlexCoventry
Cold Fusion was outright scientific misconduct. I'm not optimistic about QC
working as intended, but I think the hope around it is honest.

------
tmalsburg2
Stopped reading after the first half. The evidence for the idea that deep
learning is failing is that Deep Mind haven't produced anything revolutionary
since Alpha Go Zero which was published not even a year ago? And that
preformance doesn't scale linearly with the number of parameters? And
speculation about why Lecun made a certain career decision? Not very
convincing.

~~~
YeGoblynQueenne
You're forgetting that Andrew Ng is tweeting 30% less this year! Isn't that
enough to convince even the staunchest critic?

------
soVeryTired
Only tangentially related to the article, but it's always struck me as a
little unethical that Demis Hassabis' name goes on every paper that's written
by Deepmind. No-one produces that much research output.

------
rscho
No, but wait! We're just on the verge of replacing doctors! ;-)

There's still a lot of space for the improvement of "curve-fitting" AI in the
workplace. The potential of existing tech is far from being thoroughly
exploited right now. I believe the next big improvements will come more from
better integration in the workplace (or road system) than new scientific
advances, so that might seem less sexy. But I also believe this will be a
sufficient impetus to drive the field forward for the years to come.

------
mirceal
I would not call it the “AI winter”. If you look at what people have called AI
over time, the definition and the approaches have evolved (sometimes
drastically) over time.

Instead of being stuck on the fact that deep learning and the current methods
seem to have hit a limit I think I am actually excited about the fact that
this opens the door for experimenting other approaches that may or may not
build on top of what we call AI today.

~~~
slowmovintarget
Perhaps it'd be more correct to call it a "Strong AI Winter". We're no closer
to "aware" machines. We've simply gotten very good at automating tasks that
were once difficult to automate.

~~~
mirceal
A friend that’s more optimistic about Strong AI once said that the ML that
goes on today will probably serve the purpose of driving the peripheral sense
organs of a future AI. Although it stretches a bit what’s possible today I
could see that. I would call this a win if this ends up happening although I
still belive we’re hundreds of years away from Strong AI.

~~~
randcraw
I'm inclined to agree with your friend.

This ability of DL to convert streams of raw noisy data into labeled objects
seems like exactly what's needed to solve an intelligent agent's perceptual
grounding problem, where an agent that's new to the world must bootstrap its
perception systems, converting raw sensory input into meaningful objects with
physical dynamics. Only then can the agent reason about objects and better
understand them by physical interaction and exploration. This is one of the
areas where symbolic AI failed hardest, but DL does best.

With some engineering, it's easy to imagine how active learning could use DL
to ground robot senses - much like an infant human explores the world for the
first year of life, adding new labels and understanding their dynamics as it
goes.

I suspect the potential for DL's many uses will continue to grow and surprise
us for at least another decade. If we've learned anything from the past decade
of DL, it's that probabilistic AI is surprisingly capable.

------
majos
This reminds me of a recent Twitter thread [1] from Zachary Lipton (new
machine learning faculty at CMU) arguing that radiologists have a more complex
job than we, as machine learning enthusiasts, think.

[1]
[https://mobile.twitter.com/zacharylipton/status/999395902996...](https://mobile.twitter.com/zacharylipton/status/999395902996516865)

------
carlbordum
I think all talk about computer intelligence and learning is bullshit. If I'm
right, then AI is probably the most /dangerous/ field in computer science
because it sounds just likely enough that it lures in great minds, just like a
sitcom startup idea[0].

[0]
[http://paulgraham.com/startupideas.html](http://paulgraham.com/startupideas.html)

------
tim333
You could actually make a reasonable argument for the opposite of a winter,
that we are heading into an unprecedented AI boom.

The article's main argument for a winter is that deep learning is becoming
played out. But this misses the once in history event of computer hardware
reaching approximate parity with and overtaking the computing power of the
human brain. I remember writing about that for my uni entrance exam 35 years
ago and have been following things a bit since and the time is roughly now.
You can make a reasonable argument the the computational equivalent of the
brain is about 100 TFLOPS which was hard to access or not available in the
past but you can now rent a 180 TFLOP TPU from Google for $6.50/hr. While the
current algorithms may be limited there are probably going to be loads of
bright people trying new stuff on the newly powerful hardware, perhaps
including the authors PVM and some of that will likely get interesting
results.

------
sheeshkebab
Deep learning maybe not the complete answer to gai, but it’s moving down the
right path. Computers though are still years/decades away from approaching
human brain power and efficiency, so my take is that current ai hype is 10
years too early - a good time to get in.

~~~
felippee
> but it’s moving down the right path

Time will tell. I think DL is amazing, but is no the right path towards
solving problems such as autonomy. I think if you enter this field today, you
should definitely take a look at other methods than DL. I actually spent a few
years reading neuroscience. It was painful, and I certainly can't tell I
learned how the brain works, but I'm pretty certain it has nothing to do with
DL.

------
ThomPete
Great essay but this "Deep learning (does not) scale" I think is missing an
important point.

There are many ways to think about scale.

If you think about a learned skill then that skill actually scales extremely
well to other machines and thus to other industries that might benefit from
the same skill.

The primary problem with technology is that society doesn't just implement it
as fast as it gets developed so you will have these natural bottlenecks where
society can't actually absorb the benefits fast enough.

In other words, Deep Learning scales as long as society can absorb it and
apply it.

------
paulie_a
Has anyone done something genuinely useful with ml/ai/whatever outside of
advertising or stock trading? I am genuinely curious if it has really been
applied to real commercial applications.

~~~
colordrops
Improvements in search, translation, image recognition and categorization,
voice recognition, and text to speech off the top of my head. I'm sure there
are a lot more.

~~~
paulie_a
Yeah but those are all pretty terrible to the actual end consumer. They might
be cool technologies but at the end of the day, I am a user that hates dealing
with them. IVRs are terrible a experience. Image recognition is iffy at best.
Text to speak is terrible. In 10 years maybe they will have it hashed out...
just like 10 years ago, or 20 years ago

~~~
colordrops
I'd have to disagree. While IVR sucks (most current implementations don't use
ML by the way), image recognition and categorization is at or better than
human levels in most cases now. Cutting edge TTS is now nearly
indistinguishable from a human. Just check out some samples [1]. And while
translation still sucks, ML based translation is still far better than
previous approaches.

[1] [https://www.theverge.com/2018/3/27/17167200/google-ai-
speech...](https://www.theverge.com/2018/3/27/17167200/google-ai-speech-tts-
cloud-deepmind-wavenet)

------
d--b
Sure the thing is overhyped, but the problem is that we cannot be sure about
the next big thing. The advances are slow but then a giant step forwards
happen all of a sudden.

Everyone dropped their jaws when they saw the first self driving car video or
when alpha go started to win. This was totally unthinkable 10 years ago.

Some guy may come up with a computer model that incorporates together
intentionality, some short term/long term memory, and some reasoning, who
knows?

------
randop
AI is favorable for big companies to better scale their services. It seems
that Facebook have also faced AI scaling drawbacks and they are developing
there own AI hardware for it
[https://www.theverge.com/2018/4/18/17254236/facebook-
designi...](https://www.theverge.com/2018/4/18/17254236/facebook-designing-
own-chips-ai-report)

------
letitgo12345
AI has a lot to offer to the industry right now I think where you don't need
good worst case performance (ex., information retrieval, optimization,
biology, etc.). The big problems in terms of application start appearing when
you try and remove humans from the loop completely. That's not even close to
possible yet but that doesn't mean the economic utility of even current AI is
close to being maximized.

------
moistoreos
I know this about the state of Deep Learning but I like to point out:

While autonomous driving systems aren't perfect, statistically they are much
better at driving than humans. Tesla's autonomous system has had, what, 3 or 4
fatal incidents? Out of the thousands of cars on the road that's less than
0.001%.

There will always be a margin of error in systems engineered by man, just
hopefully moving forward fewer and fewer fatal ones.

~~~
yourapostasy
Depending upon your statistical sources, US traffic fatalities are around
1.25-1.50 per 100 million miles. [1] All forms of real-world autonomous
driving across all manufacturers across the world are still somewhere below
200 million miles, conservatively estimated. [2] [3] Between the Tesla and
Uber fatalities, by these rough back-of-the-envelope numbers, autonomous of
various grades is still roughly 2X higher than human drivers. Maybe 1X if you
squint at the numbers hard enough, but likely not orders of magnitude lower. I
don't anticipate rapid legislative and insurance liability protections for
autonomous systems until we see orders of magnitude differences on a per 100
million miles driven basis, and that will take time.

Waymo racks up about 10,000 miles per day across about 600 vehicles spread in
about 25 cities. [4] Roughly 3.6 million miles per year if they stay level,
but they're anticipated to rapidly add more vehicles to their fleet. In the US
alone, about 3.22 trillion miles were driven in 2016. [5] Don't know what a
statistically valid sample size is based upon that (I get nonsensical results
below 2000 miles, so I'm doing something stupid), though. If Waymo puts two
orders of magnitude more cars out there, they'll still "only" rack up about
365 million miles per year, and not all the miles on the same version of
software.

[1]
[https://en.wikipedia.org/wiki/Transportation_safety_in_the_U...](https://en.wikipedia.org/wiki/Transportation_safety_in_the_United_States)

[2] [https://www.theverge.com/2016/5/24/11761098/tesla-
autopilot-...](https://www.theverge.com/2016/5/24/11761098/tesla-autopilot-
self-driving-cars-100-million-miles)

[3] [https://www.theverge.com/2017/5/10/15609844/waymo-google-
sel...](https://www.theverge.com/2017/5/10/15609844/waymo-google-self-driving-
cars-3-million-miles)

[4] [https://medium.com/waymo/waymo-reaches-5-million-self-
driven...](https://medium.com/waymo/waymo-reaches-5-million-self-driven-
miles-61fba590fafe)

[5] [https://www.npr.org/sections/thetwo-
way/2017/02/21/516512439...](https://www.npr.org/sections/thetwo-
way/2017/02/21/516512439/record-number-of-miles-driven-in-u-s-last-year)

------
tw1010
Woah, I was prepared to be all gung-ho for this post, given that I've
suspected the winter was going to be here for quite a while now. But
strangely, this post actually caused the opposite effect for me. The winter
will probably come one day, but is all the evidence the poster can find?
Andrew NG tweeting less and a statement that DNNs doesn't scale based on
flimsy data is not at all convincing to me.

------
tmaly
Is this AI Winter 2.0? I was hopeful that logic programming would have
developed more and spread to a larger audience at this point.

------
eddd
As a beginner in deep learning space, I am a bit baffled about the case "You
need a lot of computational power". Good models learn fast, so if potential
model looks promising on local machine, one can do training on gcloud for 100$
on high end machines. Where am I wrong in this line of thinking?

~~~
nl
No, you are absolutely right. And modern transfer learning improves this even
more in many domains.

------
fallingfrog
Thank god. We're definitely not ready and perhaps could never be ready for
true general purpose ai.

------
bewe42
This is something I always wondered about AI and it promises. Sometimes, the
last 1% is the hardest or can be even impossible. Self-driving cars, in
particular, are a good case. We get to solve 99% of the use cases but
achieving full autonomous vehicles might be just out of reach.

------
pascalxus
But, they're getting more and more data every year, right? All those almost
millions of teslas running around could provide enough video input for the
training data

Besides "Good software takes 10 years", according to Joel Spolsky. As I see
it, we're, what 5 year into ML.

~~~
meh2frdf
5 years ha! Wow the rebranding has worked well.

------
jgrant27
Another case in point. [http://www.latimes.com/local/lanow/la-me-ln-tesla-
collision-...](http://www.latimes.com/local/lanow/la-me-ln-tesla-
collision-20180529-story.html)

------
twtw
> Nvidia car could not drive literally ten miles without a disengagement.

From the same source as the author cites, that's because their test runs are
typically 5 miles and resuming manual control at the end of a test counts as a
disengagement.

------
partycoder
Deep Learning was a noticeable improvement over previous neural models, sure.
But deep learning is not the entire field of AI and ML. There has been more
stuff going on like neural turing machines and differentiable neural
computers.

------
crb002
We are beginning to see some sweet differential embeddings of discrete things
like stacks and context free grammars. This is where deep learning gets really
fun because it is learning to program.

------
jvmancuso
[https://twitter.com/jvmancuso/status/1002387357776207872](https://twitter.com/jvmancuso/status/1002387357776207872)

------
xbmcuser
For me Google is attacking on 2 main fronts 1\. Quntam computing 2\. Machine
Learning/AI

If they are able to combine the 2. A big if though the cost analysis will
change for AI quite dramatically.

------
bfung
Number of tweets as reliable data points? Very dubious. Simple explanation:
They are busy working, so less time to tweet.

Maybe they're working on something so cool, that the AI winter may not even
come. Sure, there's a lot of marketing-speak around AI at the moment.

But this wave of AI seems a lot stronger with better fundamentals than 20
years ago. At the very least, at least we have the hardware to actually RUN
NN's cost effectively now as oppose to grinding your system to a halt back
then.

Before AlphaGo, it wasn't even clear when a computer could beat a top
professional in go, let alone crush humans in the game - low bound guesses
were 50 years.

------
ggm
[https://en.wikipedia.org/wiki/Lighthill_report](https://en.wikipedia.org/wiki/Lighthill_report)
(1973)

------
xpuente
Low hanging fruits are scarce now. With 3 orders of magnitude difference in
power (MW over few watts), clearly this is not the right way for reaching the
tree top.

------
sigi45
/shrug people need time to research;

Anyway i also don't get what the issue is with the model from radiology. It is
already that good?! This is impressive. One model is close to well trained
experts.

Just today i had an small idea for a new product based on what google was
showing with the capabilities to distinguis two people talking in parallel.

At the last Google IO i was impressed because in comparision to the previous
years, ML created better and more impressive products.

I was listing for years at key nodes about big data and was never impressed. I
hear now about ML and im getting impressed more and more.

~~~
jimothywales
Google IO is a developer's conference with an emphasis on marketing its own
products and tools. We have to take the news from it with a grain of salt.

------
m0llusk
If only there were some technology that might enable us to discern patterns so
that we could better predict fluctuations in demand for AI software.

------
thosakwe
Truly, I agree.

I've long been interested in learning about AI and deep learning, but to this
day haven't done much that truly excites me within the field. It feels more or
less impossible to make anything significant without Google-scale databases
and Google-scale computers. AI really does make it easier for the few to jump
far ahead, leaving everyone behind.

I also agree that a lot the news around AI is just hype.

Honestly, I'm yet to see _anything_ practical come out of AI.

But hey, if something eventually does, I'm all for it.

~~~
tomatotomato37
I'm kinda curious now where those giant datasets will come from now that
there's a big push for privacy with things like GDPR preventing some random
researcher from just buying data off whatever data mining corp is most
relevant to their AI's purpose

~~~
confounded
I’m kinda curious now which researchers have been buying PII from data mining
corps.

~~~
nopriorarrests
Cambridge analytica comes to mind

------
m3kw9
Bet your house on it if it’s “well on it’s way”

~~~
John_KZ
Yeah this is a dumb article. Number of tweets by AndrewNg? Really? All those
articles denying the reality of the revolution brought by AI have an emotional
basis, but I don't understand what it is. Are they feeling threatened? Or is
it an undergrad/early 20s thing, like a complete lack of understanding of the
dynamics coupled with abnormally strong opinions?

------
jonbarker
This reinforces the need to benchmark any 'human expert equivalent' project
against the wattage of the human brain.

------
mathattack
How much of this can we pin on IBM's overhype of Watson?

------
ashelmire
Yawn. Contrarianism is easy and this article offers little. The real world
application you’re speaking of has a comically small amount of data (a few
million miles?). You hear about a handful of accidents that still average to
better than human performance and suddenly the sky is falling.

When machine learning stops successfully solving new problems daily, then
maybe a thread like this will be warranted.

------
arisAlexis
without being an expert just by reading articles it seems to me that some
people wish foe an AI winter. It makes them feel better somehow

------
Bromskloss
Oh, I thought "AI winter" would refer to a state of ruin after AI had come
into existence and destroyed everything, analogous to nuclear winter.

~~~
edanm
AI Winter is a very well-known term in the industry referring to a general
lack of funding of AI research, after the last time AI was overhyped.

~~~
Bromskloss
I guess it could be used about anything that experiences a low level of
interest, then.

------
scalablenotions
A real Winter is a lack of warmth. An AI winter is a lack of ______

------
InclinedPlane
If we would stop calling this stuff "AI" it would make all our lives a lot
easier, but people can't resist.

When computers first came on the scene a lot of people had a very poor
conception of what it was the human mind did, computationally. So when
computers turned out to be good at things that were challenging "intellectual"
tasks for humans like chess and calculus many were duped into thinking that
computers were somehow on a similar level to human brains and "AI" was just
around the corner. The reality was that one of the most important tasks that
the human brain performs: contextualization, categorization, and abstraction
was taken for granted. We've since discovered that task to be enormously
computationally difficult, and one of the key roadblocks towards "true AI"
development.

Now, of course, we're at it again. We have the computational muscle to make
inference engines that work nothing like the human brain good at tasks that
are difficult to program explicitly (such as image and speech recognition) and
we've built other tools that leverage huge data sets to produce answers that
seem very human or intelligent (using bayesian methods, for example). We look
at this tool and too many say "Is this AI?" No, it might be related to AI, but
it's just a tool. Meanwhile, because of all the AI hype people overpromise on
neural networks / "deep learning" projects and people get lazy about
programming. Why bother sitting down for 15 minutes to figure out the right
SQL queries and post processing when you can just throw your raw data at a
neural network and call it the future?

One of the consistently terrible aspects of software development as a field is
that it continues to look for shortcuts and continues to shirk the basic
responsibilities of building anything (e.g. being mindful of industry best
practices, understanding the dangers and risks of various technologies and
systems and being diligent in mitigating them, etc.) Instead the field
consistently and perversely ignores all of the hard-won lessons of its
history. Consistently ignores and shirks its responsibilities (in terms of
ethics, public safety, etc.) And consistently looks for the short cut and the
silver bullet that will allow them to shirk even the small vestiges of
responsibility they labor under currently. There's a great phrase on AI that
goes: "machine learning is money laundering for bias", which points to just
one facet among so many of what's wrong with "AI" as it's practiced today. We
see "AI" used to sell snake oil. We see "AI" used to avoid responsibility for
the ethical implications inherent in many software projects. We see "AI"
integrated into life critical systems (like self-driving cars) without putting
in the effort to ensure it's robust or protect against its failures, with the
result being loss of life.

AI is just the latest excuse by software developers to avoid responsibility
and rigor while cashing checks in the meantime. At some point this is going to
become obvious and there is going to be a backlash. Responsible developers
should be out in front driving for accountability and responsibility now
instead of waiting until a hostile public forces it to happen.

------
nolemurs
I've always understood the claim that deep learning scales to be a claim about
deployment and use of trained models, not about training. The whole point is
that you can invest (substantial) resources upfront to train a sufficiently
good model, but then the results of that initial investment can be used with
very small marginal costs.

OP's argument on this front seems disingenous to me.

His focus on Uber and Tesla (while not even mentioning Waymo) is also a truly
strange omission. Uber's practices and culture have historically been so toxic
that their failures here are truly irrelevant, and Tesla isn't even in the
business of making actual self driving cars.

I'm the first to argue that right now AI is overhyped, but this is just
sensationalist garbage from the other end of the spectrum.

~~~
felippee
Hi, it appears that "sensationalist garbage" triggered quite a bit of a
discussion. This is typically indicative that the topic is "sensitive".
Perhaps because many people feel the winter coming as well. Maybe, maybe not,
time will tell.

And FYI, Tesla is in the business of making self driving car. If you read the
article, you might learn that Tesla is actually the first company to sell that
option to customers. You can go to their website right now and check that out.

Uber, like it or not is one of the big players of this game. I agree they may
have somewhat toxic culture, but I guarantee you there are plenty of really
smart people there who know exactly the state of the art. And their failure is
therefore indicative of that state of the art.

I also omitted Cruise automation and a bunch of other companies, perhaps
because they have more responsible backup drivers that so far avoided fatal
crashes. But I analyze the California DMV disengagement reports in another
post if you care to look. And by no means any of these cars is safe for
deployment yet.

~~~
nolemurs
> Hi, it appears that "sensationalist garbage" triggered quite a bit of a
> discussion.

Yes. Sensationalist.

> I also omitted Cruise automation and a bunch of other companies, perhaps
> because they have more responsible backup drivers that so far avoided fatal
> crashes.

So your explicit reason for omitting Waymo, as I understand it, is that it
didn't support your argument?

~~~
felippee
> Yes. Sensationalist.

Yes, perhaps. But I'm entitled to my opinion just as you are entitled to
yours. And time will tell who was right.

> So your explicit reason for omitting Waymo, as I understand it, is that it
> didn't support your argument?

You see, when you make any argument, you always omit the infinite number of
things that don't support it and focus on the few things that do. The fact
that something does not support my argument, does not mean it contradicts it.

You might also note that this is not a scientific paper, but an opinion. Yes,
nothing more than an opinion. May I be wrong? Sure. And yet this opinion
appears to shared by quite a few people, and makes a bunch of other people
feel insecure. Perhaps there is something to it? We will see.

But in the worst case it will make some people think a bit and make an
argument either for or against it. I may learn today a good argument against
it, that will make me think about it more and perhaps I will change my
opinion, or I'll be able to defend it.

So far you have not provided such an argument, but I wholeheartedly encourage
you to do so.

~~~
gliboc
This is a list of your phrases in this comment that I find, in my opinion,
condescending.

> And time will tell who was right.

> You see, when you make any argument

> You might also note that this is not a scientific paper, but an opinion.
> Yes, nothing more than an opinion.

> And yet this opinion appears to shared by quite a few people, and makes a
> bunch of other people feel insecure. Perhaps there is something to it? We
> will see.

> So far you have not provided such an argument

I immediately identified this same tone in your paper. In your argumentation,
you quite agressively hinted hat people which don't share your views are not
very intelligent. You also have a tendency to present your sayings as
prophetic, which appeared multiple times both in the paper and in this
comment.

These observations put me in alarm towards your arguments, which I found
mostly weak, sometimes used in bad faith. I flagged as such the Twitter
argument, analysing the frequency of A. Ng's tweets, and denouncing its
"outrageous claims", with an example where the AI score is overall only 0.025
less accurate than a practician.

I also thought that you used a different (your own) definition of scaling than
most, and used it to make an argument, which was therefore unconvincing (but
parent said that already).

Overall, to me, this was not a very pleasant read, and I dislike the fact that
you attack the hype on machine learning by enjoying the polarization that
comes with anti-hype articles such as yours. I also don't think that making
people feel insecure is such a great indicator that what you're saying is
relevant or prophetic.

I hope this helps you prophecies [https://www.physics.ohio-
state.edu/~kagan/AS1138/Lectures/Go...](https://www.physics.ohio-
state.edu/~kagan/AS1138/Lectures/GottIII.html) ;)

------
jacksmith21006
One of the more silly articles on HN in a while. Waymo has cars as I type this
driving around Arizona without safety drivers.

People were freaked out by the Google demo of Duplex a couple of weeks ago as
it was just too human sounding.

Can give so many other example. One is foundational. The voice used with
Google Duplex is using a DNN at 16k cycles a second in real-time and able to
offer at a competitive price.

That was done by creating the TPU 3.0 silicon. The old way of piecing together
was NOT compute intensive and therefore doing it using a DNN requires
proprietary hardware to be able to offer at a competitive price to the old
way.

But what else can be done when you can do a 16k cycles through a DNN in real-
time? Things have barely even got started and they are flying right now. All
you have to do is open your eyes.

DNN - Deep Neural Network.

------
myf01d
It's the same story again like exaggerating the influence of IoT 5 years ago.
The whole thing is exaggerated to raise money from investors and attract
customers instead of actually buidling superior product

~~~
cooper12
It's 99% marketing and places like HN and reddit eat it up and try to hype it
up even more. When you confront these characters about the basis on which they
claim AI will solve whatever problem or evolve to whichever point, they only
reply "it'll only keep getting better (given time, data, resources, brains,
etc)"

It's a buzzword people brainlessly use to fetishize technological progress
without understanding the inherent limitations of the technology or the actual
practicality and real-life results outside of crafted demos or specific
problem domains (for example Alpha Go beating a grandmaster has almost no
bearing on a problem like speech cognition).

It's turned me off a lot from reading about advances in the field because I
know like a lot of science releases that most of it is empty air that won't
really have bearing on the actual software I use (I've watched the past two
Google's I/O where pretty much every presentation mentions AI, but the Android
experience still remains relatively stale).

------
bguberfain
Deep Recession ‘18

------
kuroguro
Winter is coming.

------
jacinabox
What a relief.

------
fourfaces
The inconvenient but amazing truth about deep learning is that, unlike neural
networks, the brain does not learn complex patterns. It can see new complex
patterns and objects instantly without learning them. Besides, there are not
enough neurons in the brain to learn every pattern we encounter in life. Not
even close.

The brain does not model the world. It learns to see it.

~~~
edejong
This post is very uninformed.

"It can see new complex patterns and objects instantly without learning them."

Except, it doesn't. It is clearly false. When animals grow up in an
environment without certain patterns, they will be unable to see these
patterns (or complex combinations of these) at a later stage. We see complex
patterns as combinations of patterns we have seen before and semantically
encode them as such. This is very similar to how neural networks work at the
last fully connected layers.

"Besides, there are not enough neurons in the brain to learn every pattern we
encounter in life."

There is a lot of self-similarity in our environment. Compression algorithms
(and NN auto-encoders) are able to leverage this self-similarity to encode
information in a very small number of data-points / neurons.

"The brain does not model the world. It learns to see it."

Except, it doesn't. Your brain continually makes abstractions of the world.
When you 'see' the world you see a (lossy) compressed version of it,
compressed towards utility. Similar to how MP3 compression works: the
information gain of higher frequencies is low, so your brain can safely filter
these out.

~~~
erikpukinskis
We learn to see patterns, but we see through physical and cultural action
patterns that are simply present, not learned.

It’s like a river flowing... yes, the water molecules each “discover” the
their path, but the path of the river is a property of the landscape. It is
not learned.

