
Greedy, Brittle, Opaque, and Shallow: The Downsides to Deep Learning - monsieurpng
https://www.wired.com/story/greedy-brittle-opaque-and-shallow-the-downsides-to-deep-learning/
======
Animats
Brittle and opaque are real problems. The brittleness seems to be associated
with systems which put decision surfaces too close to points in some
dimension. That's what makes strange classifier errors possible.[1] (This is
also why using raw machine learning for automatic driving is a terrible idea.
It will do well, until it does something totally wrong for no clear reason.)

Opacity comes from what you get after training - a big matrix of weights. Now
what? "Deep Dream" was an attempt to visualize what a neural net used for
image classification was doing, by generating high-scoring images from the
net. That helped some. Not enough.

The ceiling for machine learning may be in sight, though. Each generation of
AI goes through this. There's a new big idea, it works on some problems,
enthusiasts are saying "strong AI real soon now", and then it hits a ceiling.
We've been through that with search, the General Problem Solver, perceptrons,
hill-climbing, and expert systems. Each hit a ceiling after a few years. (I
went through Stanford just as the expert system boom hit its rather low
ceiling. Not a happy time there.)

The difference this time is that machine learning works well enough to power
major industries. So AI now has products, people, and money. The previous
generations of AI never got beyond a few small research groups in and around
major universities. With more people working on it, the time to the next big
idea should be shorter.

[1] [https://blog.openai.com/adversarial-example-
research/](https://blog.openai.com/adversarial-example-research/)

~~~
TulliusCicero
I haven't seen anyone saying that the current trend of deep learning using
neural networks will result in strong AI soon.

~~~
nora4
Except for Elon Musk, who with all due respect, I believe does not count as an
informed person in DL.

It really bothers me that he (who is clearly an excellent entrepreneur and
human being in general) makes such strong statement on a topic he is clearly
not an expert in and is so vocal about it too. Given the amount of influence
and number of online followers he has, I find this irresponsible.

~~~
Animats
I'm worried about machine learning taking over corporate management.
Optimizing for shareholder value is a goal a machine learning system can work
on.

~~~
TeMPOraL
Yup. Machine learning driving the optimization loop, with humans working as
smart sensors and actuators, responding to high-level commands. That's a
pretty powerful AI right there.

------
ssivark
This article is a little too glib in my opinion, preferring citations and
statements to substance and explanations.

For a more cutting and insightful critique, watch Ali Rahimi's short talk at
NIPS 2017 (where he was presenting for a paper that won the "Test of time"
award, for standing out in value a decade after publication). The standing
ovation he received at the end indicate that his comments resonated with a
significant fraction of the attendees.

[https://www.youtube.com/watch?v=Qi1Yry33TQE&feature=youtu.be...](https://www.youtube.com/watch?v=Qi1Yry33TQE&feature=youtu.be&t=660)

Here's a teaser from the talk:

"How many of you have devised a deep neural net from scratch, architecture and
all, and trained it from the ground up, and when it didn't work, felt bad
about yourself, like you did something wrong? This happens to me about every
three months, and let me tell you, I don't think it's you [...] I think it's
gradient descent's fault. I'll illustrate..."

[Addendum]

Ben Recht and Ali Rahimi published an adendum to the talk, elaborating on the
direction they envision -- [http://www.argmin.net/2017/12/11/alchemy-
addendum/](http://www.argmin.net/2017/12/11/alchemy-addendum/)

Ali also has a post taking a stab at organizing some puzzling basic
observations about deep learning, and motivating that with analogous
historical progress in optics --
[http://www.argmin.net/2018/01/25/optics/](http://www.argmin.net/2018/01/25/optics/)

\-----

PS: The first 11 minutes, on the idea of using random features (the main idea
in the research he presented) are also interesting.

~~~
skybrian
In case you'd rather read text than watch a video, I believe this is the same?

[http://www.argmin.net/2017/12/05/kitchen-
sinks/](http://www.argmin.net/2017/12/05/kitchen-sinks/)

------
YeGoblynQueenne
>> Google Translate is often almost as accurate as a human translator.

This is the kind of overhyped reporting of results highlighted by Douglas
Hofstadter in his recent article about Google Translate:

 _I’ve recently seen bar graphs made by technophiles that claim to represent
the “quality” of translations done by humans and by computers, and these
graphs depict the latest translation engines as being within striking distance
of human-level translation. To me, however, such quantification of the
unquantifiable reeks of pseudoscience, or, if you prefer, of nerds trying to
mathematize things whose intangible, subtle, artistic nature eludes them. To
my mind, Google Translate’s output today ranges all the way from excellent to
grotesque, but I can’t quantify my feelings about it._

[https://www.theatlantic.com/technology/archive/2018/01/the-s...](https://www.theatlantic.com/technology/archive/2018/01/the-
shallowness-of-google-translate/551570/)

It's funny how the article above is claiming to speak of "the downsides" to
deep learning, yet it spends a few paragraphs repeating the marketing pitch of
Google, Amazon and Facebook, that their AI is now as good as humans in some
tasks (limited as they may be) and all thanks to deep learning. To me that
goes exactly counter to the article's main point and makes me wonder, what the
hell is the author trying to say- and do they even know what they're talking
about?

~~~
StillBored
That article pretty much sums up what my sister in law (runs a professional
translation business) says.

To me, someone who doesn't have a second language strong enough to verify the
accuracy of translation, I simply run things through google translate, to the
target language and then run the output back to English. Its the translation
equivalent of the game of telephone.

Take "the fat cat sat on the mat"):

"the big cat was sitting on the carpet" or "The cats of oil sat on the bed"

and lots of other things which are just odd..

~~~
Balgair
What does this prove?

Languages are different, you _should_ expect any automatic translation program
to do something like this, as it is trying to translate without prior
knowledge of the 'conversation' you are having.

Many languages put a lot of meaning into a single word. Look at English.
"Tank" is good example. It's either a thing that holds a lot of fluid, a thing
that blows up buildings on treads, or a verb that means you are taking a lot
of damage in lieu of others taking damage. One word has a lot of meaning. Then
you can get into conjugations and tenses, yeesh.

Google translate is _not_ meant to be a natural language processor, it's just
a dumb translator. It can't figure out context as it is just a simple text box
and doesn't look at the million and one things natural language processing
would use.

Trying to play telephone with it just proves that a dumb text box is dumb.

~~~
StillBored
Its proves that _really_ simple concepts aren't being conveyed despite the
claims that it is more than simple word substitution. The linked article talks
about translation of things which require some cultural knowledge. The idea
that an overweight house cat is sitting on a protective piece of material is
simply lost, and replaced with "big cat" which could just as well be a lion or
a bed is misleading at best and just plain wrong at worse.

I might understand if the target languages didn't have a concept because its
cultural. That sentence failing to be translated implies that the target
language doesn't have the concept of people/animals being overweight, or what
a mat is. I might be more accepting if it came back with "throw rug" or
something instead of mat, but it never does that, its like it has a list of
rough synonyms and it picks one at random. Hence the "Fat cat" bit becoming
what likely was "oily cat". The more subtle things (cat in this case being a
house cat/pet) might be the bit of cultural information that lends
understanding to the whole sentence, but it most cases that isn't really what
the translation is getting wrong..

Claiming any of this is even near human translation levels is misleading at
best, considering it falls down worse than most poorly translated computer
manuals I've read. Despite all the claims of the wonders of DNN's the
translations look little more advanced than direct word translations (fat=oil
or mat=something you sleep on) with a bit of fuzz that fails in strange ways.

~~~
jeffdavis
Rather than translating the description into an abstract idea, and then
translating that idea into a different language; it seems like it's trying to
go straight from English to Chinese.

~~~
Balgair
To be fair, translating to 'idea-space' is really hard. The medium is the
message. We have ideas in English that other languages do not have, and vice
versa. Chinese is famous for not having a past tense. Spanish has a
subjunctive tense that is difficult for fluent speakers to translate into
English. Some languages have cardinal direction based gender tenses. The word
'che' in Italian is a head-tweaker for English speakers.

The problem expands as a binomial (handshaking problem). Google translate has
104 languages, which means 5356 cases to deal with (n(n-1)/2) for each 'idea'
present. As such, the 'idea' could be translated, but it would result in a
mess of a response that no native speaker would translate the 'idea' into.
You'd have to teach the basics of the language to the person before real
translation could occur. Languages are meant to communicate between two people
(writing is subtly, but importantly, different) and context, body language,
and recent history play large roles into that communication.

------
andbberger
I've said it before and I'll say it again. Machine learning is specifically
_not_ magic. It only works to the extent that we can build our own priors into
the model.

A typical media story... deep learning really is great. It represents the
first time we've really figured out how to do large-scale nonlinear
regression. But it is certainly not a magic bullet. However moderate headlines
don't get as many hits as overhyped ones so every day we get another
ridiculous article spouting nonsense...

Very tiresome. The truth is pretty interesting, can we talk about that
instead?

For instance, how and why deep learning works at ALL is very much an open
question. Consider - we're taking an incredibly nonlinear, nonconvex
optimization problem and optimizing in just about the dumbest way imaginable,
first-order gradient descent. It is really amazing that this works as well as
it does.

... Why does deeper work better than wider? It has been known for many years
that a shallow net has equivalent expressivity to a deep one. So what gives?
(actually some interesting work towards answering this question in recent
years by Sohl-Dickenstein et. al)

~~~
pps43
> Machine learning [..] only works to the extent that we can build our own
> priors into the model.

AlphaZero is a good counterexample. In contrast to AlphaGo, it has no priors.

~~~
aoeusnth1
Well it does have some priors baked into it by the (convolutional)
architecture of the network. See the “deep image prior” project for a feel for
just how strong this convolutional prior is.

------
visarga
Humans, as opposed to deep learning, have embodiment. We can move about, push
and prod, formulate ideas and test them in the world. A deep net can't do any
of that in the supervised learning setting. The only way to do that is inside
an RL agent. The problem is that any of our RL agents so far need to run
inside a simulated environment, which is orders of magnitude less complex than
reality. So they can't learn because they can't explore like us.

The solution would be to improve embodiment for neural nets and to equip RL
agents with internal world simulators (a world model) they could use to plan
ahead. So we need simulation both outside and inside agents. Neural nets by
themselves are not even the complete answer. But what is missing is not
necessarily a new algorithm or data representation, it's the whole world-agent
complex.

Not to mention that a human alone is not much use - we need society and
culture to unlock our potential. Before we knew the cause, we believed disease
was caused by gods, and it took many deaths to unlock the mystery. We're not
perfect either, we just sit on top of the previous generations. Another
advantage we have - we have a builtin reward system that guides learning,
which was created by evolution. We have to create this reward system for RL
agents from scratch.

In some special cases like board games, the board is a perfect simulation in
itself (happens to be trivial, just observe the rules, play against a replica
of yourself). In that case RL agents can reach superhuman intelligence, but
that is mostly on account of having a perfect playground to test ideas in.

In the future simulation and RL will form the next step in AI. The current
limiting factor block is not the net, but the simulator. I think everyone here
has noticed the blooming of many game environments used for training RL agents
from DeepMind, OpenAI, Atari, StarCraft, Dota2, GTA, MuJoCo and others. It's a
race to build the playground for the future intelligences.

Latest paper from DeepMind?

> IMPALA: Scalable Distributed DeepRL in DMLab-30. DMLab-30 is a collection of
> new levels designed using our open source RL environment DeepMind Lab. These
> environments enable any DeepRL researcher to test systems on a large
> spectrum of interesting tasks either individually or in a multi-task
> setting.

Before we build an AI, we need to build a world for that AI to be in.

~~~
bryananderson
Embodiment is part of the picture, but I also am not sure that we have yet
developed structures that are capable of learning in the way that humans do,
no matter how rich the environment.

Rocks, flatworms, and the Hoover Dam all have embodiment in the same complex
world that we do, but none of them will ever coherently debate philosophy,
because they have no structures capable of learning to do so. I’m not
convinced that our RL agents do either.

~~~
visarga
That's easy. Flatworms can't because we took the top spot and hogged
resources. Give them a world where they are the most advanced species, and a
few billion years, and they can debate philosophy too. What would happen if a
flatworm came out of the sea and tried to take up land from us?

Hoover Dam is not embodied because it is not an agent. An agent has good an
bad, life and death. The Dam doesn't give a damn about any of it.

------
inthewoods
There are groups and companies exploring probabilistic programming as an
alternative to CNN and other deep learning techniques. Gamalon
(www.gamalon.com) combines human and machine learning to provide more accurate
results while requiring much, much less training data. The models it generates
are also auditable and human readable/editable - solving the "opaque" issue
with deep learning techniques. Uber is exploring some of the same techniques
with their Pyro framework.

Having said all of this, we're not arguing that CNN have no place - in fact
you can view CNNs as just a different type of program as part of an overall
probabilistic programming framework.

What we're seeing is that the requirement of large labeled training sets
becomes a huge barriers as complexity scales - making understanding complex,
multi-intent language challenging.

Disclosure: I work for Gamalon

~~~
BenoitEssiambre
My intuition is that the approach Gamalon is using has more potential than
deep learning.

I've been playing with the concept for a while however failing to get any good
results. Debugging probabilistic programs is so damn long and difficult since
bugs can show as subtle biases in output instead of clear cut deviations. (I
described my approach here: [https://www.quora.com/What-deep-learning-ideas-
have-you-trie...](https://www.quora.com/What-deep-learning-ideas-have-you-
tried-that-didnt-work/answer/Benoit-Essiambre) ) For me, this is just a hobby.

Joshua B. Tenenbaum et al.'s group seem to name their approach program
induction, I had called mine Bayesian Auto Programming. I see you are calling
yours Bayesian Program Synthesis. Clearly we have similar intuition about the
essence of the solution to AI.

I wish you better luck than me.

~~~
mov
Very interesting report on your discoveries. Makes me remember of Genetic
Programming. Was thinking about using the same principles to generate a more
declarative Bayesian program (like a subset of SVG).

Do you have some source available around your experiments?

------
henrik_w
Another good article in a similar vein: "Asking the Right Questions About AI"

[https://medium.com/@yonatanzunger/asking-the-right-
questions...](https://medium.com/@yonatanzunger/asking-the-right-questions-
about-ai-7ed2d9820c48)

HN discussion:
[https://news.ycombinator.com/item?id=16286676](https://news.ycombinator.com/item?id=16286676)

------
polotics
I love the way the articles trots out the line that Google translate is almost
as good as a human translator, in view of Hofsdadter's recent article.

~~~
seandhi
If good is defined by more factors than quality of translation, maybe. But
translations between the two languages I speak, English and Spanish, are never
good enough to actually use without a lot of follow up work by a human.

~~~
mannykannot
> If good is defined by more factors than quality of translation, maybe.

That translates to 'is not good' in my language. Your low-key sarcasm is much
appreciated.

~~~
JumpCrisscross
> _That translates to 'is not good' in my language_

The original line is more precise. In most cases, I need the gist of a block
of text in an unfamiliar language. Google Translate is faster and cheaper than
a human translator. That makes it better, for this purpose, than a human,
despite its lower accuracy.

When I need legal documents translated, most human translators are not good
enough. A specialist, and only a specialist, will do.

------
gaius
Calling it "deep learning" was the first mistake. It makes it sound a lot more
profound than it really is. "Machine intuition" is the term I prefer.

~~~
Quarrelsome
I prefer "just throwing cycles at the problem".

~~~
floatboth
You can throw cycles in various ways (like with regular algorithmic code that
implements various heuristics or something). So, "just throwing cycles at
statistics"

------
csours
We had a presentation at work about AI and Deep Learning a while ago, and I
asked what is the test approach or test plan for deep learning... the answer I
got was a strange look.

If you have a self driving car crash and the cause is "the algorithm", that's
not going to be satisfying to customers, insurance agencies, or regulators [I
should be clear, the team giving the presentation does not work on SDCs].

------
John_KZ
This article presents zero evidence or indications for their claim. One
argument is persistence. Quoting: " “We are born knowing there are causal
relationships in the world, that wholes can be made of parts, and that the
world consists of places and objects that persist in space and time,” Which is
unsubstantiated and irrelevant, because AI that understands 3D is only now
being developed. Also children up to 3 years of age or so, cannot understand
the perspective of 3rd parties. There might be some hard-wired rules in our
brain, but that's not intelligence anyway.

The article has a point about something: Conventional feed-forward,
convolutional neural networks can only model a very limited space in the grand
scheme of things. Backpropagation is not perfect. There are other learning
methods. Hell, sometimes there isn't a global minimum anyway. But saying that
Deep Learning will stall in the near future is just wrong, and in my opinion
the reasons why are evident to all those who follow the latest developments.

------
dsign
It's the combined enthusiasm of academics and entrepreneurs the one driving
the current revolution on artificial intelligence. Said enthusiasm is
punctuated by high profile CEOs and investors making fiery remarks from time
to time. But alas, we people are not enthusiastic forever about something, and
sooner or later our collective psyche will move on.

That doesn't mean that the technology revolution, and it's AI component will
stop. We have had machine learning for a long time, doing its thing, as best
we managed to make it work. It's impact inside our collective speech was more
subdued, that for sure, but it was there. Research and development never
stopped. Even if some big name university funded it less and shut up about
building GAI, well there were still thousands of less shiny institutions and
companies working in more tractable problems and building a foundation.

And, to be clear, I wish the current collective enthusiasm lasts a little bit
longer, because we have a long way to go still and research grants and
investment money flow better when the media is abuzz with the subject. In
particular, we need to either move out or build upon matrix crunchers like
deep learning. Better forms of AI and eventually GAI will need a little bit of
innovation in chip-making and in computing architectures in general.

------
dmix
> Marcus believes that deep learning is not “a universal solvent, but one tool
> among many.” And without new approaches, Marcus worries that AI is rushing
> toward a wall, beyond which lie all the problems that pattern recognition
> cannot solve.

The thing that fascinates me is that we've only just scratched the surface of
what today's ML tech _can_ solve. Let's worry about that wall when we get
there...

In the meantime let's not lose sight of today's potential in some misguided
idealistic pursuit of perfectionism or "general artificial intelligence".

There are countless problems which current deep learning research combined
with some well-thought out UI/UX could solve today in a myriad of industries.

The 1990's software 'revolution' in industry/business was largely just
formalization/automation of paper-based processes into spreadsheets and simple
databases, which then evolved into glorified CRUD/CMS software interfaces on
desktops, then web/SaaS, and then another massively boost with smartphones.

If such a simple translation of human processes into machines can achieve
trillions of dollars in value then there is no doubt machine learning can do
the same for hundreds of thousands of other simple problem-sets which we
haven't even considered. Plus the desktop/smartphone/internet/etc
infrastructure is already in place for it to be plugged into.

This can only be negatively judged in the context of all significant steps
forward in technology being oversold and misunderstood. But in practical real-
world utility we're very far from fully utilizing what has been researched and
accomplished today in a small set of markets. And the proliferation of this
tech should be encouraged, promoted, and accurately communicated to
tech/business talent who can potentially use it, rather than downplayed
because it fails to live up to some media hyperbole or SciFi fantasies of
where we _should_ be in 2018.

The article mentions that taking AI/ML/data science courses has just become
the "hottest new field" for young smart kids to join. Well that means we're
just on the cusp of taking advantage of that technological evolution and it's
FAR too early to look at what's been accomplished today and be pessimistic
about deep learnings potential.

------
mcguire
I would argue that 'greedy' applies to neutral networks in a different sense:
they seize on their first solution; they have no mechanism to re-evaluate a
decision that is proving false.

~~~
hadsed
Sure they do, it happens at training time. Each time you update parameters you
look at all your predictions and figure out how well you did. That informs
your next tweak to the parameters. Run online training and you are constantly
doing this.

------
sixtypoundhound
Meta Thought: Isn't this basically describing the difference between human
children and human adults? How effectively they can bridge known and unknown
context?

------
fellellor
The article seems a bit outdated in that Hinton himself has argued for the
need to go beyond backpropagation.

~~~
giardini
Marcus' article mentions Hinton's recent misgivings about backprop:

 _" Perhaps most notably of all, Geoff Hinton has been courageous enough to
reconsider has own beliefs, revealing in an August interview with the news
site Axios 16 that he is “deeply suspicious” of back-propagation, a key
enabler of deep learning that he helped pioneer, because of his concern about
its dependence on labeled data sets. Instead, he suggested (in Axios’
paraphrase) that “entirely new methods will probably have to be invented.” I
share Hinton’s excitement in seeing what comes next._

------
bobthechef
> “We are born knowing there are causal relationships in the world, that
> wholes can be made of parts, and that the world consists of places and
> objects that persist in space and time,” he says.

Here's a little too overconfident in the claim that these are innate ideas or
something a priori.

------
jokoon
I am still wondering, why is it not possible, or maybe too hard, to take a
trained neural network and reduce its amount of neurons to get a simplified
solution to a problem?

Is there some theoretical or mathematical analysis of neural networks?

------
vladislav
If the hypothesis this article is aiming to strike down is that there is
currently a clear path to solving AGI, then it's simply fighting a potential
misconception of the masses. The reality is that recent advances in AI, while
not providing the full picture, are quite strong. Object recognition ten years
ago, even at the level of 2011's Imagenet results, seemed impossible. The
downsides to deep learning mentioned in the article, are both clear side
effects of the formulation, and simultaneously being addressed by the
community in various ways. For what it's worth, I'm with LeCun and Hinton on
this one. The abstract human reasoning required for even high level perceptual
tasks is difficult for humans to consciously dissect, but it could easily be
that it's actually relatively mundane when subjected to appropriate
representations.

~~~
bobthechef
> The abstract human reasoning required for even high level perceptual tasks
> is difficult for humans to consciously dissect, but it could easily be that
> it's actually relatively mundane

I don't think so. Philosophers have theorized about abstract thought for
thousands of years. The materialist metaphysics that is uncritically and
unconsciously held by the triumphalist philistines isn't even capable IN
PRINCIPLE of accounting for it.

In any case, AI is useful, and will continue to be, but the fanfare is just
noise distracting from the truth of what it is.

~~~
vladislav
relatively mundane _when subjected to appropriate representations_.

You omitted the important part of my statement. And the difficulty humans
encounter in reasoning about the nature of abstract thought itself is
something I also pointed out. My point is that this is not necessarily an
indication of inherent difficulty. The ability to model and explain the nature
of a self-reflecting mechanism is not a guaranteed natural ability of that
mechanism itself.

------
anonytrary
Keep in mind, folks -- the following is also true:

> Greedy, Brittle, Opaque, and Shallow: The Downsides to Evolved Human
> Intelligence

I wonder if we can have our cake and eat it too, in the realm of machine
learning and artificial intelligence.

------
skybrian
The problem I see with this article is that it starts with problems with deep
learning today and extrapolates to say that researchers won't get past them:

"None of these shortcomings is likely to be solved soon."

There's no evidence for this, and I think it underestimates the large and
well-funded machine learning community. Maybe they'll run into a brick wall,
but who's to say a bunch of smart researchers won't figure out ways around
today's limitations?

Without physical constraints or impossibility results, I think the only
rational outside view is that this is unpredictable. For any of today's
pressing problems with machine learning, maybe someone will post a new
research paper tomorrow with a different approach that solves it. Or maybe
not?

~~~
randcraw
The 50 year history of AI research portends the difficulty in surmounting
secondary obstacles unearthed by each new AI technique's innovations. The
limits of deep learning are almost certainly no different from past
statistically-driven methods in reaching comparable limits, especially those
obstacles that show no signs of vulnerability to the slings and arrows
inherent in rich semantics. The recent article by Doug Hofstadter in The
Atlantic on using DNNs to do machine translation highlights this nicely.

To date, no work in DNNs has shown any potential for redressing problems of
the common sense knowledge needed to know when your translated sentence is
semantic nonsense. Likewise, no statistic-based learning method has shown
itself capable of tapping into much less creating the deep knowledgebases and
wealth of relations that give bare facts the semantic meaning needed to convey
or understand subtleties in messages that are more than trivial.

Deft mimicry may win a Turing Test, but complex syntheses of thought (like
writing a fictional story in which characters have internal thoughts and
hidden motives that depend on relationships, subtexts, and dependencies) — no
form of AI has yet shown any potential to solve such problems. Too much
internal state and complex relations must be modeled, much less, the ability
to extend and translate these to many possible worlds — as humans do every
day.

No, statistics-based AI techniques (like deep nets) show little promise to
grow indefinitely unto full (wo)manhood.

~~~
skybrian
I'm just an outside observer, but I wouldn't say "no potential". For example,
I'm reminded of a paper [1] about a technique that allows translation between
language pairs that weren't in the training data.

This doesn't rise to the level of common sense yet, but it seems impressive,
and there is a language-independent portion of the neural net that seems to be
encoding _something_.

[1] [https://research.googleblog.com/2016/11/zero-shot-
translatio...](https://research.googleblog.com/2016/11/zero-shot-translation-
with-googles.html)

------
zby
If it is shallow than maybe it should not be called deep? :)

~~~
giardini
In any case Hinton's use of the word "deep" was an excellent one from a
marketing perspective:

 _" In 2006, a publication by Geoff Hinton, Osindero and Teh[26][27] showed
how a many-layered feedforward neural network could be effectively pre-trained
one layer at a time, treating each layer in turn as an unsupervised restricted
Boltzmann machine, then fine-tuning it using supervised backpropagation.[28]
The paper referred to learning for deep belief nets."_

Perhaps this usage ( _" deep belief networks"_) was _too_ effective, as so
many marketing terms prove to be. Hinton is likely well-aware of the
importance of terminology that captures the imagination of the public (and the
marketplace for research).

------
marmaduke
Those four words convey potential problems in any model or method, which an
engineer would want to address.

------
tanilama
Nothing new to see here. Just a recycle of Gary's Marcus's recent criticism of
deep learning.

