
The Difference Between AI, Machine Learning, and Deep Learning - kercker
https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
======
ktta
Also, an interesting read related to this would 'the AI effect'[0]. A lot of
the stuff Deep Learning/Machine Learning is able to do today would be looked
at as something that only 'true' AI (whatever consensus on what that means is;
I think of it as AGI) would be able to do.

But as soon as we are able to solve a problem that we think (feel?) only true
AI (AGI) would be able to solve, as soon as we know _how_ it was solved, it is
no longer a mystery that warrants amazement and we argue that it is not _real_
intelligence, just like the link below states.

But just a decade ago, if you saw that a computer was able to recognize
pictures _better_ than humans, you would think there was something fishy going
on, or we have achieved true AI.

My opinion is true AI is which is comparable to human intelligence, in that it
is sentient and/or capable of abstract thought, not necessarily being able to
hold a conversation or solve concrete mathematics problems, or dump out a
story by neural networks.

[0][https://en.wikipedia.org/wiki/AI_effect](https://en.wikipedia.org/wiki/AI_effect)

~~~
argonaut
I hear this repeated a lot. But I've never actually seen a source from, say,
the 90's or the 2000s, actually saying something like "if we're able to
classify images better than humans, that's true AI."

I suspect it's just hearsay that's been repeated over and over again.

~~~
reacweb
In the 90's, John McCarthy was calling Go "the new Drosophila of AI". Now, we
have AlphaGo that is "just" a bunch of "generic" deep learning with MC. Maybe
John McCarthy was right and the ideas behind AlphaGo are the first steps
toward true AI. I am very impressed.

~~~
visarga
The guys at NVIDIA did something very cool recently. They took a simple CNN
(convolutional neural net) and in 2-3 days they taught it to drive a car.

Article: How Our Deep Learning Tech Taught a Car to Drive -
[https://blogs.nvidia.com/blog/2016/05/06/self-driving-
cars-3...](https://blogs.nvidia.com/blog/2016/05/06/self-driving-cars-3/)

These neural nets are really smart, and we don't know exactly how they work,
we know only in principle. But if we asked the guy who made the self driving
car what is the role of the 17th neuron in the 14th layer, he would probably
have no idea. Just like human brains evolve through learning, so do neural
nets. Yet many people think they are just clever tricks and not truly
intelligent.

~~~
c0g
That's because they're not intelligent. They're layered function approximators
with some specialisations for vision - conv layers. Their structure is
designed by humans as is the data they are fed with. Maybe the human brain is
the same, IMO it isn't. Humans seek out new experiences and can introspect and
at their best assess their level of knowledge of a sibject. DNNs cannot.

------
oliwaw
Deep learning is just a rebranding of "neural networks". When neural nets
became unpopular in the 90s and early 2000s, people talked about "multilayer
networks" (dropping the "neural") since it wasn't really useful to think about
this approach from the neuro perspective (since it's such a cartoonish model
of real neural networks anyway).

Now that very deep networks have become possible, and various graphical models
and Bayesian approaches have also been folded under "deep learning" (for
example, using back-propagation to learn complicated posterior distributions
in variational Bayes) deep learning is not just about vanilla feedforward
nets.

~~~
denzil_correa
Basically, Deep Learning is now back on the map due to two things - (1) Large
amounts of data available and (2) Faster processing power viz. GPUs.

~~~
visarga
Well, yes, mostly, but there also have been genuine discoveries in the last 10
years. We can now train deep networks because we learned how to regularize -
before it was impossible because of vanishing gradients. We can have even
1000-layer deep nets, which would have been unthinkable. Also, there are some
interesting approaches to unsupervised learning like GANs and VAEs. We learned
how to embed words and not only words, but multiple classes of things into
vectors. Another one would be the progress in reinforcement learning, with the
stunning success of AlphaGo and playing over 50 Atari games. Current crop of
neural nets do Bayesian statistics, not just classification. We are playing
with attention and memory mechanisms in order to achieve much more powerful
results.

~~~
eli_gottlieb
>We can now train deep networks because we learned how to regularize - before
it was impossible because of vanishing gradients.

Those are two different things. Vanishing gradient problems were _ameliorated_
by switching from sigmoidal activation functions to rectified linear units or
tanh activations, and also by dramatically reducing the amount of edges
through which gradients propagate. The latter was accomplished through massive
regularization to reduce the size of the parameter spaces: convolutional
layers and dropout.

Stuff like residuals are still being invented in order to further banish
gradient instability.

------
thallukrish
If I need to show 1 million cat images to train a neural net to see cats, I
wonder how a human brain can figure out cats of any kind just by seeing one or
two.

Is there something fundamental we are missing in going about building these
deep learning stuff ?

~~~
ansible
This difference is due to deep learning not being able to build models.

Check out the recent paper "Building Machines That Learn and Think Like
People":

[https://arxiv.org/abs/1604.00289](https://arxiv.org/abs/1604.00289)

Humans can see one or a few examples of a novel object, such as a cat, and
create a fully 3D mental model of it. So we know what it will look like in
different orientations and lighting conditions.

~~~
TheOtherHobbes
I don't think we have a fully 3D mental model. I think we have a set of core
invariants that are constant across a wide variety of conditions.

I.e. we can recognise static 2D photos of cats, 2D movies of cats, 3D movies
of cats, and real cats in the real world.

The invariants are probably relatively simple - a set of head geometries and
head feature shapes/distribution, with some secondary colour and texture
confirmation.

What's interesting is that we can recognise modifiers to the invariants - e.g.
a shaved cat is still recognisably a cat, but parsed as "cat without fur."

We can also recognise invariants when they're pared down to essentials in
cartoons and sketches.

[https://www.youtube.com/watch?v=R9qdyXCVNVk](https://www.youtube.com/watch?v=R9qdyXCVNVk)

A lot of learning is really just data compression - finding a minimal set of
low-resource invariant specifics from a wide range of noisy high-resource
inputs.

------
harlowja
Has anyone figured out how to debug and/or analyze deep ANN?

That's the problem I always had, you may get them into a trained state, but
good luck figuring out any reason 'why' they ended up in that state (or even
what that state really is).

~~~
dave_sullivan
> That's the problem I always had, you may get them into a trained state, but
> good luck figuring out any reason 'why' they ended up in that state (or even
> what that state really is).

Can you give a specific example of what you mean? I ask because I see this
sentiment often, but primarily from people who are very new to deep learning.

You can definitely debug a neural network. You mostly want to look at metrics
from training such as gradient norms or try adjusting parameters to see if you
can get a gain in overall performance as measured by cross validation
performance.

You can definitely analyze a neural network. You do so by forming hypotheses,
preparing datasets that reflect those hypotheses, and running them through
your trained models, noting cross validation performance. It's also possible
to visualize the weights in various ways, there are many papers written about
it.

So what do you mean exactly when you say no one has figured out how to debug
or analyze DNNs?

~~~
maaaats
You cannot say: "This image was classified as a stop sign, because this part
recognized the shape and this part recognized the color, and this part the
text", which you could do with other approaches.

When it doesn't discover that it's a stop sign, how do you debug it? Did it
recognize the shape.. who knows?

~~~
vidarh
> When it doesn't discover that it's a stop sign, how do you debug it? Did it
> recognize the shape.. who knows?

Barring other analytic tools (like looking at which parts contribute the most
to the wrong result), the same way you test other things when you have a
(somewhat) black box:

Form hypotheses and test them.

Maybe it didn't recognise the shape, so try adjusting the image to clean it
up, and once you have one it recognises, try to reduce and alter the
difference between them. Maybe it turns out the image e.g. has the stop sign
slightly covered, making the shape look wrong, and there's nothing in the
training set like that.

Maybe the hue or brightness is off and the training set is mostly all lit a
certain way. Test it by adjusting hue and brightness of the test image and see
if it gets recognised.

And so on.

There are plenty of other areas where we are similarly constrained from taking
apart that which we're observing, so it's not like this isn't something
scientists are dealing with all the time.

Within comp.sci. we're just spoiled in that so much of what we do can be
easily instrumented, isolated and tested in ways that often lets us determine
clear, specific root causes through analysis.

------
paulsutter
I disagree. Machine learning is more general than AI and therefore should be
the outer circle (unless you believe the meme "did a regression - called it
AI")

Deep learning is, yes, a subcase of machine learning, and AI may be a circle
within machine learning and enclose deep learning.

But truth be told, we will all regret the way we use the term AI now.
Eventually the term AI will refer only to general intelligence (aka AGI).

~~~
argonaut
"AI" is a specific term used in academia / research. In that context, ML is
most certainly a subset of AI.

The fact that many laymen have a different conception of the term is hardly a
reason to change all our uses of the term.

------
threepipeproblm
I recently learned that Alexander Stepanov, designer of the C++ STL, has
stated that, "I think that object orientedness is almost as much of a hoax as
Artificial Intelligence."

------
bpesquet
Great writeup about how deep learning came to be:
[http://www.andreykurenkov.com/writing/a-brief-history-of-
neu...](http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-
and-deep-learning/) (posted here a few months ago).

------
stefanv
There is also machine intelligence: [http://numenta.com/blog/machine-
intelligence-machine-learnin...](http://numenta.com/blog/machine-intelligence-
machine-learning-deep-learning-artificial-intelligence.html)

------
danvayn
Why the hell is the text not black? It's very awful to read on mobile..

------
ragebol
For machine learning and deep learning based assistive functions, like Google
Now, Cortana and Siri, we let the companies behind them gather a lot of our
personal data into "Big Data", because then you can do all the statistics.

I think a (more) true AI would not need all this data and just be a _personal_
assistant, not needing all the big data of other people too. Maybe initially,
once, and then be a good personal assistant, learning to know you like a human
personal assistant.

It would need some common sense, which is now lacking, mostly.

------
bogomipz
I was confused by this sentence in the last paragraph:

"Deep Learning has enabled many practical applications of machine learning and
by extension the overall field of AI."

Is it not the reverse - machine learning has enabled deep learning?

Can someone comment on how the two - machine learning and Deep learning
relate? Is the relationship sequential i.e a data set from machine learning is
the the input for a neural network? The diagram had the effect of confusing
me.

~~~
qntty
the relationship suggested by the article is that deep learning is a subset of
machine learning

~~~
bogomipz
Sure, that was kind of my point, and as such I would think the sentence should
instead read

"Machine Learning has enabled many practical applications of deep learning .."

No?

------
unabst
There's that one iconic image of neurons suspended in space with bolts of
electricity flashing between them. We are told that's how our brains work.

We are then shown a diagram by a computer scientist. Instead of cells and
thunder, we see circles and arrows. Then we are told there is an algorithm
that simulates what the brain does. Viola, we have our artificial neural
network. Not only do they look similar, they have two words in common, neural
and network!

And so for most of us, there is only one logical conclusion: It does what our
brain does, so once our computers have the power our brains do, we'll have the
singularity!

Of course, now we know this is complete bullshit.

Basically, computer scientists just took the names and those initial
abstractions and ran with it. They never looked back at the biology or how
brains actually work. The result is a ton of great research, but they've
strayed further and further from neuroscience and from humans. Which is
obvious, because they're staring at code and computers all day, not brain
meat. If there is one thing AlphaGo proved it is that we've made a ton of
progress in computation, but that it's a different direction. Just the fact
that average people generally suck at Go should be enough to show that AlphaGo
is not human (in many ways it's beyond human).

In the meantime, our neuroscientist have made progress also, except, they've
done it staring at the actual brain. And now it's to the point where our
brains look nothing like that original image our computer scientists were
inspired with.

Now there is this (Harvard research):
[https://www.youtube.com/watch?v=8YM7-Od9Wr8](https://www.youtube.com/watch?v=8YM7-Od9Wr8)

And this (MIT research):
[https://www.ted.com/talks/sebastian_seung?language=en](https://www.ted.com/talks/sebastian_seung?language=en)

With advancement comes new vocabulary, and the new word this time is
connectome.

Some incredibly smart computer scientists will, again, take the term and all
the diagrams, and start programming based on it. The result will be Artifical
Connectomes, and they will blow our socks off. Now, don't get me wrong. I am
not trying to be sarcastic here. This is what _should_ happen. And with every
iteration, we will get closer to AGI.

It's just that whenever I see articles about machine learning and neural
networks, I can't help but think of that classic artist's rendition of neurons
firing, and how it's basically complete bullshit. Like Bohr's atom, it's an
illustration based on a theory, not reality. Now we have wave function
diagrams and connectomes. But as a physicist would tell you, anyone caught
with a Bohr's atom is stuck in the 20th century.

~~~
yvdriess
Actual neural network papers are far removed from the impressions you get from
popular science articles. Biological neurons were some inspiration for the
abstraction, but the underlying model is more a direct evolution of the
maths/stastical models. Griping that DNNs are not actually like neurons is
akin to saying that Object Oriented programming is nothing like the
organisation of biological cells it was inspired on. A useful abstraction is a
useful abstraction, there is little reason to think that making the
computational models any closer to physical processes is going to bring any
improvements.

Neuromorphic computing is the field that tries to more accurately mimic
spiking neurons, but making something useful out of it takes a backseat. It is
still an open question if it is going to be useful.

~~~
unabst
Exactly. Wikipedia is full of disclaimers about the differences also. The
issue is just with word usage that continues to fuse the two, where is reality
any direct correlation ended ages ago.

------
vonnik
This intro is pretty good, but may be a bit high level for some readers of HN.
It seems like it's written for non-technical readers. The idea is that AI, ML
and DL are nested like Russian dolls, each subsequent one a subset of the
other.

It might be better to explain _why_ deep learning is so effective, in clear
language:

* Deep artificial neural networks are old, relatively simple combinations of math and code that are now able to produce accurate models through brute force because we have

1) vastly more computational power thanks to NVIDIA and distributed run-times;
2) much more data, and much larger labeled datasets thanks to people like Fei-
Fei Li at Stanford; 3) better algorithms thanks to the work of Hinton, LeCun,
Bengio, Ng, Schmidhuber and a raft of others.

 _Deep_ is a technical term. It refers to the number of layers through which
data passes in a neural net; that is, the number of mathematical operations it
is subjected to, and the number of times it is recombined with other inputs.

This recombination of inputs, moving deeper into the net, is the basis of
feature hierarchy, which is another way of saying: we can cluster and classify
data using more complex and abstract representations.

That clustering and classification is at the heart of what deep learning does.
Another way to think about it is as _machine perception_. So the overarching
narrative in AI is that we've moved from the symbolic rules engines of the
chess victors to the interpretation of complex sensory information. For a long
time, people would say AI could beat a 30-year-old at chess but couldn't beat
a 3-year old at basic tasks. That's no longer true. We can go around beating
3-year-olds at name games all day. AI mind, beginner's mind.

But it's important to note that deep learning actually refers to other
algorithms besides artificial neural networks. Deep reinforcement learning is
one example. RL is also an old set of algorithms, which are goal-oriented. RL
helps _agents_ choose the right _action_ in a given _state_ to maximize
_rewards_ from the environment. Basically, they learn the function that
converts actions to rewards given certain conditions, and that function is
non-differentiable; that is, you can't learn it simply by backpropagating
error, the way neural nets do.

Deep RL is important because the most amazing algorithms, like AlphaGo, are
combining deep neural nets (recognize the state of the Go board) with RL (pick
the move most likely to succeed) and other components like Monte Carlo
Decision Trees (limit the state space we explore).

So we're moving beyond perception to algorithms that can make strategic
decisions in increasingly complex environments.

We've written more about this, and implemented many of these algorithms:

[http://deeplearning4j.org/ai-machinelearning-
deeplearning.ht...](http://deeplearning4j.org/ai-machinelearning-
deeplearning.html)
[http://deeplearning4j.org/reinforcementlearning.html](http://deeplearning4j.org/reinforcementlearning.html)
[http://github.com/deeplearning4j/rl4j](http://github.com/deeplearning4j/rl4j)

------
Azuolas
this is a very interesting read. Thank you.

