
AI, Deep Learning, and Machine Learning: A Primer [video] - jonbaer
http://a16z.com/2016/06/10/ai-deep-learning-machines/
======
aficionado
Basically this video ignores the history of machine learning in general.
Jumping from Expert Systems to Neural Networks and Deep Learning is actually
ignoring 36 years (and billions of dollars) of research
[http://machinelearning.org/icml.html](http://machinelearning.org/icml.html)
(Breiman, Quinlan, Mitchell, Dietterich, Domingos, etc). Calling 2012 the
seminal moment of Deep Learning is quite hard to digest. Maybe it means that
2012 is the point in time when the VC community discovered machine learning?
Even harder to digest is calling Deep Learning the most productive and
accurate machine learning system. What about more business oriented domains
(without unstructured inputs), the extreme difficulties and expertise required
to fine tune a network for a specific problem, or some drawbacks like the ones
explained by
[http://arxiv.org/pdf/1412.1897v2.pdf](http://arxiv.org/pdf/1412.1897v2.pdf)
or
[http://cs.nyu.edu/~zaremba/docs/understanding.pdf](http://cs.nyu.edu/~zaremba/docs/understanding.pdf).

Those who ignore history are doomed to repeat it. As Roger Schank pointed out
recently [http://www.rogerschank.com/fraudulent-claims-made-by-IBM-
abo...](http://www.rogerschank.com/fraudulent-claims-made-by-IBM-about-Watson-
and-AI), another AI winter is coming soon! Funny that the video details the
three first AI winters but the author doesn't realize that this excessive
enthusiasm in one particular technique is contributing to a new one!

~~~
deprave
I think you're spot on with the observation that 2012 refers to when VCs
discovered machine learning. Anyone who has recently interacted with VCs will
tell you that they look for anything to do with machine learning (and
VR/AR/MR), even when it makes no sense. There are going to be some companies
who will be able to leverage machine learning to advance their business,
namely, Google/Facebook who will probably claim they can offer better targeted
advertising and such. Most other players who merely try to force machine
learning on other fields are likely to realize that while the technology is
cool, it's still too early for it to be generally applicable to "any" problem.

Especially dangerous is going to be the mix of machine learning with
healthcare. I believe Theranos tried it and found out it's not that easy...
I'd watch this space with skepticism.

~~~
gumby
> Especially dangerous is going to be the mix of machine learning with
> healthcare.

Medical diagnosis has been one of the primary application areas of AI since
the 70s (maybe earlier, I can't remember off the top of my head). The
widespread non-availability of automatic doctors should tell you how well that
has worked :-(.

Coincidentally enough I worked both in AI (1980s) and drug development (2000s)
and now _really_ understand how hard it is!

I do believe we will soon see automated radiology analysis as it is likely to
_appear to be_ most amenable to automated analysis. Presumably in Asia first
as the US FDA will justifiably require a lot of validation. The opportunity
for silent defect is quite high -- you are right to say "especially dangerous"

------
cs702
This is an opinionated video that tries to rewrite history. For example,
according to it, the "big breakthrough" with deep learning occurred in 2012
when Andrew Ng et al got an autoencoder to learn to categorize objects in
unlabeled images. WHAT? Many other researchers were doing similar work years
earlier. According to whom was this the "big breakthrough?"

The video at least mentions Yann LeCun's early work with convnets, but there's
no mention of Hinton et al's work with RBMs and DBNs in 2006, or Bengio et
al's work with autoencoders in 2006/2007, or Schmidhuber et al's invention of
LSTM cells in 1997... I could keep going. The list of people whose work is
insulted by omission is HUGE.

I stopped watching the video at that point.

~~~
pinouchon
Omitting the other members of the LBH conspiracy and Schmidhuber is
unfortunate, but I agree with the idea that the number one reason deep
learning is working now is scale. Hinton also says it himself, for example (at
5m45 in the video below): "What was wrong in the 80 was that we didn't have
enough data and we didn't have enough compute power. [...] Those were the main
things that were wrong".

He gives a "brief history of backpropagation" here:
[https://youtu.be/l2dVjADTEDU?t=4m35s](https://youtu.be/l2dVjADTEDU?t=4m35s)

~~~
lars
I agree that scale is an important factor in deep learning's success, but that
Google experiment ended up being a good example of how not to do it. They used
16000 CPU cores to get that cat detector. A short while later, a group at
Baidu were able to replicate the same network with only 3 computers with 4
GPUs each. (The latter group was also lead by Andrew Ng.)

~~~
espadrine
Incidentally, seeing the speaker set up an overkill neural network for a
trivial classification problem seemed off to me. Unsurprisingly, at least 75%
of the neurons were unused.

Throwing a phenomenal amount of neurons at a problem is not a goal; using a
minimal amount to solve it in a given time budget is.

The statement at the end of the video, “ _all the serious applications from
here on out need to have deep learning and AI inside_ ”, seems awfully
misguided. Even DeepMind doesn't use deep learning for everything.

------
KasianFranks
AI is very fragmented. Biomimicry has always been the way forward in every
industry and Stephen Pinker made good head way from my vantage.

[https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&e...](https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&espv=2&ie=UTF-8#q=stephen%20pinker%20how%20the%20mind%20works)

[https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&e...](https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&espv=2&ie=UTF-8#q=computational%20theory%20of%20the%20mind)

Saira Mian, Micheal I. Jordan (Andrew Ng was a pupil of his) and David Blei
were not mentioned in this video so they are off the mark a bit. Vector space
is the place.

[https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&e...](https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&espv=2&ie=UTF-8#q=michael+jordan+deep+learning)

AI has become the most competitive academic and industry sector I've seen.
Firms like Andreessen are trying to understand the impact during this AI
summer and they should be applauded for this.

One of the keys to AI is found here:
[https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&e...](https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&espv=2&ie=UTF-8#q=split%20brain)

Deep learning has very little to do with how the brain and mind work together.
In the video the highlight on ensemble (combinatorial) techniques are a big
part of the solution.

~~~
gavanwoolery
More direct link to Split-brain as it applies to computing:
[http://en.wikipedia.org/wiki/Split-
brain_(computing)](http://en.wikipedia.org/wiki/Split-brain_\(computing\))

Also, thanks for that term! Was not aware of it, but very useful to describe
what I think is one of the big problems.

~~~
KasianFranks
Yes, split brain approaches in general computing are very interesting and I
think, over lap some approaches in ai-based computational combined with
neuroscientific efforts.

------
gavanwoolery
Interesting to see the amount of "winters" AI has gone through (analogous, to
a lesser extent, to VR).

I see increasing compute power, an increased learning set (the internet, etc),
and increasingly refined algorithms all pouring into making the stuff we had
decades ago more accurate and faster. But we still have nothing at all like
human intelligence. We can solve little sub-problems pretty well though.

I theorize that we are solving problems slightly the wrong way. For example,
we often focus on totally abstract input like a set of pixels, but in reality
our brains have a more gestalt / semantic approach that handles higher-level
concepts rather than series of very small inputs (although we do preprocess
those inputs, i.e. rays of light, to produce higher level concepts). In other
words, we try to map input to output at too granular of a level.

I wonder though if there will be a radical rethinking of AI algorithms at some
point? I tend to always be of the view that "X is a solved problem / no room
for improvement in X" is BS, no matter how many people have refined a field
over any period of time. That might be "naive" with regards to AI, but history
has often shown that impossible is not a fact, just a challenge. :)

~~~
RockyMcNuts
1) sounds like exactly what deep learning is...map more complex abstractions
in each succeeding layer

2) are computers that can understand speech, recognize faces, drive cars, beat
humans at Jeopardy really 'nothing at all like human intelligence?'

~~~
throwawaysocks
_> 1) sounds like exactly what deep learning is...map more complex
abstractions in each succeeding layer_

Only at such a high level of abstraction as to be meaningless.

 _> 2) are computers that can understand speech, recognize faces, drive cars,
beat humans at Jeopardy really 'nothing at all like human intelligence?'_

They are not. Hundreds of man years worth of engineering time go into each of
those systems, and none of those systems generalizes to anything other than
the task it was created for. That's _nothing_ like human intelligence.

~~~
armitron
This is a very superficial point-of-view.

What matters here is the concept itself (deep learning as a generic technique)
but also __scalability __. Not the specifics that we have today, but the
specifics that we will have 20 years from now.

The concept is proven, all that matters now is time...

~~~
daveguy
> The concept is proven, all that matters now is time...

This is a very naive point of view. You could deep-learn with a billion times
more processing power and a billion times more data for 20 years and it would
not produce a general artificial intelligence. Deep learning is a set of
neural network tweaks that is insufficient to produce AGI. Within 20 years we
may have enough additional tweaks to make an AGI, but I doubt that algorithm
will look anything like the deep learning we have today.

~~~
throwawaysocks
This is basically exactly what I was trying to say with my original comment;
thanks for stating it in a clearer way.

------
justsaysmthng
From the presentation: "There's a lot of talk about how AI is going to totally
replace humans... (But) I like to think that AI is going to actually make
humans better at what they do ..."

Then immediately he continues that

"So it turns out that using deep learning techniques we've already gotten to
better than human performance [....] at these highly complex tasks that used
to take highly, highly trained individuals... These are perfect examples of
how deep learning can get to better than human performance... 'cause they're
just taking data and they're making categories.."

I think that brushing off the dramatic social changes that this technology
will catalyze is irresponsible.

One application developed by one startup in California (or wherever) could
make tens of millions of people redundant all over the world overnight.

How will deep learning apps affect the healthcare systems all over the world?
What about IT, design, music, financial, transportation, postal services...
nearly every field will be affected by it.

Who should the affected people turn to ? Their respective states ? The
politicians ? Take up arms and make a revolution ?

My point is that technologists should be ready to answer these questions.

We can't just outsource these problems to other layers of society - after all,
they're one step behind the innovation and the consequence of technology is
only visible after it's already deeply rooted in our daily habits.

We should become more involved in the political process all over the world (!)
- _at least_ with some practical advice to how the lawmakers should adapt the
laws of their countries to avoid economic or social disturbances due to the
introduction of a certain global AI system.

~~~
vonklaus
I just listened to Altman discussing this at a recent interview and it is
obviously something he thought a lot about. If you can't watch the video he
says that ai is an amazing resource and YC is funding a basic income study in
oakland as well as open ai. He also points out that doing a task that a
machine or computer do is pointless. I find myself agreeing with what I
believe his conclusion was;

we (humans) want ai and will all be better off for it. we are heading into a
shift on the order of the industrial revolution and the best course is
unknowable. we should harness this technology, try to distribute it bu
collaborative building & information dissemination and study the idea of
personal fullfillment & a basic quality of life.

In short, the answer is definitely not to stop working on it because it is
just basic game theory that someone/entity will not. We need to leverage to
fix the problem it creates & learn how to allocate our resources by studying
this immeadiately.

[http://m.youtube.com/watch?v=FuijDaj8DvA](http://m.youtube.com/watch?v=FuijDaj8DvA)

~~~
justsaysmthng
> He also points out that doing a task that a machine or computer do is
> pointless.

This is true for some tasks - but not true for others, even if AI could in
theory do them better.

For example, a medical doctor. It is easy to see this as just a job, but it's
so much more than that - it's the culmination of 20 years of studies, a
childhood dream, the feeling of being important and useful to society, the
"thank you" from the patient, the social circle, the social status...

There are many people who actually enjoy their jobs, because it gives them
meaning and satisfaction.

Another example - musicians. I see so many "AI composer" projects out there -
algorithms which compose music... I think these people are kind of missing the
point ..

It's easy to see music as notes and tempo, but it is much more than that. It
is a medium, a tool, through which the listener can connect to the artist and
experience his emotional state:
[https://www.youtube.com/watch?v=1kPXw6YaCEY](https://www.youtube.com/watch?v=1kPXw6YaCEY)

Having an algorithm on the other side feels so fake.. artificial..

~~~
armitron
Only because your mental map is old, archaic and will not allow you to see
things any other way.

So what is the right course according to you? Do we set an artificial
threshold at some point in the future, where these jobs will be swapped to AI?
Do we simply _never_ replace them with AI?

Do we __slow __progress in order to cater to people 's obsolete expectations
(don't want to hurt their __feelings __after all).

No. Problems will appear and we will solve them. It is our duty to the
universe, to ride the wave and see it all the way through.

EDIT: Virtual Reality will solve a lot of the "meaning of life/emotional
satisfaction" issues that creep up. And VR is getting a jump .... right about
now. It's quite amusing, just thinking about the timing.

~~~
justsaysmthng
> Only because your mental map is old, archaic and will not allow you to see
> things any other way.

I've done my fair share of acid and mushroom trips, explored the DMT
hyperspace dozens of times, travelled through time and space into alternate
realities, met and communicated with biological and machine beings and have
thoroughly explored our planet.

My mind is wide open :)

Since you mentioned the Universe, then it would help to remember that in
"galactic" terms, we've just climbed down from the trees a couple of hundred
thousand years ago and our technology is still extremely primitive and it is
not at all a proven fact that we can survive it.

More down to Earth, what I suggested in the parent post is that people who are
bringing this technology forward should also be the people who come up with
the political and social proposals for changes necessary to accommodate it.

We can't just unleash these "technological atomic bombs", which fundamentally
change the social game and then expect the politicians to handle the fallout.
Tech people and scientists need not stay on the sidelines any more - they must
be the designers not only of technology, but of the social system too, since
the two are merging anyway...

Of course this can't be "enforced", rather, it's an ethical thing to do on the
part of tech companies, justified by the fact that deep down, our motivation
is to make the world "better".

~~~
armitron
Yes and this will without a doubt take place too. In fact, some would say it's
already happening.

My point is that the process, if we might call it that, will be osmotic and
not discrete. Of course there is also the notion of people believing that
there is a layer of central planning somewhere and that "we" (groups,
organizations, nation states) actually have high-level organizational control
over what is happening.. Watching Taleb talk about tail events will quickly
put these notions to the ground.

We shouldn't waste time inventing such control when there is none or even
inventing the illusion that such control will be effective. All we can do, I
feel, is _guide_.

------
autokad
We will be closer to cracking neural nets and are closer to the singularity
when we can train a net on two completely different tasks and each task can
make other predictions subsequently better. IE: train / test it on spam / ~
spam emails, then train the same net with twitter data male / female.

~~~
nl
You can do this now(?!)

In fact something as simple as naive Bayes will work reasonably well for that.

I'm not sure if you are aware, but in (say) image classification it's pretty
common to take a pre-trained net, lock the values except for the last layer,
and then retrain that for a new classification task. You can even drop the
last layer entirely and use a SVM and get entirely adequate results in many
domains.

Here's an example using VGG and Keras: [http://blog.keras.io/building-
powerful-image-classification-...](http://blog.keras.io/building-powerful-
image-classification-models-using-very-little-data.html)

And here a similar thing for Inception and TensorFlow:
[https://www.tensorflow.org/versions/r0.8/how_tos/image_retra...](https://www.tensorflow.org/versions/r0.8/how_tos/image_retraining/index.html)

~~~
dharma1
Transfer learning at the moment works within one domain (say images), because
the low level shapes are still similar, but not between different domains of
data

~~~
nl
Sure, that's what I was pointing out.

However: Zero-Shot Learning Through Cross-Modal Transfer
[http://arxiv.org/abs/1301.3666](http://arxiv.org/abs/1301.3666)

------
31reasons
One of the main challenges in Deep Learning is that it requires massive
amounts of data, orders of magnitude more data than a human toddler to detect
a cat. It could be a great area of research on how to reduce the amount of
data it takes to train the network.

One main thing it lacks is imagination. Humans can learn things and can
imagine different combinations of those things. For example, if I ask you to
imagine a Guitar playing Dolphin, you could imagine it and even recognize it
from a cartoon even though you have never seen it in your life before. Not so
for Deep Learning, unless you provide massive amount of images of Dolphins
playing guitars.

~~~
pawelwentpawel
Interesting! Made me wonder - how would we compare the data that neural net
receives with the data that toddler's visual cortex is getting?

On a purely visual level - let's say we have 10k of static 32x32 images
defining a class of cat. Or even more of them plus some negative examples.
Each image is a different cat, in a different position (they're incredibly
flexible creatures). Having so many cases we should be trying to make some
kind of a generalization of what 1024 pixels of a cat should look like.

A family with a toddler has only one cat. From that one example, he learns the
concept of what a cat is and is able to generalize it when in different
situations. But a toddler has 2 eyes, his visual input is stereo. Even if he
sees just one cat, it's not a _static_ image interaction. The input is
temporal, he can see the cat moving and interacting with the environment.

~~~
dharma1
I think stereo vision isn't super important for this - people blind in one eye
don't do noticeably worse. Not to say depth doesn't help for segmentation or
learning concepts but I don't think it's the key

------
epberry
Pretty basic stuff - the history portion was more interesting than any of the
content that followed. Anyone who's been paying the slightest attention in the
last few years will be familiar with all of the examples used in the podcast.

On a side note I always admire the polish of the content that comes out of
a16z - its typically very well put together.

------
vj_2016
[http://www.videojots.com/davos/state_of_ai.html#2181](http://www.videojots.com/davos/state_of_ai.html#2181)

Apparently, robots still struggle to pick up things?

