
We can’t trust AI systems built on deep learning alone - laurex
https://www.technologyreview.com/s/614443/we-cant-trust-ai-systems-built-on-deep-learning-alone/
======
mindgam3
> AlphaGo can play very well on a 19x19 board but actually has to be retrained
> to play on a rectangular board.

This right here is the soft underbelly of the entire “machine learning as step
towards AGI” hype machine, fueled in no small part by DeepMind and its flashy
but misleading demos.

Once a human learns chess, you can give it a 10x10 board and she will perform
at nearly the same skill level with zero retraining.

Give the same challenge to DeepMind’s “superhuman” game-playing machine and it
will be an absolute patzer.

This is an obvious indicator that the state of the art in so-called “machine
learning” doesn’t involve any actual learning in the way it is normally
applied to intelligent systems like humans or animals.

I am continually amazed by the failure of otherwise exceedingly intelligent
tech people to grasp this problem.

~~~
Isinlor
Try learning to ride bike with inverted steering, try to navigate world with
your vision flipped over or use your non-dominant hand to do things that you
normally do. Well, try to write on Azerty keyboard if you are Qwerty native
(really, f __k Azerty :P).

Humans are also not a general intelligence.

In certain sense Deep Reinforcement Learning is actually more general than
human intelligence. For example, when playing games you can remove certain
visual clues. It makes it almost impossible to play for humans, while Deep RL
scores will not even budge. It means that Deep RL is more general, because it
does not relay on certain priors, but it also makes it more stupid in narrow
domain of human expertise. Try this game to see yourself: [https://high-
level-4.herokuapp.com/experiment](https://high-
level-4.herokuapp.com/experiment)

Here is bike with reverse steering:
[https://www.youtube.com/watch?v=MFzDaBzBlL0](https://www.youtube.com/watch?v=MFzDaBzBlL0)
Here is flipped vision experiment:
[https://www.youtube.com/watch?v=MHMvEMy7B9k](https://www.youtube.com/watch?v=MHMvEMy7B9k)

Human brains are amazing, but they also require certain amount of time to
retrain when inputs/outputs are fundamentally changed.

PS. I didn't hear about anyone testing different board sizes with AlphaZero-
esque computer players. But I saw Leela Zero beating very strong humans, when
rules of the game were modified so that that the human player could play 2
additional moves:
[https://www.youtube.com/watch?v=UFOyzU506pY](https://www.youtube.com/watch?v=UFOyzU506pY)

~~~
inimino
This is true too, we are adapted to our environment, in particular the things
that we do automatically (System I).

Playing chess well is a combination of both conscious and unconscious skills.
However when deep learning systems play, it is all the unconscious, automatic
application of statistical rules. They are playing a very different game from
the human chess game.

Because there is no abstract reasoning involved here, these systems cannot
apply the lessons learned from chess to another board game, or to something
completely different in life, which humans can and do. So even though they are
much stronger than human players, they aren't strong in the same way.

------
6gvONxR4sf7o
I'm coming to suspect that even our data isn't enough for useful AI. Imagine
you had a truly general sci-fi AI at your office. It _still_ couldn't just
look at your database and answer a simple question like "What was the
difference in client churn rates between mobile and desktop last month?" or
"What was the effect of experiment 1234 on per-client revenue?" Hell, a
_human_ couldn't do it. As far as the human or AI would know, you just
presented it with a bunch of random tables. This matters because it's
incredibly helpful to know which pieces are randomized. Which rows are
repeated measurements as opposed to independent measurements. Which pieces are
upstream of which others. There's so much domain knowledge baked into data,
while we just expect an algorithm to learn from a simple table of floats.

The human state of the art solution seems to be going on slack and asking
questions about the data provenance, which will decidedly not work for an
automated approach.

A primary reason I can do better a better job than a generic algorithm is
because you told me where the data came from (or I designed the schema and ETL
myself), while the algo can't make any useful assumptions because all that
info is hidden.

~~~
allworknoplay
This is a fantastic point that may not be relevant now, but may be
extraordinarily relevant in the future.

I’m not aware of any work or even sci-fi that addresses AGI with regards to
this question, and would be curious if there’s stuff out there?

~~~
6gvONxR4sf7o
I'm beginning to suspect that it could be relevant soon. If you wait for such
advanced AI that can understand your documentation, you're making it harder
than it has to be, kinda like Marcus is saying in OP. You _could_ solve this
with just tons of raw data, but that seems unnecessarily hard. For a firm with
the usual small dataset, maybe even unrealistically hard.

Anyways, with some naive googling I found these references which seem
interesting with regards to lineage and causality (for the query "lineage
causal database schema"):

[0] Causality and Explanations in Databases. The "Related topics in Databases"
section seems interesting.

[1] Duke's 'Understanding Data: Theory and Applications, Lecture 16: Causality
in Databases' (by one of [0]'s authors).

[2] Quantifying Causal Effects on Query Answering in Databases. This has some
interesting definitions.

[3] Causality in Databases. Seems like a more in depth version of [0].

[4] A whole course on "Provenance and lineage"

[5] Causality and the Semantics of Provenance. Defines "provenance graphs" and
some properties.

\-----------

[0]
[http://www.vldb.org/pvldb/vol7/p1715-meliou.pdf](http://www.vldb.org/pvldb/vol7/p1715-meliou.pdf)

[1]
[https://www2.cs.duke.edu/courses/fall15/compsci590.6/Lecture...](https://www2.cs.duke.edu/courses/fall15/compsci590.6/Lectures/Lecture-16.pdf)

[2]
[https://www.usenix.org/sites/default/files/conference/protec...](https://www.usenix.org/sites/default/files/conference/protected-
files/1-salimi.pdf)

[3]
[https://www.cs.cornell.edu/home/halpern/papers/DE_Bulletin20...](https://www.cs.cornell.edu/home/halpern/papers/DE_Bulletin2010.pdf)

[4]
[https://cse.buffalo.edu/~chomicki/cse703-s18.html](https://cse.buffalo.edu/~chomicki/cse703-s18.html)

[5] [https://arxiv.org/pdf/1006.1429.pdf](https://arxiv.org/pdf/1006.1429.pdf)

------
thelazydogsback
I've been waiting for the Symbolic/NN pendulum to starting swinging back the
other way and start settling in the center. NN/DL is great for the interface
between the outer world and the inner world of the mind (pattern recognition
and re-construction), and symbolic AI more straightforwardly represents more
"language of the mind" tasks, and easily handles issues like explanation and
other meta-behaviors that with DL is difficult due to its black-box nature.
DL's reliance on extension/training vs. intention/rules can develop ad-hoc
intentional emergent theories which is their strength but also their weakness
as these theories may not be correct or complete. Each can be brittle in their
own way - so it'll be interesting to see more cross-pollination.

~~~
uoaei
SAT solvers are really fast now. Some sort of "neural SAT problem definition"
followed by solving it seems to be an interesting direction, but I'm
relatively naive on it all. Not sure how training would work since there's no
backprop through Boolean logic.

~~~
w_t_payne
Differentiable SAT solvers is a thing.

~~~
ssivark
Pointers please; not just off-hand mentions leaving others to scratch their
heads about what you mean! :-)

------
mark_l_watson
I started reading Rebooting AI last night. I think that Marcus and Davis (so
far in the book) take a reasonable approach by wanting to design robust AI.
Robust AI requires general real world intelligence that is not provided by
deep learning.

I have earned over 90% of my income over the last five or six years as a deep
learning practitioner. I am a fan of DL based on great results for perception
tasks as well as solid NLP results like using BERT like models for things like
anaphora resolution.

But, I am in agreement with Marcus and Davis that our long term research
priorities are wrong.

~~~
m12k
As much I'm hoping there'll be a breakthrough in AGI, maybe the right approach
is the one AlphaGo was using: DL not as the top level decision-making, but
plugged into a traditional decision-making algorithm in specific places.

~~~
tachyonbeam
> As much I'm hoping there'll be a breakthrough in AGI

I think it probably won't be one breakthrough, but several, over decades.
Personally, I'm pretty happy that AGI is taking a long time to materialize. We
likely won't see a "fast takeoff scenario" (the computer is learning at a
geometric rate !!1). It will likely happen gradually over years (progressively
more intelligent, more aware computer systems), and we may have a chance to
adapt in response.

~~~
benogorek
A business professor told me that cars were entirely incremental innovation
all the way from the model T. Just little improvements, one at a time. I don't
know if that's true, but I wonder if it will be an apt analogy for AGI - one
feature at a time, and older attempts at it just look outdated.

~~~
maxerickson
Things like automatic transmissions and fuel injection aren't particularly
incremental.

~~~
benogorek
As someone who once got stuck in an intersection after flooding a '79 Monte
Carlo's carburetor while deciding to go on red, I'm with you on fuel
injection. But I could see the counter-argument that something that makes an
experience nicer is not radical innovation. That old junker got me where I
need to go for a while.

------
cs702
The opposing argument, by Rich Sutton, Distinguished Research Scientist at
DeepMind:

[http://www.incompleteideas.net/IncIdeas/BitterLesson.html](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)

~~~
Barrin92
there are some surprisingly weak arguments in the text. It's correct to not
treat computational resources as constant, but ot treat them as unimportant or
negligible is awful as well.

Already computational resources are becoming prohibitive with only a few
institutions producing state of the art models at high financial cost. If the
goal is AGI this might get exponentially worse. Intelligence needs to take
resource consumption into account. The models we produce aren't even close to
high level reasoning and we're already consuming significantly more energy
than humans or animals, something is wrong.

The scale argument isn't great either because deep learning is running into
the inverse issue of classical AI. Now instead of having to program all logic
explicitly we have to formulate every individual problem as training data.
This doesn't scale either. If an AI gets attacked by a wild animal the
solution can't be to first produce 10k pictures of mauled victims,
intelligence includes to reason about things in the abscence of data. We can't
have autonomous cars constantly running into things until we provide huge
amounts of data for every problem, this does not scale either.

~~~
roenxi
> Already computational resources are becoming prohibitive with only a few
> institutions producing state of the art models at high financial cost.

That is something of an illusion.

Obviously there will be some sort of uneven distribution of computing power;
some institutions will have more, some less. The institutions with more power
will create models at the limit of what they can do, because that is the best
use of their power.

So if the thesis of more power = more results holds then truly cutting results
will always be by people with resources that are practically unattainable by
everyone else. Google's AlphaGo wasn't a particularly clever model, for
example. It just had a lot of horsepower behind it to train it and the various
ranging shot attempts Deepmind would have gone through. Someone else would
have figured it out albeit more slowly in a few years as computing power
became available.

Computational power is still getting exponentially more affordable [0]. Costs
aren't really rising, so much as the people who have spent more money get a
few years ahead of everyone else and can preview what is about to become
cheap.

[0] [https://aiimpacts.org/recent-trend-in-the-cost-of-
computing/](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/)

~~~
lonelappde
That article says that doubling flops/$ used to take 1 year and now takes 3.
It's open question whether that time gap will confirm to grow. The recent
phase shift was due to approaching an asymptote in single core performance,
and switching to optimizing multicore overhead. The low hanging fruit there
will be consumed as well.

------
Mathnerd314
> _General AI also ought to be able to work just as comfortably reasoning
> about politics as reasoning about medicine. It’s the analogue of what people
> have; any reasonably bright person can do many, many different things._

The average human has extreme difficulty reasoning about politics, while
usually being reasonable on medicine (anti-vax being one of many exceptions).
And it seems strange to expect a skilled pianist to also be a skilled
neuroscientist or a skilled construction worker. On the other hand these
people all use similar neural architectures (brains). So he seems pretty off-
track when he criticizes "narrow AI" in favor of "general AI", as if there's
some magic AI that will do everything perfectly, and even more off track when
he criticizes researchers for using "one-size-fits-all" technologies, when
indeed that is exactly what humans have been doing for millennia for their
cognitive needs.

And sure, ML models in publications so far are typically one-off things that
react poorly to modified inputs or unexpected situations. But it's not clear
this has any relevance to commercial use. Tesla is still selling self-driving
cars despite the accidents.

~~~
inimino
> some magic AI that will do everything perfectly

Total straw man. He actually uses an intern as an example in the very next
sentence after what you quoted, as you would expect them to be able to read
and get up to speed on a new area regardless of what it was. Meanwhile SOTA in
NLP is a system that can be built to answer a single kind of question but
can't explain why it did so or do anything useful if given an explanation of
why its answer was wrong.

~~~
Mathnerd314
There are deep models like BERT that do pre-training and then need minimal
training to do multiple tasks such as question answering, entailment,
sentiment analysis, etc. I don't know about "explaining" an answer but there
are debuggers that find errors in data sets:
[https://arxiv.org/pdf/1603.07292.pdf](https://arxiv.org/pdf/1603.07292.pdf).

But as I said, I don't see why an artist would suddenly get up to speed as a
construction worker. He seems to overestimate the capacity of interns as well.

~~~
inimino
Deeply familiar with BERT. It lacks the very ability he is describing, to
adapt itself, get up to speed, and collect relevant information in a new
field, because that's simply not how it works. It can't possibly explain
itself because it lacks any mechanism of introspection that could possibly
give it that ability. It's an expensive way of gaining a very accurate
language model that can be tweaked and get good results on a lot of tasks, but
it doesn't understand what it's doing. It can't argue for its position or
explain why it thinks whatever it thinks. It's not operating on that level of
reasoning, at all.

An artist understands the goals of construction work, and can pick up the
skills necessary along the way, because we can understand a goal and have a
wide variety of cognitive tools to let us know how we are doing. If you've
worked closely with BERT you already know that interns have nothing to worry
about, not just from the current crop of tools that includes BERT, but from
the entire line of deep learning research, short of a sudden and dramatic
shift in direction.

------
sebastianconcpt
_In cognitive science we talk about having cognitive models of things. So I’m
sitting in a hotel room, and I understand that there’s a closet, there’s a
bed, there’s the television that’s mounted in an unusual way. I know that
there are all these things here, and I don’t just identify them. I also
understand how they relate to one another. I have these ideas about how the
outside world works. They’re not perfect. They’re fallible, but they’re pretty
good. And I make a lot of inferences around them to guide my everyday
actions._

 _The opposite extreme is something like the Atari game system that DeepMind
made, where it memorized what it needed to do as it saw pixels in particular
places on the screen. If you get enough data, it can look like you’ve got
understanding, but it’s actually a very shallow understanding. The proof is if
you shift things by three pixels, it plays much more poorly. It breaks with
the change. That’s the opposite of deep understanding._

Of course. There are an infinte way to make interpretations of perceptions and
a finite subset of possible valid ones.

It's among those possible, that the AI will be a concrete implementation of an
ideology.

To select which one is always done by humans.

------
mlthoughts2018
This article doesn’t have any substance. It’s full of anecdata like shifting
by 3 pixels to mess up a video game AI or some vague nonsense about “a model
of this chair or this tv mounted to the wall.” It’s all casual hypotheticals.

There’s plenty of research on Bayesian neural networks for causal inference.
But even more, a lot of causal inference problems are “small data” problems
where choosing a strongly informative prior to pair with simple models is
needed to prevent overfitting and poor generalization and to account for
domain expertise.

Deep learning practitioners generally know plenty about this stuff and fully
understand that deep neural networks are just one tool in the tool box, not
applicable to all problems and certainly not approaching any kind of general
AI solution that supersedes causal inference, feature engineering, etc.

This article is just a sensationalist hit job trying to capitalize on public
anxieties about AI to raise the profile of this academic and try to sell more
copies of his book.

I’d say, let’s not waste time on this crap. There are engineering problems
that deep learning allows us to safely & reliably solve where other methods
never could. We absolutely can trust these models for specific use cases.
Let’s just get on with doing the work.

------
keithyjohnson
_Understanding a sentence is fundamentally different from recognizing an
object. But people are trying to use deep learning to do both._

I agree with most of the article but I think this^^ skips over the different
types of networks used to solve perception and language problems. A CNN is
very different from say, word2vec, which isn't a very deep network at all.

~~~
mlthoughts2018
I’d go further and say that deep networks are _excellent_ for sentence
understanding, and various types of RNN or 1D convolutional layers are very
good at this in specialized domains just as CNNs and ResNets are good in
specialized vision applications.

It _absolutely_ makes sense to use deep learning for both of these tasks.

In fact, one very effective thing to do is to use a Siamese network to learn
joint representational spaces of text and imagery _in the same network._

It’s really specious and disingenuous to say “boy, vision and language sure
seem different but can you believe these DL researchers are using the same
tools for both!?”

~~~
mattmcknight
or... "vision and language sure seem different, can you believe that networks
of neurons in the brain do both?"

------
gdubs
So, I’ve been reading articles on this and I think I have a fuzzy idea of some
of these solutions would entail. But what I’m hung up on is this: if Deep
learning is about coming up with solutions to problems that are too hard for
humans, how do we hope to understand the rationale behind whatever solutions
the machine comes up with?

~~~
dreamcompiler
A lot of DL is about teaching computers to solve problems that are easy for
humans (like driving and recognizing your grandmother) but for which humans
have a tough time explaining how they do it.

The holy grail of neural nets has always been to build a simulation of the
brain, figure out how it works, and apply that knowledge to how the human
brain might work.

We're not there yet but progress has been made. Eventually we'll understand
NNs well enough to explain not only themselves but also human brains. In any
case we have no choice because we cannot deploy NNs in life critical
situations until we understand how they work, because that's the only way to
understand how they fail.

~~~
thelazydogsback
> The holy grail of neural nets has always been to build a simulation of the
> brain, figure out how it works, and apply that knowledge to how the human
> brain might work.

I'd say that's a goal for some people -- for those whose goal is to figure out
how the brain works, rather than constructing a more ideal and powerful GI.
Remember the brain is great at some things, but laughable at others -- such as
a "7 +/\- 2" items in short term memory, inability to immediately retain rote
knowledge after one instance and in great numbers, etc. It's the merging of
the fuzzy, goal-directed behavior of the mind, in conjunction with its ability
to effect the "real world", and the super-human memory and computational
capabilities of computers that makes possible future GAIs that are so powerful
and possibly scary.

------
XuMiao
One key question is whether symbolic AI is the right model of the world. It
underperforms vector based AI on many specific tasks. But human experts
heavily reply on it to communicate with each other. If symbolic AI is not the
right model, P vs NP problem might be just irrelevant. Human philosophy is
full of craps. We will lose a lot of beliefs. Elon will be right, we will
abandon human languages, and connect through a cable in the brain. Everyone
will relearn every thing from NN.

If symbolic AI is the right model, but difficult to build algorithmically.
Vector based model just help to make it faster and better. Then we humans are
fine. We simply proxy the lower level optimization to AI. Our functionalities
will be shifted just like what happened when engine was invented hundreds of
years ago.

~~~
jkp56
Of course symbolic AI is the way to go. We communicate with text messages that
consist of words that we internally convert to word2vec style vectors to
detect similar words. One more thing we do in our heads: we build a graph of
those word2vec symbols. When I read in a book "a cat is sleeping on a tree" I
instantly build a small graph where nodes Cat and Tree are connected with an
edge labeled Sleeps. I may visualize this as a picture, but that's not
necessary for AI. In fact, some people can't visualize anything, but they can
definitely think. How? They can still build this knowledge graph. I believe
that this graph representation is the limit of our intelligence: there are
many facts out there that we can't possibly think about because they don't fit
this graph model. It's like the set of real numbers can't be squeezed into the
set of rational numbers: most of the numbers are irrational.

~~~
chillacy
We perceive ourselves building a knowledge graph but at the physical level how
does that happen? Does this graph materialize physically as neuron
connections? Or is it just an abstraction the brain makes, a part of the
subjective experience of thinking?

It's certainly true that we think in symbols but they exist somewhere in the
mushy goo of neurons, could symbolic thinking emerge from large ANNs in the
same way?

~~~
jkp56
This graph is a high level abstraction, of course. How exactly neurons store
information is interesting, but hardly relevant here. My guess is that one
symbol is stored in a very sparse subset of neurons and each neuron acts a bit
like a node in a DHT. All together these neurons implement a fast DHT where a
word2vec graph node acts as a key. On top of that this "wet DHT" can quickly
find keys nearby, i.e. in can instantly return all neighbors of
word2vec("apple").

I think ANNs implement only the word2vec function that translates images or
sounds into symbols and vice versa.

------
gautamcgoel
Gary Marcus literally gave a talk about this last week in my department's ML
seminar. I asked him how sure he was that humanity would eventually achieve
AGI and he said 100%. When I asked him when that would be he replied 30-100
years. Interesting perspective.

------
brundolf
A classical AI company that's working on "a different part of the brain":

[https://www.cyc.com/](https://www.cyc.com/)

------
1e-9
I haven't read the book, but the viewpoints he expresses in the interview are
spot-on. DL can a great alert/suggestion mechanism in narrow domains, but it
should never be trusted to make critical decisions. I believe that general
intelligence will only be achieved through major advancements in general
symbolic reasoning. Something like DL might play a small role in this
breakthrough, but it will not be a core part of the solution.

~~~
hadsed
Vectors may be the proper way to represent symbols as they are the link with
perception. I would call that a pretty core part of AI.

------
known
AI cannot make Emotional decisions which are sometimes good e.g.
[https://en.wikipedia.org/wiki/Invisible_hand](https://en.wikipedia.org/wiki/Invisible_hand)

------
The_rationalist
I always found it fascinatingly ironic that AGI research only fund deep
learning which is a local minimum.

People really interested in AGI should better look at Cyc and opencog

------
aledalgrande
Very curious about classical AI mixed with deep learning. Does anyone know of
any examples?

------
wodenokoto
In this context, what is classical AI?

An SVM? A markov model? A large context free grammar with a dictionary?

~~~
dreamcompiler
Yes. Or rules, logic, or other symbolic system. AI has always been divided
into two camps: Connectionist and not. Current "AI" is all connectionist. What
we're now calling "Classical AI" is the non-connectionist kind that was
prominent in the 60s-80s but fell out of favor in the AI Winter.

------
Iwan-Zotow
We cannot trust NI (natural intelligence) systems built on mom-and-pop having
sex alone

------
jayd16
Has work been done to formally prove general AI can't arise from deep
learning? I can't help but feel its an assumption being made by those that
prefer classical research.

~~~
celeritascelery
First you need to find a formal definition for “general AI”.

~~~
jamiek88
Which get you into defining consciousness too.

Sticky wicket.

------
fuguza
Why in earth would you trust AI alone ? Same like driving my Tesla in deep
sleep. So many times is been close encounters with obstacles..

------
samstave
I have a crazy idea:

Make a system that allows one to scan their face, and OPT-OUT OF ALL FACIAL
RECOGNITION.

~~~
jayd16
But then that system would have to delete your face too! Great, now the only
solution left for the robots is to kill all humans...

