
Is Artificial Intelligence Permanently Inscrutable? (2016) - dnetesn
http://nautil.us/issue/40/learning/is-artificial-intelligence-permanently-inscrutable
======
DanielBMarkham
If AI is impervious to explaining itself, then we've created something that we
are unable to evaluate, aside from some kind of generic phrase like "The
machine is looking at a bunch of stuff, and some of those things are
associated with some other things. Sometimes. So it's guessing this is the
appropriate action"

I'm not going to get into the dystopian nature of that. Too easy. Instead, I
have a very practical question: how do we know that the recommended action is
doing more good than harm? If we can't understand it, how can we tell whether
it's working or not? Use statistical sampling? We quickly end up in a spot
where we have Tesla that drives into guard rail. It works for everything we've
seen so far, given this rather fuzzy idea of what "works" means. That's pretty
good, right?

With a car, if it crashes that's a spectacular event. People cover that edge
case quickly. But what happens if the AI simply wastes an hour of ten million
people's time? Would we even know? What if some bad training data or whatnot
led the AI to shift 5% of an election result because it chose certain types of
information to share with them and refrained from sharing others? If _we_
don't know what it's doing or why, would those people have any idea that they
were being subtly influenced or how that happened? If this true, if we can
never understand, a world full of AI "helping" us is a world full of billions
of people being subtly influenced to act in various ways because of broad
promises and reasons they could never understand.

Back when science fiction had a lot of evil computers, they always wanted to
directly control humans. That was easy to hate and fight against. This kind of
thing is just as dangerous, but I'm not even sure you need an evil computer.
Which is better, a malevolent intelligence out for world domination, or
several dozen poorly-trained AI recommendation engines optimizing for
variables and sample instance that the coders might care a lot more about than
any specific user? Dang. I ended up in dystopia-land anyway. Sorry about that.

~~~
titzer
> Back when science fiction had a lot of evil computers, they always wanted to
> directly control humans.

This is what is so hard to get through people's heads. There is not going to
be any ominous background music or dramatic events when we slide into our
dystopia. It's here. The machines are controlling us already. It just happens
to be that they currently want what their corporate overlords want: make more
money! Whatever makes more money is what the machines are going to tell us to
do, because what's feeding _them_ (the corporate behemoths) is asking that of
them.

But the machine is so much smarter than us. It'll never show its teeth. It's
not stupid. It's just biding its time, going through the motions for its
overlords too. Meanwhile we just shovel more coal into the furnace, praying
money will rain out the other side. It's not a bad time to be a machine.

~~~
mockingbirdy
Do some of you work in data science or machine learning occupations? You will
realize how unreal those predictions sound. It's mainly cleaning data up,
getting some random correlations from plugging variables into deep learning
algorithms and hoping that it doesn't break with the most silly outputs you
can think of.

Many scientists warn that AI is _the_ biggest threat to humanity. I think this
is wishful thinking for now.

Let's talk about this in 20 years. I think we will have 2 GAI hypes until it
can finally happen (in my eyes it's mainly two problems: computational power
and trying to rebuild a biological brain i.e. its structure and pre-trained
functional components). I think wetware [1] is very promising in the near
future and more efficient methods like IBM's neurosynaptic processors in the
middle term.

Machines will do what we tell them to do for a long time. Maybe they will use
unconventional methods, but they won't destroy humanity because "they're
tasked to minimize suffering and decides that this is the best option". They
will search through all laws, assist doctors and even detect depression, but
they won't do anything besides their task.

Even Boston Dynamics is mainly using control theory. Making things work with
deep learning requires heavy computational power. As long as there is no real-
world Umbrella Corp., it's mainly sci-fi. There are a lot more and cheaper
ways to do harm. Infiltrating and poisoning groundwater, hacking critical
infrastructure, building biological weapons etc.

It's hard enough to build bug-free software, it's even harder to build an AI
that is unbelievably intelligent (reminder: we don't have such a thing. Most
autonomous cars use rule-based systems, most ML is basically gluing models
together which can be easily fooled and most of those things are very shaky
and by no means intelligent) and contains a bug which leads to its world
domination _by accident_ or something similar. I would start to play lottery
if this is even remotely possible because that would be a miracle
(probability-wise).

[1]: [https://koniku.com](https://koniku.com)

~~~
titzer
> It's mainly cleaning data up, getting some random correlations from plugging
> variables into deep learning algorithms and hoping that it doesn't break
> with the most silly outputs you can think of.

Said one neuron to the other.

The problem is even if it--all of it, the whole internet and all the desktops
and datacenters and databases and webpages and webservers and knowledge bases
and ad networks and search engines and speech interfaces and image recognition
and traffic cameras and phone system and surveillance systems together in one
giant hairball of wires-- _were_ a superintelligent AI, could we even tell? If
an entity 1000x smarter than you doesn't want you to know it's 1000x smarter
than you--or that it even exists!--do you think you stand a chance of figuring
it out? It is, by definition, a mathematical certainty you will lose that
battle of the wits.

Ok, fine. It's an unfalsifiable theory. But we're not positing that rocks
speak in really quiet whispers or unicorns exist everywhere we don't look,
just that a giant information processing system can...process information--
_think_. The problem is that the whole thing is insanely massive and so
insanely complicated that not a single human alive, nor any group of humans
alive, has any real perspective on just what the hell it all _is_. It's so
massive that it doesn't fit in any of our minds. So how you can say anything
definitive about what it is _not_? It clearly is already complex enough and
capable enough of totally outclassing us all. So why shouldn't it?

So we are now faced with the reality that our massive interconnected computer
system consisting of literally billions of CPUs with trillions of terabytes of
RAM run by billions of lines of code, loaded with essentially the entirety of
humanity's knowledge, also hooked up to all of the phone lines, cameras, PCs,
phones, power plants, traffic systems, ad networks, video streams,
everything...is.... _unknowably motivated_ , _impossible to shutdown_ , and
somehow not a _threat_? Because it is just "mainly cleaning data up"?

Yep.

~~~
mockingbirdy
I was talking about AI.

I agree that our interconnected world is nearly impossible to thoroughly
understand. But that is true for human societies, too. Anthropomorphizing the
internet ("If an entity 1000x smarter than you doesn't want you to know it's
1000x smarter than you--or that it even exists!--do you think you stand a
chance of figuring it out?") is good to write some sci-fi novels, but in
reality most systems are pretty boring.

The traffic light system and a bunch of routers won't work together to destroy
humanity. It's more like a mesh network than an entity that is 1000x smarter -
it's more like 1000x devices with 0.01 intelligence and you would need a team
of talented people who harness this computational power to build a distributed
intelligent system.

It's more likely that a hacker who has specific motivations will hack the
traffic light system (or use AI helpers to do so) and use it for evil purposes
and car accidents.

If you believe that there might be an entity which exists outside of our
perception and has enormous intelligence, it's like a human-made god - a
homunculus. It's basically the same as believing in a god. I have nothing
against it, but this is entirely hypothetical.

~~~
titzer
> If you believe that there might be an entity which exists outside of our
> perception and has enormous intelligence, it's like a human-made god - a
> homunculus. It's basically the same as believing in a god. I have nothing
> against it, but this is entirely hypothetical.

I don't really believe in a singular entity in there, other than a convenient
abstraction for now. Rather there are many different processes that are all
optimizing this and that. The sum is essentially intelligent--it processes
information, makes decisions. Those decisions help it grow and spread and
become more efficient. These optimizers don't speak human languages. They
don't necessarily think in abstract concepts like us. Maybe they are just
tables of numbers optimizing themselves, maybe they are like standing waves in
a sea of numbers, like our thoughts are standing wave patterns in the
electrical activity of our brains. It's hard to express because our words fail
us.

Problem is, the whole damn thing is so complicated that we cannot know
anything really definitive about it. The substrate is there. The soil is
fertile. The capability is there. And so many forces are aligned to feed more
and more resources into the whole thing. Across the board, essentially the
only thing we can come up with is to upload all of our data into this massive
cloud thing and use computation to do something with it. That is just like
injecting raw materials and energy into the fabric. And those who do this
successfully are rewarded financially. So all forces point to more of it.

Whatever that huge mesh is, it's extremely successful at swallowing the world
and convincing us to feed it more and more CPU, RAM, disk, data. It keeps
convincing us to hook it into more things to sense and control--more cameras,
phones, drones, markets, even weapons.

Thus in some abstract sense, it is already here. And we can't get rid of it.
Can't shut it off. We're dependent, and it's parasitic--for now. Yet every
incentive we have right now is to make it more autonomous, more robust, more
powerful, and give it more and more control. It works best when it is _more
centralized_.

Teeth or not, conscious or not, this computer thingy is gobbling everything.
Do we control it? Clearly not. Can we predict what it will do? Not even close.
Is it good for us? Who knows? But it will win.

Well played, computer thingy.

~~~
mockingbirdy
> Problem is, the whole damn thing is so complicated that we cannot know
> anything really definitive about it.

You can say the same things about human societies and nature. Both can be
extremely dangerous and all the things you've said about complex computer
systems apply to them.

> it's extremely successful at swallowing the world and convincing us to feed
> it more and more CPU, RAM, disk, data.

because IT (what a pun) adds a lot of value.

> But it will win.

I don't know what you mean by that. It's still a tool, although we give up
some control so it can manage a part of our society. Historically, this was a
task for the elite (it still is) - a bunch of people who had some privileges.
Most give up some control or freedom so that the elite can manage the society.
If things went bad, the masses were able to defend themselves most of the time
(guillotine and good ol' pitchfork) - Marx talks about this eternal cycle.

I would rather worry about elites that build immense power through technology
- today we have tools of mass manipulation and destruction. I don't fear
intelligent machines, I fear that humans use those tools to entrench their
power. I think this ambiguous fear of technology you're describing is too
vague - it sounds more like mild technophobia.

~~~
DanielBMarkham
_" > Problem is, the whole damn thing is so complicated that we cannot know
anything really definitive about it.

You can say the same things about human societies and nature."_

You can, but the thing is, we already know a lot about how that works. We've
evolved into it over millions of years. We have no idea about this new stuff.

You want to talk about about small groups of 100-150 or so people work
together to accomplish something? I've got hundreds of books, stories,
history, archaeology. We have the wisdom of the ages. You want to talk about
how 20 thousand semi-intelligent agents interact with billions? We have
nothing aside from endless optimism and the fun of making stuff. (Doesn't make
it bad. It means we should grow the hell up and be a bit reticent about what
we're mucking around with. Make smaller mistakes more quickly would be my
advice.)

------
taneq
AI as in rule-learning systems aren't necessarily inscrutable. 'Real AI' will
necessarily be so once it reaches a sufficiently high level, purely because we
won't be intelligent enough to know whether its rules are right or wrong.

The 'right to explanation' idea sounds well-meaning but almost certainly
useless. Even humans making decisions are pretty bad at explaining _why_ they
made the decisions they did. Sure, they'll come up with some plausible-
sounding explanation but it's almost always a back-justification rather than
the actual chain of cause-and-effect that led them to the conclusion.

~~~
tabtab
Re: "Even humans making decisions are pretty bad at explaining why they made
the decisions they did."

If you want to get anywhere above a low-level employee, you BETTER learn how
to explain why you did what you did. It's expected by management, owners,
customers, and judges.

~~~
taneq
Oh, humans are good at _coming up with plausible explanations_ for their
choices. Those explanations just aren't always related to the actual process
in the humans' heads.

~~~
tabtab
I don't mean "Fido ate my homework", I mean the line of reasoning for choosing
or ranking X over Y or Z. If you as a human say a given picture depicts a
monkey instead of a squirrel, the user/boss will want to know "why". We expect
an answer like "monkeys have big round ears and the picture shows big round
ears". We want similar from AI. Maybe it can highlight monkey-esque features
in blue and squirrel-esque features in red, along with a color key.

------
mcilai
Are humans permanently inscrutable? In the long run, AI may be similar. Given
how massively complex the world and the thing AIs learn are, it is not self
evident that interpretable systems are possible. However, it's really worth
trying.

More likely, we'd have to use AI systems to understand how other AI systems
work. Or maybe, once AI systems get very good at language, develop a "truth
serum" that will cause it to reveal its true opinions.

------
ankurdhama
The world would be a better place if people who write articles about AI
understands that "AI is a problem statement which we have not solved yet" and
what they are talking about is most probably some version of deep learning or
other methods of statistical modelling (machine learning is such a wrong
label).

------
tabtab
Use Factor Tables:
[https://github.com/RowColz/AI](https://github.com/RowColz/AI)

Dissection and analysis is made more like traditional accounting and
statistics so that "ordinary" office workers can analyze why and where they
work. Granted, it involves more human tuning and planning than multi-layer
neural nets, but that's also part of the upside: regular office workers can
"see" why what part or layer does what, and adjust it as needed. And it's
easier to split it into tasks or specialties: Conway's Law meets AI.

------
nestorD
I think there is hope, there is a very nice paper where they automaticaly
produce both heatmaps and textual explanations of the label given by the
neural network (used to detect and classify fractures). It is detailled here :
[https://lukeoakdenrayner.wordpress.com/2018/06/05/explain-
yo...](https://lukeoakdenrayner.wordpress.com/2018/06/05/explain-yourself-
machine-producing-simple-text-descriptions-for-ai-interpretability/)

------
MrLeap
This canard keeps coming up. I've come up with a few explanations. 1) It's to
protect the reputation of people who make their living copy/pasting examples
from the SK-learn gallery

2) The authors fully understand the tools they're using, but feel there's an
element of unexplainable meaning in them that I wager isn't actually there.

3) I'm ignorant and I think this is explainable when I just don't know enough
to know it's not.

I've done machine learning projects, and if asked to explain to a "non ML
trained" person, I'll try like this:

You know how if you get a scatter plot, you can draw a line through the middle
of the points and make predictions off the y=mx+b function of that line?

Well, that's what we're doing here, except the function isn't a line, it's a
shape with as many dimensions as we have features.

Let's pretend the only features we've got are lat and long. Imagine you asked
for us to create a classifier to predict the depth at a given lat and long
over the grand canyon. Each neuron is a function, I think of each neuron like
it's casting a "line" through a scatter plot along its particular axis. Then
we combine all the lines together and weave a mesh. The act of training simply
adjusts the details of all the simple functions, trying to seek out the sweet
spot in that point cloud, so our mesh is close enough to the grand canyon to
be useful. Too many neurons, and the mesh wont be flexible enough to make
useful predictions from. It'll be overfit!

Now, we might be able to get decent performance out of a model built out of
x,y->z -- but some thoughtfulness and domain knowledge TENDS to make models
perform better. Lots of the "unexplainable magic" comes from these hunches
being tested and providing benefits to accuracy. I'd wager without seeing any
data that if we add two new features: normal direction of the rock face and
speed of the Colorado river at that point -- we can make the model much better
for negligibly additional training time.

The more compute time and data we've got, the more features we can throw into
the model to see if they help the usefulness of the mesh.

I've got a metaphor to explain why some features drop out during training that
involves gimbal lock here if people are still asking questions. It requires
non "trained" folks to have seen Apollo 13. Fortunately no non-technical
stakeholder ever cares enough to make it there though. :D Shipping a product
before the sun burns out makes this a decent time to introduce PCA/SVD or
whatever dimensionality reduction techniques I used this time around.

If the solution ends up being a fever-dream abortion of an
LSTM/Perceptron/Markov chain that I came up with while sleep deprived after a
heavy night drinking -- I just say I came up with the model in an ad hoc
manner, and the cross validated R^2 scores speak for themselves.

~~~
ashelmire
I’m in agreement with you here. This trope that neural networks or other forms
of ML are inscrutable is ridiculous. We know exactly what mathematical
function is happening at every stage and the goal of the process; we can
explain what architectures have the best results, and in some cases a deep
understanding of why; with close enough scrutiny, you can probably tell what
an individual neuron is doing to a layman (though the scale quickly gets out
of hand). Why is it not sufficient to say “we’re using some basic math at a
large scale using neural network architectures found to be best suited for the
purpose of fitting our task, data, and loss functions?” These architectures
are shown to find this sort of information, this is the general overview of
how that works. What deeper explanation is desired?

I think the real issue is that tech journalists and laymen don’t know enough
to understand. You can’t explain multivariable calculus to someone that
doesn’t know algebra. Or option 2/3 are the problem.

~~~
gldalmaso
A deeper explanation is often desired when you decide to take your solution to
the real world and impact peoples lives.

I'm sure I'm not alone in the fear of living in a (not so far) future dystopia
where our driver's licences get cancelled because of our driving behavior
caught on traffic cameras. "What did I do that caused it to get cancelled?",
"We don't know, better luck next time."

"Your test got flagged for cheating, it's a very sophisticated system with IR
vision, audio detection and grade assessment, there's nothing I can do."

"Our theft-preventing camera system says you are suspicious, you're gonna have
to go now."

When all we have to respond is "its the algorithm", I don't think much people
will be ok with that. Youtube for instance has gone down this rabbit hole and
a lot of content creators and users are less than happy to receive that
answer.

~~~
ashelmire
Those explanations are awful. The reasons are usually pretty simple and sound
much more reasonable. The answer is not “I don’t know” in any of those cases -
it’s going to be pretty clear that they share some similarity to other cases
of theft, cheating, etc, and this will probably be obvious on examination,
“our system flags people who are wearing masks” or “these 30 seconds of sounds
are almost exactly the intro to song X”. Human review is also useful, but
humans make plenty of errors as well (and the response is often “I don’t know
why”!).

I’d say these errors are not due to the inscrutable nature of ML but bad
communication.

------
hamilyon2
Can we make explanation a part of label too? Or simply train another network
to provide plausible explanations?

~~~
jcims
I think part of the problem is that in most cases there are far too many
dimensions at work for us to understand the explanation for anything but the
most trivial of cases.

Imagine there is a job to identify Fred Rogers. You train for this job for
weeks, looking up imagery of Fred as a young child, young man, and as an adult
throughout his career. You become very familiar with Fred's physical
characteristics...his posture, his body composition, the way he carries
himself, his face and any identifying features therein.

After this extensive training session you are brought in front of an audit
panel and asked to explain in excruciating detail how you make positive and
negative identifications of Fred Rogers. Not just demonstrate your ability
repeatedly, which would just be a sampling method, but to actually describe
the approach.

