
For AI to thrive, it must explain itself - jkuria
https://www.economist.com/news/science-and-technology/21737018-if-it-cannot-who-will-trust-it-artificial-intelligence-thrive-it-must
======
chrisfosterelli
I think most people agree in the deep learning community that ways to
understand NN's are useful, if for nothing other than debugging, but I don't
think the community agrees at all over whether explainability is _necessary_
to use it.

In many cases, it depends on the context. For self driving cars, most
companies only use neural networks for components of the cars and an
explainable algorithm on top of that that interprets data from the components.
If there was a crash, they can tie it down to "the neural network parsing the
LIDAR did not classify the stop sign correctly, so the driving algorithm did
not stop" without needing much interpretability of the neural network
directly. If it's an end-to-end system, such as the nvidia car which connects
the camera inputs directly to a wheel output, then explainability of that
network is much more important.

There was a poll I saw on twitter a while ago (can't seem to find now) that
had asked if users would prefer to be treated by an AI doctor with low
explainability but 99% accuracy, or high explainability but only 70% accuracy.
I think context is key here as well. If the expert doctor working in tandem
with the explainable system is better than the non-explainable system, you'd
want the explainable system. But if the non-explainable system is better even
than experts, I think most people would want the non-explainable system.

~~~
ghaff
>But if the non-explainable system is better even than experts, I think most
people would want the non-explainable system.

Maybe? You'd have to convince me in areas as medicine. Why are you
recommending this treatment path? "Dunno. The computers says so."

Mind you, the doctor may well not in reality have a lot more solid basis for
his recommendation. But at least there's the appearance of logical
justification.

~~~
chrisfosterelli
Is that because you care about verifiability though, or actually
interpretability? If the system provides you a diagnosis without context, but
later can be confirmed by actually testing for that condition, do you still
refuse treatment because you don't know how the original diagnosis was
suggested?

~~~
ghaff
Oh, if it's a recommendation for running some tests, I can't imagine much
pushback unless they're particularly invasive or expensive. It's not like
doctors don't order large panels of tests at the drop of a hat.

If you then have treatment based on those tests, that's really a doctor being
assisted by an AI at that point.

------
kartan
Some people are missing the point.

> "In particular, machine minds that cannot explain themselves, or whose
> detailed operation is beyond the realm of human language, pose a problem for
> criminal law."

It doesn't says that the AI is NOT useful, but that it can't be used for the
liability that it creates.

> "Dr Datta feeds the system under test a range of input data and examines its
> output for dodgy, potentially harmful or discriminatory results." ... " If
> the randomisation of sex produces no change in the number of women offered
> jobs by the AI, but randomising weightlifting ability increases it (because
> some women now appear to have “male”abilities to lift weights), then it is
> clear that weightlifting ability itself, not an applicant’s sex, is
> affecting the hiring process."

This is the most interesting part of the article. As it shows that it's
possible to test the system for unlawful decisions without actually
understanding how it thinks.

~~~
twblalock
Here's another example: an AI, perhaps in a self-driving car, did something
that caused harm to people and got the manufacturer hauled into court, and the
judge issues a court order to "make it stop doing that."

How can the manufacturer comply, if they don't understand why the AI makes
that decision, and how to change its decision-making process? This isn't just
about accountability, it's also about the manufacturers' inability to control
the behavior of a black-box AI system.

If the manufacturer explains to the judge that they don't _really_ know what's
going on inside the AI, the judge won't be inclined to cut them a break.

~~~
d33
On the other hand, can you actually make a human not repeat given action? Or
explain it?

------
natch
I’d like to simply know “how good is AI right now” at any given time. This
information seems largely hidden.

For example, can any AI expert tell me: what is the exact date when Target
corporation crossed the threshold of being able to face match 20% of its
customers across different stores with 20% accuracy? How about 50% (for both
of the different numbers, to keep it simple). How about 80%? 90%?

That’s just one example. If you’re tempted to reply with insights about face
recognition, that’s not the point. The point is more that these numbers for
ANY real world AI task (not just face recognition) are generally not shared,
much less explained in the real world, even if would be possible to do so in
an academic setting.

~~~
freeone3000
These are published in papers, but for real-world products, they're
competitive advantage. Why should we share them with anyone other than our
customers?

~~~
natch
Data for very specific narrow tasks like face recognition is published. How
the capabilities are combined in a special sauce, not so much.

So yeah, going off what you said, all these people saying “don’t worry,
nothing to worry about with AI” haven’t the slightest clue what AI is up to
outside of whatever bubble they are in.

------
rdlecler1
>This means that even the designer of a neural network cannot know, once that
network has been trained, exactly how it is doing what it does.

This is just false. The reason neural networks seem so mysterious is because
in a fully connected neural network a large portion of the interactions (non
zero w_ij in the weight matrix) are completely spurious. We look at the fully
connected network topology and we throw our hands up. We can apply algorithms
to trim out the spurious weights and what we’re left with is a logical circuit
that we can analyze. Show an electrical engineer the circuit diagram of a 3
bit added and she’ll know exactly the function. Add a bunch of spurious
circuits to that same diagram and of course it’s not going to make sense.

~~~
kazinator
If those elements of the graph are "spurious", why not remove them from the
very beginning? Or, should I say, not have them there?

~~~
throwaway2048
Because we dont know which ones are spurious and the parent is pretty
optimistic about the idea of being able to analyize which ones are/are not.

~~~
rdlecler1
We only know which ones are spurious if we run the trimming algorithms. I
wrote a paper about 10 years ago showing how to use generic algorithms on the
weight matrix to remove spurious interactions.

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/)

~~~
kazinator
Do we know which ones are absolutely spurious? Or do we just know which ones
are spurious with respect to a particular result that we got?

Like for this image of Grandma, these cells are recognizing it, so the others
are spurious. Oops, for this other image of Grandma, a different subset are
recognizing it.

If, say, the same 90% of the structure is not involved in recognizing any
instances, then why wouldn't we have made the whole thing 10X smaller in the
first place and trained that much smaller network? Or why don't we make it 10X
smaller now? Trim away the dead weight and make it lighter and faster.

~~~
rdlecler1
That’s what the genetic algorithm is for. You essentially add a entropy
function that trims connections. If the performance is neutral or improves
then the leaner network topology is subject to positive selection in a
population.

------
mark_l_watson
Great article. I found a library that works with Keras last weekend; it shows
which input features most strongly contribute to classifications. My little
experiment: [http://blog.markwatson.com/2018/02/trying-out-integrated-
var...](http://blog.markwatson.com/2018/02/trying-out-integrated-variants-
library.html)

------
ims
Just as with self-driving cars, you have to look at the counterfactual. We
don't have an interpretable audit trail for the decisions made by human
employees either.

So under the status quo, what happens in the case of an aberrant or
problematic decision and how does society cope with this lack of
interpretability?

We try to piece together a plausible, retrospective narrative by looking at
fact patterns and taking into account the education, experience, actions,
explanations, and rationalizations of other humans. (Trusting these accounts
is further complicated by human traits such as self-interest, emotion, and
cognitive biases.)

There exist entire professional specialties devoted to this problem
(litigation, internal investigations, police detectives, accident boards) who
spend a lot of time on this. Even so, much of the time we still don't really
know why exactly people made the decisions they did.

This inconvenient fact does not stop us from employing humans, and society has
not collapsed under the weight of the liability issues.

------
narrator
It would be an interesting breakthrough if Alpha Go could explain its
reasoning for various plays.

Maybe one day we'll write a deep net that captions other deep nets. Who will
build the supervised learning training set though?

~~~
jonbarker
Since alphago plays the game differently (optimizing on odds to win, even if
by a little, instead of the human method which involves intuition about
shapes), it may be that its explanation if it were programmed to provide one
would not provide any human readable 'insight' which an expert could then take
to improve his or her game. Imagine a simpler example, some optimization
solution using multiple dimensions and linear algebra. A description of how
this works would be unreadable to most people. Since most readable language
operates in the world of at most four dimensions without introducing jargon,
this is the real problem, not that the program cannot explain itself.

~~~
hinkley
At the higher level it isn’t just shapes. The human player uses the shapes the
help estimate the future value of any group of stones. They are running a
priority queue of likelihood’s for each part of the board. When they resign
that probability has dropped to zero.

If you’ve played against a dan player a few times, you might have had the
experience of them ignoring a move you made and play elsewhere. That’s what
happened. It’s also probably the moment you lost the game...

~~~
jonbarker
I've played against dan players who describe this decision (tenuki, like the
tenuki suit in mario, or "to deceptively play elsewhere") as mainly an
intuition about "balance" which they somewhat vaguely describe as deciding
that although the shape may not be good, the context of the shape is balanced,
so it is OK to leave it be. According to Michael Redmond's commentary during
the alphago matches (paraphrasing) often times the ability to calculate value
is one of the last skills professionals actually master, far behind the
concepts of shape and balance, and even among them some rely more on that
skill than others. Lee Sedol, as an example, is relatively weak in this
regard, Lee Chang-ho, on the other hand, seemed to be a prodigy just at this
skill alone, allowing him to play a style which otherwise most observers
called 'rigid'. So it is in fact common to find relatively high level amateur
players who aren't good at calculating value.

------
shady-lady
Random amount of nodes with random connections assigned random weightings
until something works best.

Not so sure it's possible for anyone to seriously claim to be able to explain
the inner workings.

Biggest problem with these I can see is that there is more than one way to
arrive at the given answer.

Simplistic analogy, let's say the equivalent nodes were represented by the
value 20. Was this number a result of 5 * 4, 4 * 5, 10 * 2, 10 + 10 etc..

Right now, it doesn't matter, we're all just happy that 20 is correct and
works for our use case. Maybe that will be good enough but I doubt it.

------
jraines
Counterpoint: explain, in detail, why you clicked the "comments" link on this
post. What weights did you assign to all possible other actions you could have
taken, and why, etc?

------
mcintyre1994
There's a pretty cool paper that visualises the different layers in a ConvNet
trained on ImageNet:
[https://arxiv.org/pdf/1311.2901.pdf](https://arxiv.org/pdf/1311.2901.pdf) \-
page 4 has the layer imagery.

------
jes
I don't think AI needs to be able to explain itself in order to be trusted.
Human beings, in general, cannot give arbitrarily deep and valid explanations
for their actions, and yet, somehow, we manage to come to trust them.

~~~
throwaway2048
That's true, but in general humans are concerned about the consequences of
their actions and are metacognitive about them, or at the very least can be
punished for making the wrong choices, its an iterative corrective process.

How do you apply the same logic to a neural net? It has no comprehension of
desirable outcomes, or what its choices actually entail and never can.

There is no way to tell a neural net that a human is not a signpost directly,
so it knows in the future no humans are signposts. The only thing you can do
is train it on a larger data set and hope it "gets it".

You can tell a human being that a human is not a signpost once, and they can
abstract it to all situations.

~~~
jpindar
Hell, you can't even tell them that a red octagon on a post with the letters S
T O P on it is a stop sign.

------
hyperpallium
Historically, observation and even usage precede understanding. Like planetary
motion.

NNs (don't call it "AI", it's just one approach) are simply in a pre-
scientific phase.

As are our own brains.

------
danellis
Sounds like robopsychologist might become a real job after all.

------
jwatte
Humans don't explain themselves particularly well, yet they thrive.

------
CottageCarry
'“the reason to give reasons is so that others will evaluate your actions and
beliefs”. Today’s autonomous machines do not have their own interests to
serve. Instead, their explanations are forged by and for human beings.'

This seems like the fundamental difference between AI and human intelligence
at the moment. Our intelligence is based on our social prowess. We're always
competing with, and exchanging ideas with our peers. AI should be modeled
based on this.

