
'Explainable Artificial Intelligence': Cracking Open the Black Box of AI - sherm8n
http://www.computerworld.com.au/article/617359/explainable-artificial-intelligence-cracking-open-black-box-ai/
======
harperlee
Newbie question: I've heard that PGMs are a superset of neural networks. In
PGM materials that's I've read, the topology of the networks shown as example
is made of node that are manually chosen and represent concept (smart student,
good grades, difficult subject, etc.). Whereas a neural network example is
usually a huge set of nodes that end up finding their meaning on their own. I
also vaguely recall a tutorial in which you can highlight the nodes that
contributed to the classification - the only thing is that they don't have
meaning for a human. Then when the article states:

> restrict the way nodes in a neural network consider things to ‘concepts’
> like colour and shapes and textures.

Aren't these just PGMs? Are they NNs? Is it just a methodology approach on how
to select the topology? Don't you lose the automatic meaning / structure
search? I'm a little bit confused...

~~~
imh
PGMs are interesting in how they represent distributions over the values of
their nodes. In neural networks, (for the most part) those nodes are
deterministic, so from a PGM perspective the distribution is trivial (up until
the final output prediction). Performing inference in a neural net with
stochastic nodes would be crazy hard, so the best you can do is usually MC
with some kind of reparametrization trick to keep your gradients around.

------
bko
I think the author is overstating the importance of being able to explain in
human terms decisions made by a neural network. For instance , there is no one
reason that I am able to recognize a dog as such. Any feature or combination
of features I can think of can be had in another animal. Something deeper is
happening when I am able to correctly identify dogs that is unexplainable, at
least by me.

The examples normally given for wildly inaccurate predictions were concocted
by training a separate neural network to trick the original neural network
which seems be just showcasing the effectiveness of neural networks rather
than highlighting a weakness.

Also, I would note that human intuition is not immune to tricks. For instance
optical illusions regularly trick our perception.

~~~
mwexler
No, the importance is massive, and you understate it. While we may all want to
just believe, the regulatory ecosystem worldwide (where it applies, esp. in
financial realm) demands that we provide explanations of why models made
certain decisions on certain data. Without this ability, models will not be
allowed to drive innovation or decisions in many areas of life, from financial
(credit and risk) to medical (recommendations for treatment) to legal (best
contract approach or best defense approach for a lawsuit).

Saying that humans make mistakes and cannot explain their decisions is, in
fact, one of the very reasons we want to have better models. We hope they will
do better than most people to create a better world. And their explanations
will hopefully provide insight into how we as people make good (and bad)
decisions.

And sure, we can change the laws over time, but having advanced models which
can allow humans to understand decisions and even provide diagnostics to
improve the models will be transformative. Until then, we will see massive
impact in some areas of our lives, and frustrating holdbacks in others, driven
either by the distraction of building for regulatory constraints or by
choosing not to build in regulated areas at all.

~~~
zebrafish
We know why these models make decisions on data. They're optimizing for lower
error rates.

The black box unveiled for a convolutional neural network is this: over the
course of several thousand rounds of performing the dot product of the RGB
values of a given training datum against a weight vector, this network has
determined that a weight vector containing _these values_ optimally reduce the
error produced by "squashing" the output matrices of said dot products in a
softmax function when the "squashed" value is compared against the pre-
determined true value.

If you would also like to be able to correctly predict whether a given input
is part of the class for which this model was optimized for, we suggest your
weight vectors also contain _these values_ as this will reduce the number of
false positives and false negatives you will produce from your prediction.

~~~
ColanR
No. If the machine is telling me what medicine to administer to the patient,
then I want to know exactly what data points the machine thought were relevant
(i.e., the relevant symptoms). Furthermore, I want to know what about those
symptoms indicated to the machine a particular diagnosis.

------
yummyfajitas
Similarly, I feel that a car shouldn't drive too fast. If it does drive too
fast then a human running after it might be unable to catch up!

~~~
parenthephobia
This seems flippant. If a car is fast we generally understand why. We don't
need to worry that under some rarely-encountered combination of circumstances
it will unexpectedly do a handbrake turn and open the fuel cap.

------
bencollier49
There's a hell of a lot of money to be made by the person who cracks this. The
major blockers preventing a lot of AI being rolled out across the EU are laws
which stipulate that you have to be able to explain a decision to, for
example, refuse a person credit.

Not to mention the fact that we can correct faulty assumptions on the fly if
we can get the networks to introspect.

~~~
sdenton4
Meh. Why not just use a decision tree for the credit decision? Perfectly
explainable, and the feature domain isn't as difficult (and full of symmetry)
as vision or language problems.

~~~
PeterisP
Well, a good way to use decision trees for that is random forests, and you're
right back to something that's not really easily explainable.

------
TeMPOraL
One issue I don't see considered is - how to ensure that explainable
artificial intelligence _doesn 't lie_? Right now, it may not be an issue, but
as AI systems get complex ("smart") enough, one need to be sure that the
introspective output isn't crafted to influence people looking at it.

~~~
vitiral
Right now it looks like it's being used more as a "debugging" output to make
more intelligent Al's. Once they can lie, we will have achieved that goal...

------
cr0sh
Let's say this is possible. How would we know that it (the AI) isn't doing a
post-hoc rationalization, or just outright lying about its reasoning?

In other words, why do we trust humans more than machines? In fact, why do we
not think of humans as machines - just ones made out of different materials?
Why do we have this bias that machines are and must-be deterministic, and
since humans aren't, they must not be machines? Furthermore, since we know
that these AI models are sometimes stochastic, why do we still insist that
they be explainable; when humans exhibit the same kind of output, we don't
insist upon their determinism...?

I'm not certain that we can make these models - especially complex deep-
learning CNNs and others like them - explainable, any more than an individual
can tell you how his or her brain came up with the solution; most of the time,
we employ post-hoc reasoning to explain our decisions, depending on how the
output resolves. That - or we lie. Rarely do we say "I don't know" \- because
to do so is to admit a form of failure. Not admitting such is what helps
religion continue, because when we don't know, we can ascribe the reason to
another external force instead. If we would just be willing to say "I don't
know - but let's try to find out" (insert XKCD here), we might be better off
as a species.

I don't think an AI model will be any different - or can be. If we insist on
having an AI be able to deterministically and truthfully tell us exactly how
it arrived at such a conclusion, we must be ready to accept that we should do
the same with human reasoning as well. Anything less would be hypocritical at
best.

