
Explainable Artificial Intelligence (XAI) Darpa Funding - Dim25
https://www.fbo.gov/index?s=opportunity&mode=form&id=1606a253407e8773bdd1a9e884cc5293
======
wyc
This is important because there exists a trade-off in statistical learning
models: in general, the more flexible your models are, the less understandable
they become[0]. Modern machine learning techniques are typically very
flexible.

To gain intuition and reasoning of a model is to have understanding and trust
--transparency. When you strike a nail with a hammer, it's pretty predictable
what might happen: the nail could get hit, the hammer could miss, or very
rarely, the hammer's head may fly off of the handle. When you replace the
hammer with a black box that works correctly 99.999% of the time, but for
0.001%, something completely unpredictable happens, then there's a problem
with volatility because that unpredictable event may have unacceptable
consequences. I think explainable AI could help with intuitive and more fine-
grained risk analysis, and that's certainly a good thing in high-stakes
applications such as defense.

[0] ISLR, Page 25: [http://www-
bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.p...](http://www-
bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.pdf)

~~~
taneq
This is interesting because usually "human-level performance" is the benchmark
by which AI is judged. If it can do the thing as well as a human can do the
thing, then that's good.

Human intelligence isn't explainable, though. We can make up explanations but
they're not based on our actual neural function (heck, we don't even know how
that works in general, let alone how we generate specific outcomes).

[https://en.wikipedia.org/wiki/Introspection_illusion](https://en.wikipedia.org/wiki/Introspection_illusion)

~~~
Cybiote
Illusory Introspection and confabulation are fascinating but generally have
more to do with _what was going on in your head when you took that action?_
than _can you explain why you think that?_. XAI is not interested in
psychological state or an explanation in terms of ion concentration, voltage
changes and synaptic activations. Or in the case of AI, the meaning of the
contents in vectors after maps on matrix vector products.

A large part of human success is that we are able to transmit knowledge to
each other and accumulate experience over generations. A human starting from
scratch is not very much more than its ape cousins.

Human beings can explain efficiently because we _must_ learn concepts
decomposably and can take deductive shortcuts. In its purest form we have
mathematical proofs as explanations. Explanations work with core concepts and
generates on that basis. E.g. We want to know A and we know B, B is applicable
in condition C. We will show condition C applies. With B, A can be shown in
terms of...

When you give big data to machines they work with raw correlations and output
predictions. But humans have to be more creative because our scratch space is
so small. Newton took Brahe's big data and compressed it to a theory of
universal gravitation. Darwin made a relatively small number of observations,
all things considered, and came up with On the Origin of Species.

~~~
Houshalter
Only a tiny fraction of human knowledge is transmitted through knowledge.
Humans can't describe the wiring of their visual cortex, or the intuitions
they feel. Expert chess players find it impossible to explain why they make
the moves they do. Mathematicians can make proofs and explanations, but they
still find it very difficult to transmit their underlying mathematical
intuition. Even the purest fields of math can only really be learned through
experience and practice, not just following a few proofs mechanically.

~~~
seanmcdirmid
Language allows us to communicate abstractions, which are then of course
subject to interpretation. This is much better than other animals, which lack
language facilities as far as we can tell. The entirety of human knowledge
cannot be passed, but enough information is communicated that we can rebuild
much of it in every generation.

------
dkarapetyan
This is fantastic. DARPA gets it. I look forward to whatever fruits come from
this labor. Maybe one day I won't have to look at stack traces and reverse
engineer 3rd-party dependencies to figure out why things are breaking. Maybe
one day error messages will have explanatory power. Maybe one day IDEs will
understand abstractions other than ASTs and types and instead will understand
things that convey human intent that is not so closely tied to rigid
constructs like type systems. What a wonderful world that will be.

~~~
BenoitP
> Maybe one day error messages will have explanatory power

I firmly believe that closely integrating the raw ML internals with the user
experience will yield tremendous rewards.

The coding experience, with the stack trace debug experience loop (get
stacktrace -> google -> stackoverflow.com -> try a new thing) could be vastly
improved, and be made to be like an Akinator session [1].

How about having the IDE's console output be integrated in ML pipelines? You
would have boxes with questions and suggestions like:

* Please select the words in the console output that are not supposed to happen.

* Is you current goal related to the following tag: a) library_upgrade, b) first_time_library_addition, c) <tag search box>

* Please describe with tags and words the context you are in.

* Go read this stackoverflow page. Did it help?

* I see you have been doing x and y, and getting these errors. Would you be interested in this tutorial?

* Last week you had this problem. It is resolved? What things (urls, boxes I presented you, etc) did it?

* Here is another context: eclipse, java, email, library_upgrade, ConcurrentModificationException. Are you having the same issues?

* Here are statistics about people being in the same context are you are in. Here is also the top remark they have said about it.

* Here is a decision tree node your context is currently in. Here are all the child nodes (lets you explore the tree without tainting your current context)

* Would you like to do some semi-supervised clustering for trying to tie your current context to other contexts?

And then have the dataset be openly accessible, with third party being able to
provide boxes, and publicly emit new features, and all boxes being rateable.

With it implementing the base stackoverflow feedback loop, it would yield a
user experience superior or equal to it.

[1] akinator.com

~~~
shostack
Are there any good Stack Overflow IDE integrations out there?

------
Dim25
Direct link to detailed specification [PDF]:
[https://www.fbo.gov/utils/view?id=ae0b129bca1080cc7c517e8dad...](https://www.fbo.gov/utils/view?id=ae0b129bca1080cc7c517e8dadfa3ca2)

Related FAQ [PDF]:
[http://www.darpa.mil/attachments/XAIFAQ8-26.pdf](http://www.darpa.mil/attachments/XAIFAQ8-26.pdf)

~~~
jarmitage
Thanks for sharing. Do you know anymore about the authors / who might have
been consulted to write this?

The contact listed in the document ([http://www.darpa.mil/staff/mr-david-
gunning](http://www.darpa.mil/staff/mr-david-gunning)) used to work for PARC
and ran the PAL (Siri) program.

~~~
Dim25
Great question, no idea who are the exact people behind this (just found this
online), but according to [http://www.darpa.mil/tag-
list?tt=73&PP=2](http://www.darpa.mil/tag-list?tt=73&PP=2) some interesting
folks joined them recently (after 2013):

Mr. Wade Shen (Program Manager – interests: machine learning)
[http://www.darpa.mil/staff/mr-wade-shen](http://www.darpa.mil/staff/mr-wade-
shen)

Dr. William Regli (Defense Sciences Office (DSO), Deputy Director – interests:
artificial intelligence, robotics) [http://www.darpa.mil/staff/dr-william-
regli](http://www.darpa.mil/staff/dr-william-regli)

Dr. Reza Ghanadan (Defense Sciences Office (DSO), Program Manager – interests:
data analytics, autonomy, machine learning and artificial intelligence in
information and cyber-physical systems) [http://www.darpa.mil/staff/dr-reza-
ghanadan](http://www.darpa.mil/staff/dr-reza-ghanadan)

Dr. Paul Cohen (Information Innovation Office (I2O), Program Manager -
interests: artificial intelligence, machine learning)
[http://www.darpa.mil/staff/dr-paul-cohen](http://www.darpa.mil/staff/dr-paul-
cohen)

Most of them can be found on linkedin.

------
iverjo
This brings Lime [1] to mind. "Explaining the predictions of any machine
learning classifier"

[1] [https://github.com/marcotcr/lime](https://github.com/marcotcr/lime)

~~~
vonnik
That's what I was thinking of, too. If any one wants the direct link to the
arXiv paper, it's here:
[https://arxiv.org/abs/1602.04938](https://arxiv.org/abs/1602.04938) . One of
the authors is Carlos Guestrin, one of the co-founders of Dato/Turi/Graphlabs
that was recently acquired by Apple, fwiw.

------
Houshalter
There was a machine learning system designed to produce interpretable results,
called Eureqa. Eureqa is a fantastic piece of software that finds simple
mathematical equations that fit your data as good as possible. Emphasis on the
"simple", it searches for the smallest equations it can find that works, and
gives you a choice of different equations at different levels of complexity.

But still, the results are very difficult to interpret. Yes you can verify
that the equation works, that it predicts the data. But why does it work? Well
who knows? No one can answer that. Understanding even simple math expressions
can be quite difficult. Imagine trying to learn physics from just reading the
math equations involved and nothing else.

One biologist put his data into the program, and found, to his surprise, that
it found a simple expression that almost perfectly explained one of the
variables he was interested in. But he couldn't publish his result, because he
couldn't understand it himself. You can't just publish a random equation with
no explanation. What use is that?

I think the best method of understanding our models, is not going to come from
making simpler models that we can compute by hand. Instead I think we should
take advantage of our own neural networks. Try to train humans to predict what
inputs, particularly in images, will activate a node in a neural network. We
will learn that function ourselves, and then it's purpose will make sense to
us. Just looking at the gradients of the input conveys a huge amount of
information of which inout features are the most and least important. And by
about how much.

But mostly I think the effort towards explainability is fundamentally
misguided. In the domains where they are supposedly the most desirable, like
medicine, accuracy should matter above all. A less accurate model could cost
lives. Accuracy is easy to verify through cross validation, but explainability
is a mysterious unmeasurable goal.

~~~
woodman
Eureqa uses (used? I haven't touched it in years) symbolic regression.
Symbolic regression can certainly produce weird results, especially with low
quality data, but I wouldn't rate it anywhere near classic ANNs in black-
boxiness.

Also, science is unfortunately full of magic numbers - but it is pretty
amazing when you feed in pendulum motion data and the resulting equation ends
up being Newton's second law: [http://phys.org/news/2009-12-eureqa-robot-
scientist-video.ht...](http://phys.org/news/2009-12-eureqa-robot-scientist-
video.html)

~~~
Houshalter
Yes it's amazing that it can find an equation that fits the pendulum's motion.
But if you _didn 't already know_ the physics math, it would just seem weird
and arbitrary.

~~~
woodman
No more weird and arbitrary than the motion of the pendulum itself. I think
you're underestimating the human intelligence component of the system. Include
in the dataset an additional variable for rope length, record the data for
multiple lengths - there, you've just isolated a variable in the equation and
it is no longer arbitrary. The scientific method still works, but now you're
working on a much more tightly bounded problem.

------
bluetwo
(Looks over shoulder)

Anyone else thinking this mirrors their own experimental work?

Anyone else thinking of putting in an abstract?

Abstract Due Date: September 1, 2016, 12:00 noon (ET)

Proposal Due Date: November 1, 2016, 12:00 noon (ET)

------
Eliezer
(I am relatively excited about this research direction. It seems like the sort
of thing that might lead to genuinely useful components of a safer AGI system
later.)

------
dschiptsov
Yeah, all they want is a simple mechanism of how to jump from a mere "blind",
mechanistic feature extraction to the notion that creatures of this Nature
usually have two eys and make a hard-wired heuristic, a short-cut which
improves pattern recognition in orders of magnitude with less computational
cost.

Every child will tell you that cars have eyes, and even a crow could track the
direction of your gaze.

Well, I would also give away some govt. printed money to know how to make this
kind of a jump from raw pixels to high-level shapes.)

The answer, by the way, is that the code (which is data) should be evolved
too, not just weights of a model. This is an old fundamental idea from the
glorious times of using Lisp as AI language - everything in the brain is a
structure made out of conses^W neurons.

And feature extraction and heuristics should be "guided". In the process of
evolution it is guided by way too many iterations of training and _random_
selection of emerging features. Eventually a short-cut "creatures have eyes"
will be found and selected as much more efficient. We need just a few millions
of years or so of brute forcing.

Hey, Darpa, do you fund lone gunmen?)

------
vonnik
Integration of Neural Networks with Knowledge-Based Systems [https://www.uni-
marburg.de/fb12/datenbionik/pdf/pubs/1995/ul...](https://www.uni-
marburg.de/fb12/datenbionik/pdf/pubs/1995/ultsch95integration2)

------
yoav_hollander
If there is real progress towards Explainable AI, this would also be very
useful for _verifying_ machine-learning-based systems (i.e. finding the bugs
in them).

I wrote about this in [1], but I am not a machine-learning expert (I am coming
from the verification side), so would love to hear comments from other people.

[1] [https://blog.foretellix.com/2016/08/31/machine-learning-
veri...](https://blog.foretellix.com/2016/08/31/machine-learning-verification-
and-explainable-ai/)

------
michaelscott
This is sorely needed for machine learning if it's to get both more complex
and more accurate. Coincidentally Alan Kay brought the "expert systems" idea
up in his recent AMA as well. It'd be inconceivable to write code today that
couldn't be thoroughly debugged, so we should expect the same of our machine
learning systems.

------
vonnik
Bringing some form of feature introspection to deep neural networks will
probably involve clever ways of visualizing the feature activations of
unstructured data
[https://arxiv.org/abs/1603.02518](https://arxiv.org/abs/1603.02518)

------
pattisapu
"If you can't explain it to a six year old, you don't understand it yourself."
-Einstein

~~~
dkarapetyan
I like Feynman's version better. If you can't explain it to a college freshman
then you probably don't understand it.

------
breezest
The goal is ambitious. I have similar ideas in my mind but do not have a team
to complete the details.

------
hackcasual
My Aunt worked on systems for explaining early generation networks for medical
diagnoses:
[http://link.springer.com/article/10.1007/BF01413743](http://link.springer.com/article/10.1007/BF01413743)

------
syats
"I know of an uncouth region whose librarians repudiate the vain and
superstitious custom of finding a meaning in books and equate it with that of
finding a meaning in dreams or in the chaotic lines of one's palm ... "

JL Borges, The Library of Babel

------
dmix
Off topic: I checked other FizBizOpp listings (which is now famous thanks to
the War Dogs film in theaters). There is a listing for a "Big Ass Fan" 16'
long for the Air Force:
[https://www.fbo.gov/index?s=opportunity&mode=form&id=8de699e...](https://www.fbo.gov/index?s=opportunity&mode=form&id=8de699e71f1dc9a084509651f2221cf3&tab=core&_cview=0)

~~~
aoki
apparently had a preferred vendor in mind:
[http://www.bigassfans.com/](http://www.bigassfans.com/)

------
eli_gottlieb
Well, if you were gonna submit an abstract, the deadline was a week ago. Good
luck getting a grant proposal ready if you haven't already!

