
The Mythos of Model Interpretability in Machine Learning - alanfranz
https://queue.acm.org/detail.cfm?id=3241340
======
cosmic_ape
Its funny, and characteristic of that part of the field, that mathematical
provability in not even mentioned explicitly.

A model (and a learning algorithm) should be interpretable if we can clearly
state the assumptions, and prove that under these assumptions, we get what we
state we get.

We can predict the movement of the planets (short term). This is
interpretable, because we only assume a model of 3d space, and Newton's
laws(1). The rest is mathematics, which gives us elliptic trajectories, etc.
The resulting predictions may be wrong, due to wrong assumptions, but the
model is still interpretable.

Linear regressions, histograms, decision trees(2) are all interpretable. We
know exactly what they do, under proper assumptions. Yes, these models can be
sometimes manipulated. But we know this precisely because we know exactly what
they do. Whether we should use these models, or statistics in general, in
courts etc, is an ethical decision, and perhaps practical decision, similar to
whether we should use the death penalty. Has nothing to do with
interpretability.

With CNNs, at the moment we have very vague understanding of both the
assumptions and the models. When we do have this understanding, we will likely
also have better and simpler models.

Mathematics was always a tool for understanding reality. Proving is
understanding. And it was pretty successful so far.

(1) and some other things -- the rest of the planets are far enough, etc. (2)
given enough data wrt the size of the tree, under proper distributional
assumptions.

~~~
mlthoughts2018
I don’t agree with this. For example, you can set up a collection of
assumptions to underpin frequentist statistics, and then create a set of
theorems about consistent or unbiased estimators, and develop a theory like
that of p-values.

But then in a practical setting, the model doesn’t correspond to something
physical or to the inference goal of a practitioner. The p-value tells you
something about the relative extremity of a certain statistic that, under
certain assumptions, will have a particular distribution.

The practitioner wants to know the posterior probability of a particular model
or hypothesis given the data, and the frequentist outcome literally can’t
comment on it.

In this sense I think being able to state assumptions and connect them to
outcomes with theorems is good, but not always necessary or even sufficient
for applied work.

And there can creep in nasty subjective aspects of the problem that are
uniquely defined by the specific inference goals at hand. Proofs about how a
model would behave under assumptions are often totally useless in these cases.
Practitioners don’t use linear regression for complex financial models because
the assumptions hold or because of nice properties _if_ the assumptions held.
They use them because it’s simple and easy and sort of “just works” despite
glaring flaws.

~~~
cosmic_ape
I don't think there's a contradiction. My comment was about what it means to
understand the model, or perhaps about what it means to do science, but not
about what is necessary or sufficient for applied work.

But if we do talk about that though. Yes, science proceeds by modifying or
sometimes totally abandoning the assumptions. No silver bullet, and the point
is to assume as little as possible.

In general, if things "just work", its not a reason to abandon attempts to
understand them. Things "just work" until they don't, especially in finance.

Consider two guys. Guy A, he predicts that the sun rises every day and is
there until dusk. It "works", and so, as the joke goes, he does not touch
anything. And guy B, he knows physics and has telescopes. And he can predict
the solar eclipses. He knows the failure modes of the model of guy A. Consider
the difference between the two.

~~~
mlthoughts2018
I feel like you’re arguing against yourself here. Guy A’s shallow model would
easily be considered more interpretable than Guy B’s sophisticated
astronomical physics model, though Guy B’s model is clearly able to articulate
more detailed predictions about what might happen. Guy B’s model is less
interpretable, but because it “just works” in a vastly greater number of ways
(can be applied to the moon, constellations, other planets), people use it or
care about it.

If all that astronomy offered was a more verbose description of why the sun
rises, yielding literally zero different predictions from Guy A’s super simple
model, no one would care, and might call Guy B a witch!

I’d argue that what matters for science is pure predictive efficacy. That’s
it. If you can explain something, it means you can accurately predict
something about an unknown state of affairs that would falsify your model if
your prediction is wrong. That’s it. Any other kind of explainability is just
a matter of linguistic convenience.

Of course there is overfitting, etc., but that’s just part of refining the
model to yield greater predictive accuracy on unseen data (less generalization
error). It’s still all about putting your predictions where your mouth is.

If the differing theories of Guy A and Guy B cannot be separated by actually
testing the predictions they make, then there is no “interpretability” — just
linguistic hand waving.

Incidentally I think a lot of modern focus on interpretability is actually
about how to be a political gatekeeper or taste-maker through linguistic hand
waving, and is not about developing models who better survive the rigors of
being required to make real predictions about states of affairs.

------
DanWaterworth
You may prefer this link:
[https://arxiv.org/abs/1606.03490](https://arxiv.org/abs/1606.03490)

~~~
black_puppydog
extra points for linking to abstract, not pdf :)

------
Scea91
I think interpretability is much more correlated with model size than model
type.

Small neural net is much more interpretable than decision trees with thousands
of nodes.

~~~
taeric
I challenge that last statement. Have there been any neural nets that actually
have a solid interpretability? Usually those are more on the lines of
effectiveness in training and validation data. With no real clue as to what
the driving features were.

~~~
Scea91
[https://distill.pub/2018/building-blocks/](https://distill.pub/2018/building-
blocks/) This seems pretty good to me. And the nets are not exactly as small
as I meant.

I work with random forests and build forest which have more than 80000 nodes
per tree. Other than some basic computation of feature importance, it is a
black box on the same scale as modern neural nets, maybe even worse.

~~~
taeric
Thanks for the link. Will take me some time to digest it. Last time I looked
at one of these, it was less informative than I'd care to admit. I want to
know why a classifier found a vase. The answer is typically some form of
"because it was able to see the vase."

Would be more interesting to see a classifier that predicts something is going
to happen. A predictor that can predict a person is about to step off of a
curb, for example. Is it pose of the person? Did it require seeing multiple
frames of the person, such that it was an inertia preditor?

So, yes, comparing things to gigantic trees can make things tough. But I
thought that was the beauty of boosting and building up smaller trees. Most of
them are usually more interpretable than you might expect.

~~~
Scea91
> So, yes, comparing things to gigantic trees can make things tough. But I
> thought that was the beauty of boosting and building up smaller trees. Most
> of them are usually more interpretable than you might expect.

Regarding your last paragraph, I found this paper
[https://arxiv.org/pdf/1504.07676.pdf](https://arxiv.org/pdf/1504.07676.pdf)
worth reading. Excerpt from the abstract: "We conclude that boosting should be
used like random forests: with large decision trees and without direct
regularization or early stopping."

------
ArtWomb
An alternative proposal put out by NYU's AI Now think tank, is to require a
kind of "environmental impact statement" for any black box used in government
or public sector applications. Instead of rolling out an experimental
Predictive Policing agent en masses for example. A "sandbox" is created first.
And actual humans in the loop judge if it is prone to bias, unfairness or
harm.

Public accountability is designed into the algorithm at the outset.

[https://ainowinstitute.org/reports.html](https://ainowinstitute.org/reports.html)

------
relenzo
I thought this was a good post!

In fact, I had just read a CVPR '18 paper that did the kind of thing he
mentioned--presented an 'interpretable neural network' that assumed what
interpretability meant...

------
jarmitage
Anyone have other good references for this topic?

~~~
jclos
Interpretable ML [1] is a small online book on interpretable methods. Even if
the content itself is rather shallow, it has links to a lot of more focused
papers on the topic. Towards A Rigorous Science of Interpretable Machine
Learning is one of the most thorough papers on interpretable ML that I have
come across. The main author, Finale Doshi-Velez, has a done a lot of
interesting work on interpretability [3].

[1] [https://christophm.github.io/interpretable-ml-
book/](https://christophm.github.io/interpretable-ml-book/)

[2] [https://arxiv.org/abs/1702.08608](https://arxiv.org/abs/1702.08608)

[3]
[https://finale.seas.harvard.edu/publications](https://finale.seas.harvard.edu/publications)

