
The Mythos of Model Interpretability - jboynyc
https://arxiv.org/abs/1606.03490
======
cs702
When educated people like doctors, lawyers, managers, etc., ask for
"interpretability," what they mean is, "what are the 'key variables' we can
control that will help produce the outcomes we want?"

Simpler statistical models, and particularly linear models, provide easy-to-
grasp "answers" to that question -- for example, "to get more of y, we need
more of x1 and x2, and less of x3."

An entire generation of leaders and professionals has been trained to think
this way by statistics professors and teachers who have been rigorously
trained to think that way too. (For background, I highly recommend Leo
Breiman's now-classic paper, "Statistical Modeling: The Two Cultures:"
[https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...](https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726)
)

Alas, there's no going back. Simpler models _do not work_ for complicated
tasks like, say, predicting the next best move in a game of Go, for which
there are no 'key variables' in the traditional sense.

~~~
p4wnc6
It goes further. Folks who care strongly about the interpretability factor
generally only care because it gives them authority and political control --
they definitely don't actually believe it's a superior, nor even cost-
effective, way of obtaining prediction accuracy.

Robin Hanson summarized this well in his article "Who about forecast
accuracy?" [0]. Here's a nice quote:

> The benefits of creating and monitoring forecast accuracy might be lower
> than we expect if the function and role of forecasting is less important
> than we think, relative to the many functions and roles served by our
> pundits, academics, and managers.

> Consider first the many possible functions and roles of media pundits. Media
> consumers can be educated and entertained by clever, witty, but accessible
> commentary, and can coordinate to signal that they are smart and well-read
> by quoting and discussing the words of the same few focal pundits. Also,
> impressive pundits with prestigious credentials and clear “philosophical”
> positions can let readers and viewers gain by affiliation with such
> impressiveness, credentials, and positions. Being easier to understand and
> classify helps “hedgehogs” to serve many of these functions.

> Second, consider the many functions and roles of academics. Academics are
> primarily selected and rewarded for their impressive mastery and application
> of difficult academic tools and methods. Students, patrons, and media
> contacts can gain by affiliation with credentialed academic impressiveness.
> In forecasts, academic are rewarded much more for showing mastery of
> impressive tools than for accuracy.

> Finally, consider next the many functions and roles of managers, both public
> and private. By being personally impressive, and by being identified with
> attractive philosophical positions, leaders can inspire people to work for
> and affiliate with their organizations. Such support can be threatened by
> clear tracking of leader forecasts, if that questions leader impressiveness.

FWIW, having worked for a while in quant finance, I can testify that this
precisely nails it. Even when it would be cheap, easy, obviously worthwhile,
and likely impactful, projects to switch from politically-controlled,
simplistic regressions couched in the buzzwords of academic finance and
instead embrace principled statistical computing, proper model fitting
hygiene, machine learning, etc., such improvements will _never_ be considered.
There's too much political gatekeeper rent-seeking at the higher levels of the
management who oversee the investment process. Sure, they'll claim plausible
deniability by saying something is "a black box" or making a fallacious
comparison with an algorithmic trading firm that lost money, but the real
reason is to protect their fiefdom, over which they are the lord and master
because of their credentialed authority in some "interpretable" buzzword.

[0] < [http://www.cato-unbound.org/2011/07/13/robin-hanson/who-
care...](http://www.cato-unbound.org/2011/07/13/robin-hanson/who-cares-about-
forecast-accuracy) >

~~~
moultano
> they definitely don't actually believe it's a superior, nor even cost-
> effective, way of obtaining prediction accuracy.

Prediction accuracy is not the only goal. There are only a handful of times in
my life where I've seen a machine learning task where the training data was a
perfect match for the business objective. The training objective is a proxy
for the actual objective, where the gap between the two includes a lot of
systematic bias. Interpretability is necessary to make sure the model isn't
merely exploiting that gap. (And also to motivate further improvements.)

~~~
p4wnc6
I don't think what you are saying is incompatible with my comment. Asking a
question such as, "what is it about the data that is causing the model to be
fitted like this?" is very healthy and important, but is rarely what anyone
means by 'interpretability.'

Instead, 'interpretability' is usually taken to mean something more akin to:
which readily human-understandable cognitive structures (out of the set of
such structures that I am the authortity on) is the model directly (and
easily) mappable to? By mapping onto those structures, I can now weave a story
about the model within the domain in which I am the established authority. If
such mappings don't exist, or are very difficult to compute, then my authority
is threatened.

Having said all of this, the most egregious abuses of the nefarious side of
'interpretability' that I have seen have all come in quant finance, which is a
situation in which the optimization criteria are _literally_ the identically
same thing as the actual objective.

Given that when prediction accuracy is paramount to the definition of success
you still see politics trumping science, I have very little confidence that
when you move to domains where the objective criteria of the models differs
from the real-life objective in non-trivial ways, then somehow
'interpretability' will be used appropriately rather than overwhelmingly just
as a political tool.

~~~
radicalzebra
With respect, I think there's a fundamental error in this reply. You're making
a dogmatic statement about what interpretability is taken to mean, which
assumes of course that the vast body of work, both in academia, journalism,
and business meetings, holds a consistent view. The point of this paper,
however, in part, is to point out that this view is mistaken.

~~~
p4wnc6
You are incorrect in your claim. The paper does make a good point that under
the umbrella term "interpretability" there are varied and conflicting
understandings of what, specifically, is meant.

That fact has nearly zero relevance to what I'm saying.

When I say that people look for mappings from the model to a space of
cognitive structures over which they have authority, it could be (and observed
in practice always is) a smattering of all the different things talked about
in the paper -- authority over which diagnostics represent 'trust,' authority
over which visualizations are adequately informative, authority over
explaining 'how' an algorithm works, and so on.

Though all of these are discordant takes on the specifics of interpretability,
as discussed in the paper, they still are unified in the sense that the _true_
intention in appealing to any of them is to map things into a domain where
someone wields the authority. It's not about actually caring about trusted
diagnostics, or caring about quality visualizations, or caring about
communicating the intuition behind a model. Those are just the plausible
deniability excuses proffered to try to win the battle over which faction's
authority gets center stage.

I don't require that anyone have a cross-domain consistent view of the
specific definition of interpretability in order for my claims to be a viable
explanation, and so it is perfectly compatible with the paper. The paper looks
at the on-the-ground specifics used in the arguments over interpretability,
meanwhile politics explains the behind-the-scenes motivations of _any_ of
those specific choices.

The _Moral Mazes_ chapter _Dexterity with Symbols_ is a good place if you want
to read on the evidence about how this is carried out in management
bureaucracy. It has actually been formally studied in some interesting ways.

------
gmisra
Maybe the prevalent mental model regarding models is wrong? The real elephant
in the room is the idea that "all models" are similar, and therefore occupy
the same space regarding interpretability, forecasting, and other uses. Models
are designed to solve specific problems, usually within specific scopes.
Confusion, misinterpretation, and error occur when consumers of models don't
understand (or simply ignore) the limitations of these models.

Whenever I sit down to build a new model, or work with a new-to-me model, I
start with questions like:

\- What questions was this model built to answer? Be careful when asking other
questions, and know the limitations of those answers.

\- If it's been in use for a while, how effective has it been? How is that
efficacy measured? A highly accurate decomposition of historical behavior may
involve a lot of post-fact knowledge that has no predictive power, so be
careful interpreting cross-use results.

\- What are the ranges and distributions of the input variables? What are
their boundary conditions? This is not always applicable to all variables,
especially unstructured data, but when it is applicable, it is usually
straightforward.

\- What are the ranges of the model outputs? Do they have any boundary
conditions? Do boundary effects impact the situations in which the model can
be applied? For example, when working with a binned variable, often bins 0 and
n include less homogenous data than the rest of the bins.

\- How is model accuracy calculated? This is usually far less objective than
model constructors are willing to admit to. How does your error vary along
different dimensions? Which dimensional error do you look at more closely, and
why?

In my experience, the best way to reason about models is to work with a lot of
different models, and to be honest about their flaws. That learning generally
happens more efficiently, and more broadly, in the real world than in the
classroom. With the recent rise in popularity of applied models, we have lots
of inexperience modelers out there, so these growing pains are to be expected.

~~~
scribu
It seems like you're talking about human-created models (e.g. a manually-
constructed decision tree), whereas the paper's primary concern is machine-
created models (e.g. the weights derived mechanically by a neural network).

So the problem would be: how do you get the "modelers" (i.e. the machines) to
be "honest about the flaws" of the models that they generated?

