
Sometimes, requiring explanability is counterproductive - metahost
https://www.facebook.com/story.php?story_fbid=10156577899252143&id=722677142
======
6gvONxR4sf7o
Okay, I feel like a crazy person arguing against a luminary of the field, but
this is so misleading as to be disingenuous and make me trust the guy less.
He's throwing causality and explainability into the same bucket and arguing
against the need for the latter.

>We often hear that AI systems must provide explanations and establish causal
relationships, particularly for life-critical applications.

>Yes, that can be useful. Or at least reassuring.

>But sometimes people have accurate models of a phenomenon without any
intuitive explanation or causation that provides an accurate picture of the
situation.

It goes on to argue mostly against the need for intuitive explanations, not
the establishing of causal relationships.

>Now, if there ever was a life-critical physical phenomenon, it is lift
production by an airliner wing.

>But we don't actually have a "causal" explanation for it, though we do have
an accurate mathematical model and decades of experimental evidence.

The physical models we have are causal ones. The intuitive abstractions like
bernoulli's principle may not work, but analysis based on navier stokes sure
does. You plug your shape (cause) into the equations and see what forces
(effect) occur. That's causation.

>You know what other life-critical phenomena we don't have good causal
explanations for?

> The mechanism of action of many drugs (if not most of them).

Using an industry that's nearly synonymous with a randomized controlled trial
as a refutation for the need of a causal relationship is crazy talk. The
_mechanism_ may be missing, but the causal explanation is that via a series of
RCTs it's established that the drug _causes_ the effects.

I get that half of this is trying to go against a percieved need for intuitive
explanations, but it weirdly lumps causation in there.

~~~
uoaei
Explainability -> gradients

"How much does this input seem to confuse this output? What is the pattern
across inputs for how this model is systematically confused?"

Causality -> counterfactuals

"How would the outcome be different if _x_ was different? If I acted
differently, would I get a more favorable outcome?"

You're right to say these are two different things. They are.

And they're different _still_ from _interpretability_ , i.e., "What are the
explicit patterns that this model is seeking in the data?"

DL practitioners routinely mix up explainability and interpretability but I
would never in a million years have seen LeCun be so intellectually dishonest
as to lump causality in there with them.

~~~
joe_the_user
The thing is, I would claim that causality, explainability and
interpretability are all mixed together in human informal discussions of
various phenomena. As others on the thread have pointed out, Pearl's causality
isn't everyone's causality. A tension structure can disrupt our "common sense"
idea of what's holding up what but tensions structure doesn't seem at all like
a black-box, _unexplainable_ item. The way the article mixes the range of
these issues seems definitely wrong but I don't think that means the line
between all things is normally crystal clear.

------
arcanus
> Now, if there ever was a life-critical physical phenomenon, it is lift
> production by an airliner wing. But we don't actually have a "causal"
> explanation for it, though we do have an accurate mathematical model and
> decades of experimental evidence

This is argument is complete nonsense. Navier Stokes has a rigorous derivation
based on extremely high-fidelity assumptions, such as conservation of mass,
momentum, and energy. We understand these assumptions, and we understand
regimes in which using N-S would result in catastrophe (such as rarefied
gases, relativistic velocities, etc.)

Neural networks require data. The Navier stokes does not need to be 'trained'.
Deep networks have very little a-priori knowledge baked-in (from a baysian
perspective there are priors such as translation invariance that are
intrinsic). They are admittedly extremely useful, because they are high
dimensional (and so are universal approximators) and can be trained
efficiently.

Furthermore, you can develop an intuitive approach to many fluid flows. I can
provide a much better estimate of the drag profile for a given wing geometry
than an untrained person. No such analog is possible with deep nets, which as
significantly more opaque in terms of dynamics and non-linear response.

The only way his comments make any sense is if you assume he isn't talking
about physical models, like Navier stokes, and instead consider turbulence
models, such as RANS or LES. These are parameterized models, and are used for
turbulence modeling. They also have little physical intuition. However, this
is not the same as saying we do not have high confidence in physical-based
systems such as Navier stokes.

Source: I have a PhD in CFD and several ML publications.

~~~
ericjang
Yann was not making a statement about deep nets' ability (or lack thereof) to
fly a plane in a way that matches expert design.

He's making the point that the ML field's obsession with causal inference (and
causal discovery) is overrated, _precisely_ because our gold standards of
interpretable, safety-critical systems (airplane flight) are based on Navier
Stokes/CFD. Planes were made to fly based on empirical validation of these
models, long before we gained a more detailed understanding of how causality
(the equations and models themselves are time reversible, implying that they
contain imperfect knowledge of causality)

~~~
ssivark
> Planes were made to fly based on empirical validation of these models, long
> before we gained a more detailed understanding of how causality

And their success is so repeatable that if they fail once we make a Really Big
Deal out of it. I don’t think any ML model is close to that level of
engineering rigor, let alone deep learning.

Moving on to aerodynamics, we have a pretty good causal model and can simulate
the system given a pattern of boundary conditions. Further, some people (who
have studied aero/CFD) can even intuitively predict approximately what happens
(otherwise we would have a hard time designing planes!). It just so happens
that it’s not as simple as high school physics, and cannot be compressed in to
perfect+simple rules (trade off between those two).

Speaking in the context of time reversibility, you are being fast and loose,
and using the word “causality” in a sense that is irrelevant to the rest of
your comment.

------
gambler
_> Now, if there ever was a life-critical physical phenomenon, it is lift
production by an airliner wing._

I nominate this as the worst analogy of the year.

Airplanes are rigorously tested under the same conditions they will operate
in. AI _by definition_ is tested under conditions that are different from the
environment it will operate in, _because that 's the whole point of AI_ \- we
want algorithms that adapt themselves to novel information.

------
huyegn
I agree with Yann here ... I think the demand for explainability is like a
person asking for a "faster horse" when what they really need is a car.

When people ask for explainable models, what they really want (in my opinion)
is calibrated and robust _uncertainty estimates_ .

Good uncertainty estimates would let them know when to trust a model's
prediction and when to ignore it.

For example, a model trained to predict dog breeds should know nothing about
cat breeds, and there should be some way to quantify when it doesn't know!

I've been doing a review of techniques that are becoming more popular in this
area:

[https://blog.everyhue.me/posts/why-uncertainty-
matters/](https://blog.everyhue.me/posts/why-uncertainty-matters/)

~~~
ssivark
While uncertainties are useful, they are certainly not enough. Eg: I can
construct adversarial examples which are predicted incorrectly with high
confidence.

The underlying reason why high confidence is not enough is that even
strong/confident correlations could be misleading when seen in causal light —
a black box model trained to predict credit performance might be very
confident in rejecting loans for applicants from “poorer” zip codes and
approving those from “richer” zip codes — even though those are not actual
causes... therefore somebody could exploit the system by renting an address in
a rich neighborhood for a couple of months when taking out a big loan (the
analogue of adversarial examples).

~~~
huyegn
I stated that rather than explainability, for most use cases, people just want
_calibrated and robust_ uncertainty estimates NOT _low quality and
uncalibrated_ uncertainties estimates.

Your example points to models that provide low quality uncertainty estimates,
but that's not true for all deep learning models.

I believe it's these low quality uncertainty estimates that lead people to
look toward "explainability" as a solution, but for the majority of use cases,
I think people just want better uncertainty estimates so that they can "know
when they're model doesn't know".

There are techniques now to get higher quality, calibrated, uncertainty
estimates that don't suffer from the problems you mentioned and I've outlined
these solutions in my posted link above.

Additionally if you're interested, there is some nice recent research from
google on the subject:

[https://ai.googleblog.com/2020/01/can-you-trust-your-
models-...](https://ai.googleblog.com/2020/01/can-you-trust-your-models-
uncertainty.html)

and from oxford:

[https://arxiv.org/abs/1912.10481v1](https://arxiv.org/abs/1912.10481v1)

~~~
ssivark
Thanks for the links here and above; will take a look.

> calibrated and robust uncertainty estimates NOT low quality and uncalibrated
> uncertainties estimates.

Could you explain what you mean by “calibrated” and briefly summarize the
essential idea behind what allows the learning of robust uncertainty
estimates, if not a causal understanding?

If you haven’t already, look up work by Scholkopf, Janzing , Peters and co
(over the last decade) for a justification of why causal reasoning is exactly
what you want if you want to generalize across covariant earth/dataset shift
(which is basically what the Google blog post is about).

------
wnoise
I hate the comparison to how airfoils work. It's a single question that can be
answered at a variety of levels, but the basics of how they work is simple and
cannot be gainsaid: that they deflect air down.

All the complications come in the exact details of _how_ they deflect air
down. How much is lower than ambient pressure above the wing redirecting
slipstreams vs higher than ambient pressure below the wing doesn't
fundamentally change the answer, though those details certainly matter
(especially when designing a wing).

In contrast, even for a single "AI", how it responds differently to different
input is unlikely to be even remotely explainable by the same high level
principles, and it's not clear that the details don't matter.

~~~
jsmith45
Hmm... I'm certainly rusty on physics principles, but as far as I remember the
(flawed) classic bernoulli effect explanation of airfoil (assume equal travel
time ... pressure differential causes lift) does not really directly talk
about moving air down at all, and would hypothetically still work if all
downward air movement from the top of the wing were magically canceled out,
(and airfoil had an attitude of 0 degrees, so no downward air movement from
the bottom).

Now granted even with that magic, there would be downward air movement
relative to the airfoil, due to the airfoil rising, but that would be an
effect of the lift, not a cause.

Now I could well be overlooking something here, and having invoked magic to
violate conservation of momentum, this thought experiment is not rigorous, and
is discounting the fact that the equal time hypothesis is unfounded, and even
wrong. (But in reality the air above moves faster, so we would see a larger
effect than predicted, even before accounting for redirected airflow.)

Of course there is no magic canceling out any downward momentum of the air
from the top of the wing, and planes normally have a nose up attitude, so the
bottom of the wing also directly reflects air downward, so downward directed
airflow is definately also a cause of lift.

------
opless
I challenged LeCun about explain-ability of neural nets a few years ago, he
seemed to dismissed the need for it. As I recall his explanation was that the
weights & configuration of the neural net is enough to give you an equation
which is explanation enough. I'd link the post but it seems to have
disappeared.

LeCun is also heavily biased against Bayes.

~~~
lars
He's not wrong though. We know precisely how any given neural network works,
it's just not understandable to us.

The point is though, the functions we're trying to learn, like image
classifiers, have no responsibility to us to be understandable. In fact, the
brightest minds of several generations have worked hard on trying to write
down rules to do image classification, and they never came up with anything
that works.

There is a huge space of functions that are beyond what a human can
understand, where we can't write down rules to express the function. This is
precisely where we need to use machine learning to find the functions. Lack of
explainability is not a bug, the entire point of a neural nets is to find
functions that are beyond what we can understand.

~~~
opless
Lack of explainability is not a bug, it's a wholly missing feature.

There are a whole set of mathematical functions that are beyond the range of a
typical human mind to understand. But there are proofs that exist that explain
how they are correct.

I don't think it's a big ask for a new model to be able to justify its own
determinations.

Of course, that makes things slightly more difficult for all those in research
and those selling snake oil and everyone in between. But that's what the
difference between science and alchemy is isn't it?

~~~
lars
If I show you a picture and ask you if its a dog or a cat, you can answer the
question, but you can't tell me why. You can make up a story to justify why
it's a dog or a cat, but if someone tries to implement the procedure you give,
it won't work. If it did, we could do computer vision by implementing your
justification.

You are a black box AI. I can nevertheless trust you to classify dogs vs cats.

~~~
opless
No. I'm natural general intelligence with no safety features. One that has a
hallucinationary view of the world an a neural architecture tracing back a
couple of hundred of millions of years that continuously learns and has
redundant features, and has the ability of self reflection.

We're talking about algorithms that are based on a simplified neural
architecture, no redundancy, no self reflection and are still quite immature.

Nevertheless we're being asked to trust a black box AI, that you cannot
interrogate?

Yes, of course, what's the worst that can happen?

~~~
lars
It may well be that the human visual system is better for most tasks, but then
that is what matters, which system is better for the task. The presence of an
explanation doesn't matter. Neither system comes with an explanation.

On the other hand, if you place Magnus Carlsen against AlphaZero in a game of
chess, I will bet on AlphaZero. If however you reduce the complexity of
AlphaZero down to a level that it can produce an explanation I can understand,
I would instead bet on Magnus Carlsen.

Of course we should care about the quality of AI systems, but chasing a human
understandable explanation is just the wrong way to go about it, since it in
many cases necessarily limits quality of the decisions.

~~~
naresh_xai
Again the non-sequitur argument of an explainable model must be worse than a
deep learning system and there has to be a tradeoff.

You don’t need to reduce complexity to induce explainability. You just need to
decompose the function into smaller parts which you can understand.

Contrastive LRP for example is a Function decomposition technique for
explaining deep neural networks with high fidelity.

------
heyitsguay
I think the type of explainability Yann LeCun is describing in this post is
not the same as what people really want when they talk about explainable AI.
To use his example, I think most people would be fine with an AI that said
"we're doing X because that's what heuristic Y prescribes for the situation",
and getting from where we are now to something like that is the challenge.
FWIW i see a similar dialogue in biomedical contexts quite a bit, where
there's some cool but hard-to-trust work being done in data-driven image
restoration
([https://www.biorxiv.org/content/10.1101/236463v5.abstract](https://www.biorxiv.org/content/10.1101/236463v5.abstract)).
There's still a lot that could be done short of programs providing a priori
mathematical proof for all models.

~~~
buboard
> because that's what heuristic Y prescribes for the situation

people will keep asking "why" and keep escalating, but that's a flawed
approach to evaluating a model. every answer is literally constructed and thus
can't be trusted on its own. The best that an evaluator can do is to evaluate
the training dataset

------
omarhaneef
Interesting exchange in the comments:

Gaurav Mukherjee:

Well argued! Yes, it is true that we don’t have causal relationships for a
number of phenomena. But the absence of evidence is not the evidence of
absence. So corner cases of failure may exist in all these phenomena which can
cost lives. Does this mean that we halt the fast paced progress of AI research
or any other scientific pursuit? No! But leveraging empirical evidence that
can’t be fully explained in situations where lives are at stake, should
require a very high bar of regulation. Responsible scientists and engineers
agree that causality is important to understand and they do all they can to
understand how systems work. However there are likely many among us who do not
employ similar standards to the application of ill understood techniques. When
it comes to regulation, we must pay heed to the worst in us.

Yann LeCun:

Actually, there is no clear definition of causality in classical and quantum
physics, simply because the elementary equations of motions are time
reversible (if you also reverse charge and parity). For every phenomenon that
can occur, the same phenomenon can occur backwards in time (with corresponding
anti-particles).

Take a Feynman diagram where an electron and a positron annihilate to produce
a gamma ray photon. It can be interpreted as a gamma ray photon spontaneously
creating an electron-positron pair. It's the same diagram where time goes
right to left instead of left to right.

How can one possibly say that A causes B, when B could very have caused A in a
time-reversed view point?

Even worse, most physical phenomena have loopy causal graphs. Motion "causes"
friction. But friction limits the velocity of motion. Most differential
equations have coupled terms with loopy interactions in which quantity x
affects quantity y and vice versa. You rarely have y(t+dt)=f(x(t)) in physics.
More often than not, you have coupled equations y(t+dt)=f(x(t)) and
x(t+dt)=g(y(t)).

In these (frequent) cases, x cause y, but y also causes x. There is something
like that going on in fluid dynamics, which is why it's difficult to come up
with "simple" causal explanations.

Only when collective phenomena are considered does the "arrow of time" appear
to have a definite direction (with the 2nd law of thermodynamics).

Edit: para spacing

~~~
6gvONxR4sf7o
>Actually, there is no clear definition of causality in classical and quantum
physics, simply because the elementary equations of motions are time
reversible (if you also reverse charge and parity). For every phenomenon that
can occur, the same phenomenon can occur backwards in time (with corresponding
anti-particles).

This is super misguided. Causation doesn't need physical grounding. The usual
definitions are grounded in counterfactuals and interventions.

It's like defining the causal effect as the difference in potential lines of
code I write today, depending on whether I do or don't eat breakfast this
morning, but then arguing that under classical mechanics the universe is
deterministic so I couldn't possibly take an intervention to not eat
breakfast. Really, it's just another argument that counterfactuals are counter
factual, and therefore don't have a clear definition. But yeah, that's in the
name already.

Regardless, physical equations are pretty much all causal, because they are
stable under intervention. If I intervene to do whatever, they still hold.
That lets me _cause_ a change in friction be _intervening_ and swapping out a
material.

~~~
sjg007
I thought causality was a function of time by definition?

~~~
6gvONxR4sf7o
Not quite, especially since time is such a weird concept when you dig far
enough into the math. There are a bunch of competing definitions of a causal
change, but arguably the most conventional one is the difference between an
outcome if you do one thing versus another.

For example, looking at whether I write this comment now instead of getting
back to work after lunch, you could imagine that I'm more inclined to slack
off after a heavy meal. We have two hypotheticals. Scenario 1) I eat curry for
lunch. Scenario 2) I eat salad for lunch. In scenario 1, I am a bit sleepy and
log on to HN and write this. In scenario 2 (hypothetically) I just get right
back to work and this comment never exists. The causal effect (hypothetically)
is that having curry instead of salad for lunch made this comment exist.

Compare that to the effect of the comment I wrote before lunch. We can still
talk about the causal effect of lunch on what I wrote before lunch. Assuming
that physics works the way I think it does, the causal effect is that nothing
changes, but it's still a thing you can formalize. You could imagine an
episode of star trek involving time travel, where my actions today do affect
things yesterday. Starfleet statisticians could still run time-reversed
randomized trials, they'd just have to be really careful about their
experimental designs.

~~~
ssivark
I like your original comment further up, but disagree with the following bit:

> We can still talk about the causal effect of lunch on what I wrote before
> lunch.

To the extent that the situation occurs in the physical universe, that is
meaningless. The future can never be a causal parent of the past. That says
there is much more to the typical causality question than just the direction
of time.

The difference between the “time reversibility” in physics and the typical ML
example is that once you’ve decided on something as the effect, the other
thing must be the cause. So the problem is quite simple if you’re not trying
to figure out the nature/direction of time, and especially not with
microscopic physics.

Anyone curious about the relation between time reversibility in physics and
causal reasoning could look at Scholkopf+Janzing 2016 for an interesting line
of thought.

Somebody should tell Yann Lecun that a little knowledge is dangerous.

~~~
6gvONxR4sf7o
>The future can never be a causal parent of the past.

That's a testable statement. The reason it's testable is because we can define
a hypothetical causal effect of the present on the past, and then do
experiments and show that it's always zero. I only mean that we can talk about
it, not that we can show it to be anything but a non-effect, so I'm not sure I
see where we're in disagreement.

~~~
ssivark
Ah, guess I misinterpreted “can still talk about” in the context of response
to its parent comment.

------
joe_the_user
From article:

" _How does lithium treat bipolar disorder?_ "

The thing is, that this isn't a question that's _insignificant_. There has
been quite a bit of debate whether various drugs reverse the basic process
happening with a given psychiatric disorder or whether they provide a
different change which allows a person function. The phrase "chemical
imbalance" has been based on the supposition that various drugs that change
brain chemical distributions "cure" various conditions - but the question
these drug are directly reversing a condition or whether they are adding
something more seems important, even if we assume the drugs are broadly useful
for helping people function in society.

------
screye
Yann is undoubtedly a pillar of the deep learning era of AI. But, I strongly
disagree with what he is saying.

Firstly, the Navier-stokes equations existed before mechanized flight was
discovered. It was not explicitly invented to reason about flight, in that way
that some of the deep learning "theory" is being projected. It also worked
perfectly (in sub-sonic speeds) for almost every situation involving flight.

ML has time and again proven to not be internally consistent, with
contradictions on nearly every corner.

Batch norm was "theoretically" thought to be an essential part of neural
networks, until suddenly it wasn't. Drop out was considered essential to avoid
overfitting, until it wasn't. We still do not know if an information bottle
neck is good or bad. ADAM's whole formula was incorrect and no one realized
that for 6 years.

> The mechanism of action of many drugs (if not most of them). > An example?
> How does lithium treat bipolar disorder?

This example does more to disapprove his point that approve of it. Medical and
nutritional sciences are among the least understood fields out there, with
some of their "fundamentals" flipping on their head over the last 50 years.
The only reason these "sciences" continue being used is because medicine is
essential. A reasonably effective solution with side effects is still better
than dead people. It is a begrudging compromise, not a example to be emulated.

AI is not essential. AI imposes on your life without consent. AI will soon be
ingrained into every single aspect of your life.

Yann seems to be conflating explainability with causality. Explanaibility can
also mean fully observed correlation. Explainbility can mean predictable and
reproducible behavior of ML models given a hypothesis. Explanability can mean
the ability to ascertain if the change in model performance was because of the
hypothesis or a way to exploit an unintended aspect of the data/model
architecture.

Explainability fundamentally allows ML researchers to make strides in the
field in a meaningful way, and not blindly throw different computational
structures at thousands of GPUs and let luck of the draw decide what works and
when.

Looking back at ideas such as Transformers and ResNets, there was literally no
way for the authors to guess that this new computational structure would
revolutionize the field. It could easily have been an idea someone else tried
or rotted in someone's TO-TRY list. Explainbility and some theoretical logic
around NN development would allow for a systematic way to go about research in
the ML community as whole. That's unlikely to happen, but I would rather see
people strive for it than not.

------
naresh_xai
When a person’s claim to fame and research is dependent on ignoring
explainability and causality in research, he will ignore it to the best of his
means.

To him, all the precursors to clinical trials (selection of a molecule from
restricted set of molecules which satisfy certain causal criteria) then
followed by extensive multi million dollar experiments to safeguard the last
1-2% of uncertainty is equivalent to a barely preprocessed neural network.

I mean let’s ignore the basic principles of pharmacology and medicine and just
run every possible molecule through humans since that is a valid approach
according to him.

------
tomrod
For those interested in a great read on why causality really does matter, give
"The Book of Why" a read. I'm not convinced Judea Pearl's modeling approach is
the most rigorous, but it does a clear and convincing job putting causality
(esp. w/ data fusion!) at the heart of modern systems (including ML).

For those on a more mathematical bend, check out "Causality" by the same
author or "Causal Inference for Statistics, Social, and Biomedical Sciences"
by Imbens and Rubin

------
carlosdp
Another more basic example is gravity. We have a statistical intuition of
gravity, we depend on it being there and acting in a consistent way for almost
everything in physics. But we don't have an accurate understanding of how
gravity works or why it's there. We have theories, but nothing proven.

But it's so easy to prove gravity's properties (on Earth) by consistent
results with experimentation, that it's literally a science project in every
elementary school.

------
krishnagade
It depends, while Yann Lecun has an interesting point of view.

The argument for explainability depends on the risk of harm from an AI model
decision. Explaining why airplanes fly is moderately interesting but why did
the 737-max crash is much more interesting and needed. While the former is
probably only needed for people studying aerodynamics, the latter is meant for
passengers, regulators, airlines, governments, etc.

Here is a tweet thread we posted in the past:
[https://twitter.com/krishnagade/status/1182317508667133952](https://twitter.com/krishnagade/status/1182317508667133952)

------
haecceity
In re airplanes, deflecting air down is a perfectly intuitive explanation.
Change in momentum of air means change in momentum of wing in opposite
direction by the third law.

------
buboard
he s probably correct. A lot of what we call "human explanation" is in reality
a rationalized version of a hunch, but not necessarily the best representation
of it. At most situations, some part of the brain crosses a threshold and
causes a decision or action, but it's rather rare that humans can explain that
action correctly. It's also rare that causation can be established from
unambiguous temporal order (e.g. clouds/rain); in most cases our
rationalizations are post-hoc.

We get a glimpse of it from recent language models: It's become rather easy to
start blurting out language that is convincibly and comfortingly coherent, and
it can be nudged to point to one direction or another. that doesnt mean it's
true

------
RookyNumbas
I'm surprised he made the drug analogy. It takes decades and hundreds of
millions dollars to be able to test a drug on humans. I'm curious if he thinks
we need a similar safeguard for all AI-human interactions.

~~~
PaulRobinson
It takes decades and hundreds of millions of dollars in order to build a
statistical argument that explains likely effects, side-effects and risks of
prescribing that drug to the population at large.

It takes that long and costs that much because there is no safe way to do this
with human physiology other than to carefully, slowly and expensively try it
and measure the consequences.

We can take any AI system and model its behaviour with statistical certainty
much more cheaply and quickly, and be more confident in its future behaviour.

Remember, AI and ML are fancy words for some form of a regression to a mean
more often than not. If we run hundreds of millions of tests of such a system
in simulation for a very wide range of contexts/inputs (which is cheap to do),
we can have a much higher degree of confidence in a short space of time around
behaviour of that system than we will for any drug test, even if we still
don't fully understand the causality.

~~~
naresh_xai
If you were ever involved in the drug discovery process, then you would know
that statistical evidence through clinical trials is only for the last ounce
of drug testing. Out of 10,000+ drug molecule candidates only ~ 5 molecules
get selected for clinical trials. A lot of molecules are removed as candidates
for toxic components/ solubility issues/ chemical stability issues etc.

So even when we do not understand the precise impact of drugs on humans and
there is no safer mechanism to test, we leave only 0.5% of candidate molecules
to empirical/statistical evidence in the form of clinical trials.

On the other hand if drug discovery was treated as a pure AI problem, we would
have thousands of unverified and unsafe molecules in clinical trials.

Causal principles get us to 99.5% of the way in drug discovery. Unfortunately
not so in AI.

~~~
PaulRobinson
I'm not involved in the drug discovery process, but I can imagine that
manufacturers would be interested in helping to use AI to deal with the funnel
much more effectively to get to those ~5 molecules much more efficiently.

You're still left with double-blind trials and having to get large sample
groups to try those molecules though.

And it's for that reason that drug discovery is always likely to be quite
slow, complex and expensive - the efficiency gains will be pushed towards the
top of the funnel to make new ideas reasonable to explore, I would imagine.

My point was that when you're not dealing with human physiology and instead
dealing with problems that are more tractable through AI - i.e. using
regression to tune algorithms through patterns in data - you are going to get
quicker and more impactful returns without the same complexity.

And - critically - it's OK to often trust the AI solution you have without
understanding causality. If you later find it's doing something odd that is
undesirable, you can use that data to help tune the algorithm again without
having to understand the causal relationship.

Put another way, you can teach an AI to get better without necessarily
understanding the subject completely yourself.

~~~
naresh_xai
Let’s see. Robotics problems were claimed to be tractable through AI. However,
a large majority of robotics solutions today are 90% derived through control
systems (which follow some degree of causal analysis) followed by AI to
optimize the last few percentages if possible.

Finding something odd for an algorithm (especially a deep neural network) is
hard because they fail in just so many ways. For example, lenet for mnist
almost always gives high confidence predictions for random
tensors(torch.randn). Most imagenet models fail in the presence of just 20-30%
salt and pepper noise. (Both of these are problems solvable through simple
preprocessing techniques)

~~~
naresh_xai
Not to mention the fact that most models are trained without a background
class and tend to give overconfident predictions on out of distribution
samples.

------
leereeves
Yann LeCun and others debated this for an hour at NeurIPS 2017.

[https://www.youtube.com/watch?v=93Xv8vJ2acI](https://www.youtube.com/watch?v=93Xv8vJ2acI)

There's a lot of good points in there from both sides, but what really stuck
with me is that given the choice between an explainable model and a black box
model that works better (more accurate predictions), most people choose the
black box model.

~~~
naresh_xai
Which is strongly misleading tradeoff. For a ton of tasks, deep learning
methods are no better than white box regression or tree ensemble methods.

And there is no reason to expect that a deep learning model has to be
unexplainable. He’s putting up a mystical interpretability vs accuracy
tradeoff which does not exist.

There is a ton of research out there making deep neural networks (slightly to
significantly interpretable)

------
xaryk
Not all explanations are causal. The explanation literature in the philosophy
of science goes pretty far back, but here are some of the highlights:

The Deductive Nomological Model (Hempel and Oppenheim, 1948) tries to explain
a phenomenon using a deductive argument where the premises include particular
facts and a general lawlike statement (like a law of nature) and the
conclusion is the thing to be explained.[1]

The Statistical Relevance Model (Wesley Salmon) attempts to fix some
shortcomings in the DN model that allowed explanations using particular facts
and general laws that were not at all relevant to the phenomenon being
explained. The idea is that you can explain why X hasn't become pregnant by
saying that X has taken birth control, and people who take birth control do
not become pregnant, and that would fit the DN model, but this explanation is
not statistically relevant if X is male.[2]

Unificationist accounts (Philip Kitcher) seek to unify scientific explanations
under a common umbrella as was done with, e.g. electromagnetism. If it is
possible to have a unified theory of something, each element becomes more
explainable based on its position within that unified theory [3]

pragmatic and psychological accounts tend to fit more closely with the kinds
of rationalizations that we've seen as some explanations of AI. They can be
fictional, but they don't have to be [4]

IMO we don't currently have an adequate account of explanation within the
philosophy of science that works for deep neural networks. This is what my
dissertation research focuses on.

[1] [https://en.wikipedia.org/wiki/Deductive-
nomological_model](https://en.wikipedia.org/wiki/Deductive-nomological_model)

[2] [https://plato.stanford.edu/entries/scientific-
explanation/#S...](https://plato.stanford.edu/entries/scientific-
explanation/#SRMod)

[3] [https://plato.stanford.edu/entries/scientific-
explanation/#U...](https://plato.stanford.edu/entries/scientific-
explanation/#UniAccExp)

[4] [https://plato.stanford.edu/entries/scientific-
explanation/#P...](https://plato.stanford.edu/entries/scientific-
explanation/#PraTheExp)

------
t_serpico
As a somewhat outsider to deep learning, intuitively it seems to be true that
if you were able to demystify the black box, then it would be easier to
improve your models (as you understand where it succeeds and fails and why).
From this perspective, explainability would be incredibly productive.

~~~
b3kart
I am not convinced the explanations we’d get from these models would be
interpretable enough to drive their conceptual improvement. Deep nets just
don’t learn nice interpretable features, nothing in the training objective
makes them do so. E.g. they make use of really dubious features [1], and might
not actually learn “hierarchies” of features as we thought they were [2].

[1] [https://arxiv.org/abs/1905.02175](https://arxiv.org/abs/1905.02175)

[2] [https://arxiv.org/abs/1904.00760](https://arxiv.org/abs/1904.00760)

~~~
naresh_xai
Or you can actually visualize and quantify impact of aggregates of meaningful
features within networks using the methodology described in NetDissect or in
TCAV. But of course, that casts doubt into a lot of claimed mechanisms and
tons of ML research as I have found out.

------
halayli
I wonder where/if incompleteness theorem fits in this picture.

[https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...](https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems)

------
anthony_doan
Yeah AI world have a loose definition of explanability and interpretablity.

I also see this very dogmatic mindset that Deep Learning will do prediction
and interpretability.

What is stopping you from building two models? a regression statistic model to
do interpretability/explanability and another deep learning for prediction?

Like each coefficient in a regression have a t-test for significant in
correlation with response. You don't have something like that in deep
learning. Also I've seen many MLer use logistic regression as a classifier and
ignoring the probability aspect like the Titanic dataset highlight the
different mindset between statistician and ML. ML often will see this as a
classify problem dead or not dead. Statistician will phrase to "What's the
probability of this person dying with these covariates?"

You know why this matter? It really matter in health/medical/social science.
Often time inference is what they want and they need to know what affect your
health not just shoving in tons of data and covariates/features. Not only that
you many not even have enough data for these data hungry ML models.

Another example is biostatistician figure out threshold between the benefit of
taking an invasive procedure versus not taking it. We figure it out but giving
a percentage and the doctor and experts will tell you where the threshold is,
20%, 40%? It's certainly not 50% that many MLer do.

> We often hear that AI systems must provide explanations and establish causal
> relationships, particularly for life-critical applications. Yes, that can be
> useful. Or at least reassuring.

To me this just an excuse to not learn statistic. He should really look into
Propensity modeling under Rubin-Neymer causality model. This is what statistic
is going into after regression for observational data.

With all the criticism I have for ML. I think it's just the mind set. I think
the ML algorithms have a place and they're very good in certain domain such as
NLP and computer vision. But to act as if they're the end all be all when
statistic models have been there and use extensive in biostatistic and
econometric fields is just hubris and ignorance.

While ML is making excuses for causality. Econometric and statistician are
working to build causality model. IIRC econometric is doing structure equation
while statistician are going for Rubin-Neyman model. There is debate on which
model is better but that's ongoing we'll wait and see from all the research
papers.

------
krnsll
Would be interested to see what Judea Pearl says about this. Not on twitter
any more but back when I was, I recall him regularly tweeting about causality
and AI's shortcomings in that regard.

------
monadic2
How do you validate the model without understanding where you might find
problematic results? That doesn’t require a narrative, maybe, but it does
require not using black-box-only testing.

------
gatherhunterer
Hopefully a mod can change the link.

> Please submit the original source. If a post reports on something found on
> another site, submit the latter.

~~~
albertzeyer
This is less about the article linked by Yann, but more about what Yann
writes. He basically uses the argument made in the article, and adopts it to
make a similar argument about AI. And that this argument about AI is the
interesting content here (although the linked article is also interesting).

------
bordercases
This is true for the strict Pearl-style form of causation.

The concept of what counts as a causative explanation can be more expansive,
and it varies between disciplines. See the work of Nancy Cartwright.

[https://www.researchgate.net/publication/30527010_What_Is_Wr...](https://www.researchgate.net/publication/30527010_What_Is_Wrong_With_Bayes_Nets)

TL;DR we've been explaining "causes" without Bayes Nets for awhile, Bayes Nets
unsubtly disregards the common-sense logic scientists use for their practice,
including the way that explanations tend to be qualified by context.

------
pesenti
Einstein had a great quote: "The most incomprehensible thing about the world
is that it is comprehensible." Well, turns out, a lot of the world is not
comprehensible. And that's what made Einstein irrelevant for the latter part
of his life.

Our overestimation of the comprehensibility of the world may very well be some
version of the Drunkard's search principle. We are much more likely to know
about what's comprehensible than what's not.

~~~
drcode
As far as I know, a lot of the criticism Einstein leveled against the
foundations of quantum mechanics later in his career are more recently being
revisited... instead of being irrelevant, some of his later work may simply
have been too far ahead of its time.

------
Rumudiez
The word the author was looking for when they wrote "explainability" is
actually "explicable," as in something that can be explained, and is converse
to the more commonly used word "inexplicable."

~~~
thanatropism
Good point.

Compare with efficiency vs efficacy.

------
pesenti
For those not on FB:

Link: [https://www.scientificamerican.com/article/no-one-can-
explai...](https://www.scientificamerican.com/article/no-one-can-explain-why-
planes-stay-in-the-
air/?fbclid=IwAR0-S3nl5P0-vKUsr0oN5eeXmTVPTHMRZ9L_5bDqQStD-7uEHdukXC5JR6o)

"We often hear that AI systems must provide explanations and establish causal
relationships, particularly for life-critical applications. Yes, that can be
useful. Or at least reassuring.

But sometimes people have accurate models of a phenomenon without any
intuitive explanation or causation that provides an accurate picture of the
situation. In many cases of physical phenomena, "explanations" contain causal
loops where A causes B and B causes A.

A good example is how a wing causes lift. The computational fluid dynamics
model, based on Navier-Stokes equations, works just fine. But there is no
completely-accurate intuitive "explanation" of why airplanes fly. Is it
because of Bernoulli principle? Because a wing deflects the air downwards?
Because the air above the wing want to keep going straight but by doing so
creates a low-pressure region above the wing that forces the flow downwards
sucks the wing upwards? All of the above, but none of the above by itself.

Now, if there ever was a life-critical physical phenomenon, it is lift
production by an airliner wing. But we don't actually have a "causal"
explanation for it, though we do have an accurate mathematical model and
decades of experimental evidence.

You know what other life-critical phenomena we don't have good causal
explanations for? The mechanism of action of many drugs (if not most of them).
An example? How does lithium treat bipolar disorder? We do have considerable
empirical evidence provided by extensive clinical studies.

This is not to say that causality is not an important area of research for AI.
It is. But sometimes, requiring explanability is counterproductive."

------
datastoat
If Facebook lawyers understood the implications of LeCun's argument, they
wouldn't be happy!

There are two types of explanations here: (1) why did the data come to be as
it is, (2) why did my ML make the prediction it did.

Science looks for the answer to (1), and causal models are a great way to
think about it. Science and engineering, when they go hand in hand, build a
machine by saying "Here is data, let me do science to understand nature's
underlying laws, and my machine shall be based on those laws". The machine is
inherently explainable because it's based on scientific laws.

In the ML world, we can bypass the "learn scientific laws" part, and jump
straight to "build a machine based on data". So the best answer to (2) has got
to be "my ML made the prediction it did because of its training data". As
Pearl said, ML is just curve fitting, so the only way to "explain" a ML
prediction is to say "here are the points that the curve was fitted to".
Prediction is just reading a value off the curve. Think the machine is biased?
Look for bias in the training dataset! Think the machine is inaccurate? Look
for sparsity or conflict in the training dataset!

So the consequence of LeCun's distinction is that, when the GDPR calls for
explainability of ML decision making, it is really calling for sharing of the
training data. Facebook, watch out!

------
Antwan
This guy is a total genius. Shame he doesn't care more about who he is working
for and what his researches are used for.

