
Statistically Controlling for Confounding Constructs Is Harder Than You Think - gwern
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719
======
Houshalter
This is really interesting. Trying to summarize the idea of the paper using
their example: If people swim more on days when ice cream sales are high, one
might conclude that ice cream causes swimming. But if you have information
about the temperature, you can fit a statistical model that predicts swimming.
The model will learn that temperature predicts swimming perfectly, and the
feature of ice cream isn't used at all. So you could say after controlling for
temperature, ice cream no longer correlates with swimming.

But what if your temperature estimate is noisy or imprecise? For example, if
you only have information if the temperature was hotter than 75 degrees, but
not exactly what it was. Then this trick no longer works. The temperature
variable predicts swimming a lot, but not completely. So ice cream sales are
still useful, and so still correlate with swimming after controlling for
temperature.

I wonder if the way to solve this is using bayes nets. You could test a model
where the variables "swimming", "ice cream sales", and "observed temperature",
are all caused by some unobserved third variable. This model should fit the
data better than alternative models, like ones where "ice cream sales" cause
swimming and temperature. But simple linear models can't represent
dependencies like this.

~~~
nkurz
Perhaps you know it, but there's another classic example involving ice cream.
It was thought in the 1940's (by the public, by doctors, and by scientists)
that ice cream consumption caused children to contract polio. More ice cream
is consumed in the summer, there are more polio cases in the summer:
[https://theglyptodon.wordpress.com/2012/08/21/polio-
caused-b...](https://theglyptodon.wordpress.com/2012/08/21/polio-caused-by-
ice-cream/)

~~~
Noseshine
My personal favorite:

[http://tylervigen.com/view_correlation?id=1597](http://tylervigen.com/view_correlation?id=1597)

US spending on science, space, and technology

    
    
       correlates with
    

Suicides by hanging, strangulation and suffocation

------
ivan_ah
I always wondered what exactly people mean when they say “controlling for ...”
as if somehow they can tell apart cause A and cause B for the effect C.

It turns out if you can measure A and B without noise, the whole "controlling
for..." business is legit, but if you only have noisy estimates A' and B' of
the actual A and B, then the "controlling for..." technique used by many
researchers does not work very well at all.

interesting quotes:

> _The present results demonstrate that, for a very common class of
> incremental validity arguments, such a strategy runs a high risk of failure.
> The scope of the problem is considerable: literally hundreds of thousands of
> studies spanning numerous fields of science have historically relied on
> measurement-level incremental validity arguments to support strong
> conclusions about the relationships between theoretical constructs._

> _We conclude that many previous findings in the social sciences are at high
> risk of having concluded that two constructs are distinct when they may not
> in fact be so._

Note how polite the authors are. They don't say it straight up that many
(most?) multiple regression results are bullshit, they just say "at high risk
of being wrong."

~~~
Noseshine
I think that as long as one remember that a lot of words used in statistics
have very different meanings from how one would use them in normal life one
should be fine. "Controlling for" is one such term, "explains" is another.

Of course, especially once study result make it into mass media all
statistical meaning of the words used is completely lost and reinterpreted
from the point of view of normal-life usage.

~~~
viraptor
Could you say how "explains" is different in statistics and normal-life? I
know what it means in statistics, but they're actually pretty close, so I'm
not sure how it could lead to a non-trivial misunderstanding.

~~~
Noseshine
Statistical "explains" is just that. It is based purely on the model _and
numbers_. Try this: Create statistical "explanations" for things like Newton's
laws, or for everyday explanations like why the uncle didn't go on vacation
this year.

When you get a statistical "explanation", for example in a linear model,
factor X explains 50% of the observed variation, you never know if that is
actually the factor - or if it's random, or if it's confounded by an unknown.

"explains" in statistics is meant to be used within the context of your model
equations and numbers.

Even if you get causation by experimenting and controlling the variables and
don't just have correlation, who would be satisfied with a statistical
"explanation"? You still won't know if there isn't a common factor that you
just don't know about yet.

Same with statistical significance: It is useless unless someone who actually
knows the field checks how/if the model you have relates to the real world.
It's not enough to show that everything works fine within your model - you
have to know if you have the right model, which requires knowledge outside of
statistics. It's just a tool (statistics), by itself it means as much as naked
numbers.

~~~
Noseshine
...or in other words (can't edit what I wrote any more):

Statistics can tell you _that_ there is a connection. It can't tell you _how_
(it works).

