

Simpson's paradox: why mistrust seemingly simple statistics - waldrews
http://en.wikipedia.org/wiki/Simpson%27s_paradox

======
calambrac
This should be part of a required reading regimen for anyone about to post yet
another 'Bullshit Study Reveals Whimsical Quirk' article.

~~~
waldrews
More importantly, it ought to have been part of the required reading regimen
for the authors of said study.

~~~
po
Even more importantly, it ought to have been part of the required reading
regimen for every single high school student.

------
orangecat
This also comes up in immigration studies. Say you have two countries A and B.
A has an average income of 5, and B has an average income of 20. If a resident
of A earning 10 moves to B where he earns 15, the average income of both
countries goes down even though total income increases.

------
tjic
This is fascinating.

OTOH, it depresses me - we try to do a fair bit of analysis on surfing
patterns, A/B testing features at <http://SmartFlix.com>, looking at average
basket size, etc.

...and now I've got to worry that I've got "confounding variables". Maybe
feature X decreases check out rates ... but actually increases check out rates
both among male and female customers....

ARGH. My head hurts.

~~~
waldrews
The good news is that when you do A/B testing, you can create a properly
randomized experiment, where variations of e.g. the number of men/women
assigned to the A and B group are accounted for as part of the sampling error.

Where this stuff really gets you is in retrospective data analysis, where with
the right choice of would-be confounding variables you can pretty much argue
both directions on any question.

~~~
trapper
The problem that I have seen in these frameworks that handle A/B testing is
that they ignore the basic laws of statistics. All groups are different given
large enough n. That's what p is about, do you have enough data to tell the
means apart. The closer the means, the more data you need. What you really
care about is how large the difference is. That's called effect size, which is
really just how far apart the means are in divided by a combined standard
deviation for both groups.

I certainly wouldn't be making rash decisions on data without a good effect
metric.

------
prpon
In the example of editing wikipedia it mentions: "This imagined paradox is
caused when the percentage is provided but not the ratio."

We see this so many areas. China's GDP is growing 12% every year and the US
has grown just 2% type of arguments.

I haven't been able to improve on my yearly salary raise of 300% in the last
15 years when I went from 5$/hr to 15$/hr

~~~
waldrews
Just comparing percentages off different bases isn't quite what the paradox is
getting at, but it's also a fine approach to representing data in misleading
way. There's a great collection of such less technical tricks in the book _How
to Lie with Statistics_ by Darrell Huff.

