
The Most Dangerous Equation [pdf] - jfriedly
http://press.princeton.edu/chapters/s8863.pdf
======
guylhem
Absolutely totally true.

Ignore the laws of statistics at your own peril.

Knowledge of statistics is the best investment one can make with one's time.
A/B testing is the tip of the iceberg - a simple practical application, which
can be exploited much better with statistics.

Learn about statistical distributions - Bernouilli, Binomial, Normal, Poisson
and Hypergeometric at least, then Chi distribution to grasp _real_ tests like
chi2 goodness of fit, 2 way, calculating intervals - and you will see so many
possible daily applications and avoid so many pitfalls (such as the ones
outlined in the article)

BTW I saw in another comment something about significance. DeMoivre is not
directly related to significance- it only means that if you know the
population standard deviation, the standard deviation of any sample extracted
from the population will depend on the size of the given sample - ie smaller
samples will go below and above the expected value much more often

Consequently, if you try to deduce the SD of a population using a sample, the
bigger sampler will give the best results (ie smaller or more accurate
intervals)

~~~
forrestthewoods
If someone doesn't know much about statistics what is your recommended source
to learn from?

~~~
guylhem
I love stats, but I do not consider myself a professional statistician, so
maybe someone else can give you better suggestions.

I'd just say I understand the basic ideas such as DeMoivre or the Central
Limit Theorem, and it helps me a lot.

First I'll assume you have a basic understanding of probabilities (odds,
dices, cards, etc).

If you don't yet get probabilities, try the CK-12 books probabilities and
advanced probabilities book - free on the kindle, and easy to read :
[http://www.amazon.com/CK-12-Probability-Statistics-Course-
eb...](http://www.amazon.com/CK-12-Probability-Statistics-Course-
ebook/dp/B00831TRPC/ref=pd_sim_kstore_6)

After that, my recommendation is to study the distributions suggested - from
Bernoulli to Hypergeometric - and any source you "understand" will do.

The important thing is not the source, but to understand how these things work
together, how they "articulate" - i.e. why taking a bunch of samples that
follow any distribution will get you something that follow a normal law (LLN,
CLT, etc) - even if the law they follow has a big hole in the middle, that'll
where the mean of the normal law will be. Or under which conditions you can
replace a law by another law, etc.

Then it's a good time to learn what moments do - how they shape the graphs you
get. After that, you can try intervals - calculate intervals given a
population parameters to see how a sample can predictably differ, then from a
sample of a given size how you can estimate the population parameters.

After learning all that, to bind all this knowledge I'd suggest the free
courseware on MIT 15_075 (even reading only the slides online on
[http://ocw.mit.edu/courses/sloan-school-of-
management/15-075...](http://ocw.mit.edu/courses/sloan-school-of-
management/15-075j-statistical-thinking-and-data-analysis-
fall-2011/index.htm))

I've recently "refreshed" my knowledge of statistics, and used the slides from
15 075 as a base. They get to the point and give a better mathematical
understanding - something important to build your knowledge on a solid base
after you understand how the things work together and what to go down the
rabbit hole.

The course suggests the Tamhane and Dunlop book (which I haven't purchased yet
but which is on my buy list) ; some other people recommended it to me for the
demonstrations - I did the E(S^2n) E(S^n-1) by hand and I would love to see
the proof for the Chi2 stuff, because I usually understand better after I see
or do the demonstration.

Regarding Chi2, "Introduction to business statistics" has a great chapter #13,
giving practical application, but I strongly suggest you understand the basics
first - it's too easy to make mistakes with statistics.

Yes I don't fully trust myself with a tool as powerful as statistics - it
takes a professional - but even with my limited understanding, I can see the
value it provides, the warnings it gives (ie the article read like some basic
logical stuff, but then I realized it wouldn't have been _that_ obvious if I
hadn't known basic statistics.)

------
JDDunn9
While statistics are certainly abused an misunderstood, I don't think that
small sample sizes are the worst problem. The media usually reports margins of
error, and most people know small samples may not represent the population.

I think a much larger problem is in the underlying assumptions that are made.
For instance, assuming that an experiment on animals can be applied to humans
(sometimes it can, sometimes it can't). These can be more nuanced and much
harder to detect than a simple math error.

Also, the importance of truly random sampling is not emphasized enough. Even
medical researchers are guilty of using international cluster sampling to make
generalizations about the population. Overlooking sources of bias like
geography, culture, lifestyle differences, etc.

~~~
kylemaxwell
The media report margin of error but not confidence interval, which is highly
important. If you have a 95% confidence that the poll has a 3% margin, that's
a very different statement than 80% confidence.

------
grannyg00se
Excellent. Reminds me of the lecture describing that "the greatest shortcoming
of the human race is our inability to understand the exponential function"

<http://youtu.be/F-QA2rkpBSY>

~~~
wes-k
Thanks for sharing this! Albert Bartlett does a wonderful job explaining the
consequences of sustained growth rates.

I find overpopulation to be one of the greatest concerns that we are not
adequately dealing with. Albert's bacteria in a bottle analogy made me think
of Elon Musk's attempt to bring human life to other planets.. will we not just
be expanding our time of growth? Maybe we should start choosing a way to reach
a population growth rate of 0% and see what countless problems are resolved.

~~~
foxylad
I didn't understand why everyone was wittering on about sustainability until I
read "Collapse" by Jarod Diamond. It illustrates how dangerous it is to ignore
sustainability, and how rare it is in human societies.

It seems to me that we have a choice:

1\. Figure out how to cut our population growth to 0%, and then educate
everyone on the planet as why they should do this instead of breeding to the
max. Our kids will be happier than us.

2\. Breed to the max until everyone on earth (and maybe Mars) is only just
hanging on by their fingernails, probably living extremely unpleasant lives of
violent competition and constant starvation. Our kids will be less happy than
us.

Sadly I think option 1 is unlikely due to the tragedy of the commons.

------
confluence
Ah hello law of small numbers, survivorship bias and fundamental attribution
error - we meet again. If you understand these 3 biases - you'll wonder what
kind of world you have actually been living in all this time.

I'm going to repost a my comment about this very concept as related to
startups from a while ago because I believe HNers will appreciate it - it's
from an article called "Startup School And Survivor Bias" (hope that's ok :)

Source: <http://news.ycombinator.com/item?id=4685042>

============================================================

Startups: never have so many understood so little about the statistics of
variance present in the outcomes of small samples.

People like to speak of 10x productivity, non-stop work and geniuses - but the
reality is much less interesting. A large number of small teams working on
many different problems will by definition have a great variance in outcomes
just by random extraneous factors (also known as the law of small numbers and
insensitivity to sample size).

 _> A certain town is served by two hospitals. In the larger hospital about 45
babies are born each day, and in the smaller hospital about 15 babies are born
each day. As you know, about 50% of all babies are boys. However, the exact
percentage varies from day to day. Sometimes it may be higher than 50%,
sometimes lower.

For a period of 1 year, each hospital recorded the days on which more than 60%
of the babies born were boys. Which hospital do you think recorded more such
days?

1) The larger hospital

2) The smaller hospital

3) About the same (that is, within 5% of each other)

56% of subjects chose option 3, and 22% of subjects respectively chose options
1 or 2. However, according to sampling theory the larger hospital is much more
likely to report a sex ratio close to 50% on a given day than the smaller
hospital.

Relative neglect of sample size were obtained in a different study of
statistically sophisticated psychologists_

\-- <http://en.wikipedia.org/wiki/Insensitivity_to_sample_size>

_> A deviation of 10% or more from the population proportion is much more
likely when the sample size is small. Kahneman and Tversky concluded that "the
notion that sampling variance decreases in proportion to sample size is
apparently not part of man's repertoire of intuitions. For anyone who would
wish to view man as a reasonable intuitive statistician such results are
discouraging."_

\-- <http://www.decisionresearch.org/pdf/dr36.pdf>

Taking lessons as gospel from these "10x" events is by definition foolhardy
and merely an extension of the bullshit pushed by the entire "Good To Great"
Jim Collins business book industry.

It's like taking lessons from survivors of the Titanic on how to survive the
sinking of a ship. It's quite simple - be a young female child with a life
vest and rich parents (or in startup land - a young upper-middle class male
living in California during a venture bubble, a cyclical investment in the
Valley with a convergence of secondary technologies, above average
intelligence and a college degree from a reputable university).

I have a personal rule with any kind of advice or explanation coming out of
anyone working in a "soft" industry - if it's vague - it's bullshit. All of
the advice given at these events are bullshit by this definition. So are many
other things - and yeah it doesn't preclude me from spouting it. Or using the
advice at my discretion.

But honestly - startup founders literally have no idea why things take off and
they have no idea why they win. That's why they have to keep pivoting - it
increases their luck surface area and their ability to gain traction - after
which they simply must hold on tight while surfing the wave.

YouTube was a dating site - didn't work - pivot - video traction - venture up
- ride.

PayPal was a Palm Pilot app - didn't work - pivot - traction - venture up -
ride.

Google sold corporate search - didn't work - pivot - copy PPC from Overture -
lever up - traction hits - ride.

Instagram - started with a location checking HTML5 app 2 years too early -
pivot - copy PicPlz and Hipstamatic - hit traction - lever up - ride.

Angry Birds - fail at hitting nearly every game in the past decade - pivot -
take a shot at the iPhone - hits traction - lever up - ride.

Of the startups that didn't pivot - they either skipped the pivot thanks to
previous side projects/companies or already had traction - and all they had to
do was lever up and ride.

I'm going to make this clear - there is absolutely, positively nothing wrong
with this - not at all - it is merely reality and not particularly unfair.

People stating pointless platitudes that success is due to things like "Be 10x
more productive", "Commitment" and "People, product, and philosophy" are
simply wasting their breath, other people's time and confusing what actually
happens. These things may or not be either actionable, predictive or
sufficient for success.

Here's my list of startup advice:

Be alive. Be male. Be young. Don't have health issues. Be born in America or
move there. Enter the cycle after a recession. Speak English. Enter a
growing/new field where the level of competition is low and so is the
sophistication of your competition. Surf cost trends down from expensive to
mass consumer markets. Work bottom up - on small things. Be of above average
intelligence. Have family support. Have a college degree.

Oh and most importantly of all: Get fucking lucky.

The hindsight/survivorship biases in combination with faulty causality and the
narrative fallacy will completely hose your thinking - so be careful.

More interesting stuff:

[http://en.wikipedia.org/wiki/List_of_biases_in_judgment_and_...](http://en.wikipedia.org/wiki/List_of_biases_in_judgment_and_decision_making)

<http://en.wikipedia.org/wiki/Black_swan_theory>

<http://en.wikipedia.org/wiki/List_of_fallacies>

<http://en.wikipedia.org/wiki/List_of_memory_biases>

<http://www.econ.yale.edu/~shiller/behfin/2000-05/rabin.pdf>

Disclaimer: Biases rule your thoughts and mine - this post is also subject to
both bullshit and biases (mostly bullshit - I do love that word). Think for
yourself.

~~~
jamiequint
Also interesting and on topic with this are any of Nicholas Taleb's books. I
highly recommend his new one: Antifragile
(<http://www.amazon.com/gp/aw/d/1400067820>)

~~~
treeder
I've been reading Fooled by Randomness by Taleb, great book, same topic.

------
6ren
The last section on sex differences is interesting. It explains boys having
greater variation in ability than girls by boys having only one X chromosome
(XY) while girls have two (XX).

This would be a neat theory, if girls somehow used an average of the two
X's... which seems compellingly logical, though the (current) theory is that
only one X is used, chosen at random. <http://en.wikipedia.org/wiki/Barr_body>

~~~
ef4
I don't understand how that fits with the existence of X-linked-recessive
diseases.
([https://en.wikipedia.org/wiki/Genetic_disorder#X-linked_rece...](https://en.wikipedia.org/wiki/Genetic_disorder#X-linked_recessive))

These seem to be well-known cases in which you really do get a phenotype
that's a function of both X chromosomes.

~~~
6ren
You're right, it doesn't fit. Now I recall that rates of red-green colour
blindness are exactly predicted by whether one or both must be defective (7%
for boys, .49% for girls). Both being active but the defective version having
no effect would explain the evidence, but doesn't fit the Barr body theory...
it would need to know which one to choose (and it wouldn't be "random").

~~~
russelldavis
X-chromosome inactivation _is_ actually consistent with X-linked "recessive"
conditions, but it's a little tricky, and the wikipedia page doesn't really
explain it. Basically, X-linked phenotypes aren't dominant/recessive in the
same way as the others. Usually, dominance takes place between chromosome
pairs within each cell. However, with X-chromosome inactivation, dominance
takes place _between_ cells. For example, women who are labeled "carriers" for
colorblindness actually are colorblind in half of the cells in their eyes, but
the other half are sufficient to perceive color almost as well as non-
carriers.

------
timmclean
I'm fascinated by the theory of increased variability in males being caused by
brain-related genes in the X chromosome. I'd highly recommend checking out
pages 18 and 19.

~~~
george_pavlov
While an interesting hypothesis, it seems unlikely: one copy of each
x-chromosome is deactivated in each cell, and this is done randomly in
development.

~~~
barry-cotter
This is one of the reasons women have superior colour vision to men so it's
likely enough. Some women have two different types of red cones while men have
a maximum of one. There are many other sex linked diseases as well. Unless an
allele is fully dominant if it's on the X chromosome men will display hreater
variance.

------
tokenadult
This book chapter is an interesting read. It illustrates the importance of
considering sample size, especially, when looking at preliminary research
findings.

After looking up the book from which this chapter is excerpted, I followed
other recommendations from Amazon to another very useful book,

[http://www.amazon.com/When-Can-You-Trust-
Experts/dp/11181302...](http://www.amazon.com/When-Can-You-Trust-
Experts/dp/1118130278/)

When Can You Trust the Experts: How to Tell Good Science from Bad in Education
by Daniel T. Willingham, a very astute psychologist with an interest in
education policy.

------
FrojoS
quote: "Obviously, they assumed that variability decreased proportionally to
the number of coins and not to its square root."

Why is this so important? The fact, that the variability increases with
smaller sample size was ignored completely by the protagonists in the provided
examples. Realizing weather this inverse effect is linear or not doesn't seem
to be the main problem in peoples intuition.

disclaimer: I have poor understanding of statistics.

~~~
sesqu
In the first example, which you quoted, they ended up allowing for greatly
more variability than intended, leaving the regulator vulnerable for
exploitation. The author speculated as to what sort of exploitation might have
occurred, but did not state that it did.

The rest of the examples had nothing to do with the specific relationship
between standard deviation and sample size, but with the more basic fact that
a relationship exists. This observation is arguably the more important one,
and is poorly argued in the chapter. It's also why some people always demand
error bars, though I personally prefer plotting individual data points where
possible.

The last example, while interesting, had very little to do with the equation
(despite a claim to the contrary), which makes me believe the topic was an
afterthought.

------
mturmon
Nice exposition and examples. Some of the most subtle and surprising phenomena
I've seen in looking at stochastic data have been due to sampling effects.

------
Kynlyn
I have no affiliation with Code School, but I saw that they recently offered a
free course on R, which is a programming language built around statistics.

------
hn-miw-i
Really interesting paper but the use of comic sans on the axes labels is a
turn off. Why comic sans?? Why? It's a crime against fontology.

------
j2kun
Equations aren't dangerous. People who make policy decisions about things they
don't understand are.

~~~
CamperBob2
Addressed in the text (in fact, that's the author's whole point.)

------
K2h
so is the conclusion to be that statistically significant sample size is as
important as the 'result' when measuring standard deviation?

<http://en.wikipedia.org/wiki/Sample_size_determination>

~~~
jfriedly
I don't think so. "Statistically significant" is a relative term and when
testing an entire population is infeasible (as it often is), we instead sample
some fraction that we believe is "statistically significant" on the assumption
that it will accurately reflect the whole.

The point of this article is that a sample only accurately reflects the whole
in _some_ ways. Variability in particular scales with the square root of the
sample size. And since misconceptions about variability have been at the heart
of many controversies (male vs. female intelligence, school size, cancer risk,
etc.), De Moivre's equation is important; even dangerous in the sense that
ignorance of it has led to billions of dollars wasted.

------
turbulents
I just started skipping through it once I saw the Comic Sans.

------
tlarkworthy
solution: quasi random sampling

