
Type I and type II errors - aburan28
http://en.wikipedia.org/wiki/Type_I_and_type_II_errors
======
yiransheng
In main street interpretation of statistical findings, type II errors are
often swept under the rug.

A good illustration of the danger in only telling the type I side of the story
is [xkcd/1132]([http://xkcd.com/1132/](http://xkcd.com/1132/)), the
frequentist looks hilariously stupid, and we all hail the Bayesian smartness.

Yet a closer look from the frequentist view, the type II error of the test:
H_0: the sun has gone nova is Pr[detector reports no|the sun has indeed
exploded] = 1 - 1/36 > 95%; in other words the power of the test(1-Pr[type II
error]) is extremely low (1/36). Even though the test has a reasonably low
level alpha=0.05(type I error), it's still pretty useless as its power is also
pathetic.

The comic made the frequentist look idiotic, however, he's just not a good
frequentist with a sufficient grasp of type II error. In reality, many people
are like the freqentist, easily satisfied with a low p-value without thinking
about the fundamental usefulness of their tests (power), becoming victims of
type II errors.

EDIT: I made a mistake, see eli_gottlieb's comment below for correction.

~~~
eli_gottlieb
Actually, the comic specifies a 1/36 chance that the detector _lies_ , no
matter which result is true. So the P(detector reports no | sun has in fact
exploded) = 1/36, making it a very strong test. The real problem there is that
the p-value just doesn't mean what he thinks it means, and he in fact hasn't
factored in any prior for how often the sun actually explodes.

~~~
yiransheng
Oops, you are right, I am a bad freqentist. Just had the discussion about the
comic with a friend verbally, thought we figured it out. I guess it is really
easy to get even the most basic concepts applied wrong.

~~~
eli_gottlieb
And this is why I am currently in my office at 8:15pm reading Jayne's
probability theory book.

------
HCIdivision17
And my favorite: Type III Errors, wherein the very premise for the calculation
is flawed. Always be wary of precisely answering the wrong question.

[http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.105...](http://www.tandfonline.com/doi/abs/10.1080/01621459.1957.10501374)

------
dccoolgai
This is one of those fundamental things they should just make mandatory in
grade school. I constantly find myself explaining this to people I work with -
and even though it's kind of intuitive ("penny wise, pound foolish",
"straining gnats to swallow camels"), it helps to have a formal concept like
this to hang the concept on.

------
nimrody
The terms "false positive" and "false negative" are so much clear.

~~~
mewwts
True, but we get to sound a lot smarter saying stuff like "type I error" and
"type II error".

