
Odds Are, It's Wrong - Anon84
http://www.sciencenews.org/view/feature/id/57091/title/Odds_are,_its_wrong
======
aphyr
I'm somewhat amused to see people complain that "commonly accepted" alphas of
.05 are yielding high error rates. Surprise: setting your confidence interval
at one-sigma yields a ton of false positives. There's a reason particle
physics uses three-sigma as a threshold for further investigation, and five-
sigma for discovery.

The difficulty is that in biology, psychology, and other really complicated
systems, attaining even two or three sigma can be difficult and expensive.
That suggest that we should continue to use low-confidence results... but
accept their tentative nature instead of bestowing them with the aura of
truth.

And before you start treating a few hundred thousand people with a drug, get a
bigger sample size. :)

~~~
PhysicsAndYou
That isn't the only problem addressed.

Misinterpretation and/or abuse of statistics is prevalent in scientific
literature more often then you would think. Repeatability, consensus and peer
review are good ways of minimizing this problem but it is still true that any
random paper you open on say arXiv or even a standard journal is probably
going to contain some form of abuse intentional or not.

Is this a serious problem? Mostly in those fields plagued by low confidence
results and heavy approximations being made but it exists in all levels.

The article is rather alarmist though for my taste.

~~~
aphyr
Oh, yeah, the 5% thing is only a small part of the article. I've just read
four or five "OMG 80% of science is wrong!" articles this month, and think
it's kinda tired.

------
tjic
One thing to note is that there's a ton of selection bias: noteworthy findings
get published, non noteworthy ones do not. Therefore scientists will submit
papers containing noteworthy findings.

If you do 1,000 experiments in your lab, and (a) absolutely nothing
interesting happens; (b) you get a naturally noisy dataset on each, and (c)
you write up only those phenomena which would occur randomly only 5% of the
time

... you will submit 50 articles, each of them claiming something noteworthy,
and all of them false.

Perhaps there are a few actually noteworthy things that were discovered in
your lab. ...in which case in addition to the 50 BS papers, you'll also submit
1 or 2 that describe some real effect.

------
linhir
A great article on how the Naymen-Pearson Hypothesis Testing is not
particularly useful as a method of inference (e.g. the way most people tend to
think of statistics with p-values and 5% significance doesn't say much). Some
of what he is saying is a little shoddy, since Fisher actually came up with a
much more robust, but less used, method of inference called Maximum likelihood
estimation (MLE), which currently stands with The Bayesian Theory as the
competing (yet partially complimentary) theories of statistical inference.
Strict, but arbitrary, distinction between the null H0 and alternative H1
hypotheses, which is what p-values suggest, is as careless as the article
points out.

------
tokenadult
I see no one commenting on this interesting article has yet mentioned an
article on experiment design by a scientist-hacker, Peter Norvig,

<http://norvig.com/experiment-design.html>

that I mention frequently in comments here on HN because

1) it is accessible to general readers,

2) it is written by someone with excellent formal training and work experience
in the subject,

3) it links to excellent primary source literature for further reading,

and

4) it gives cogent, thought-provoking examples relating to errors of reasoning
that are common even among trained scientists.

Because I like science, and I like science to progress based on firmly
verified factual conclusions, I like my friends here on HN to know about the
pitfalls to look out for when anyone does experimental research. Thanks for
the submission of the latest journalistic article on related issues here,
which also gives us a lot to think about and which has elicited informative
comments.

------
nollidge
This is an interesting critique, but equating science with "a crapshoot" is
journalistically irresponsible. IANAS, but I read a few science blogs, and the
impression I get is that most scientific papers hedge and qualify extensively,
laboriously mentioning statistical significance, noting possible alternative
explanations for the observed data and imagining future experiments that ought
to be run to provide extra confidence in the findings.

He gives the example of all the studies linking genetic variants to acute
coronary syndrome, only one of which was confirmed in a later (presumably
larger?) study. Well, what does it mean for a study to link a gene to a
condition? Maybe they concluded, "gene FOO19 causes heart attacks, bitches!"
Or maybe it was more like, "4 out of 5 coronary sufferers we studied had this
FOO19 gene. Someone should look into it." Those are two very different
statements; one is way out-of-line, and the other is responsible reportage.

