I'm somewhat amused to see people complain that "commonly accepted" alphas of .05 are yielding high error rates. Surprise: setting your confidence interval at one-sigma yields a ton of false positives. There's a reason particle physics uses three-sigma as a threshold for further investigation, and five-sigma for discovery.
The difficulty is that in biology, psychology, and other really complicated systems, attaining even two or three sigma can be difficult and expensive. That suggest that we should continue to use low-confidence results... but accept their tentative nature instead of bestowing them with the aura of truth.
And before you start treating a few hundred thousand people with a drug, get a bigger sample size. :)
Misinterpretation and/or abuse of statistics is prevalent in scientific literature more often then you would think. Repeatability, consensus and peer review are good ways of minimizing this problem but it is still true that any random paper you open on say arXiv or even a standard journal is probably going to contain some form of abuse intentional or not.
Is this a serious problem? Mostly in those fields plagued by low confidence results and heavy approximations being made but it exists in all levels.
The article is rather alarmist though for my taste.
Oh, yeah, the 5% thing is only a small part of the article. I've just read four or five "OMG 80% of science is wrong!" articles this month, and think it's kinda tired.
The difficulty is that in biology, psychology, and other really complicated systems, attaining even two or three sigma can be difficult and expensive. That suggest that we should continue to use low-confidence results... but accept their tentative nature instead of bestowing them with the aura of truth.
And before you start treating a few hundred thousand people with a drug, get a bigger sample size. :)