
Why We Need a Statistical Revolution - hownottowrite
http://www.stats.org/super-learning-and-the-revolution-in-knowledge/
======
glesica
The dangers of fishing for p values (to focus on those paragraphs) are covered
pretty extensively in undergraduate courses in, at the very least, economics
and political science. Something happens between that point, and the point at
which a researcher is trying to publish in a journal. I think it is fairly
obvious that the problem is related to competition for funding and
advancement. Of course this is probably a much more difficult problem to
solve.

~~~
ende
There's another potential source of the problem: the time gap in between those
undergraduate lessons are learned in the abstract and the later point in time
when the exercise if those lessons is required. Unexercised knowledge
atrophies like any other muscle.

Perhaps rather than placing so much emphasis on memorization recall exams,
undergraduate evaluation should be based on exercising the research and peer
review process. The best way to learn research is to do research.

~~~
glesica
That's what we did in undergraduate. Practically every class I had required a
research paper. We didn't always have to use statistics, but we were graded on
methodology. Recall exams were rare, most exams involved interpretation and
explanation. Of course I studied political science, things may be different in
other fields.

------
gwern
Odd editorial. It goes from a not-terribly-good explanation of some criticisms
of null-hypothesis significance-testing to an advertorial for his lab's
particular variant on ensembles. Or am I missing something important here?

------
lessthunk
As a statistican doing machine learning I think there is much more to it; You
can do statistics with models. There is progress in statistics, like in all
other sciences.

A catchy title, not explaining well what is really going on.

------
the8472
I always found the initial assumption that things are normally distributed
quite odd.

For example if you work with latencies then the normal distribution is a
terrible thing to use. The closer to zero you get the more skewed it has to
be, simply because everything that can influence the result will mostly change
it in one direction.

So in that particular case an Erlang distribution might be more applicable.
And even that often falls short if you have periodic things (background tasks)
occasionally messing with your latency, creating a multi-modal distribution.

~~~
rcthompson
The assumption is reasonable for a lot of cases because of the central limit
theorem. In particular, it applies if the variable of interest is the sum of a
large number of independent random variables. For example, the number of heads
in N coin flips is the sum of the number of heads in each of N independent
experiments each consisting of a single flip.

Of course, there are also a lot of important cases where that N isn't quite
large enough to justify the normal distribution, or where the quantity of
interest is not a sum of a large number of small effects, so the CLT doesn't
apply at all.

------
jkyle
The scenario presented by the author seems more explicit and nefarious than
the typical "p-hunting" that I've seen in academia. Though I've seen the
situation he presents, successively trying different models or parameters
until you find the expected effect, it's pretty obviously unethical.

What I've seen much, much more of goes something like this. Say your lab does
research that requires a very large quantity of data that's very expensive to
acquire. In academia, labor is relatively cheap and funding can be difficult
to come by. So a dataset that consumed potentially 100's of thousands of
funding dollars represents a significant resource investment.

So what do you do when the study is done? You try to figure out every way
possible to reuse that data. And the most common way of doing that is running
it using previous models looking for effects...enough people keep looking long
enough and, voila!, you find a significance.

In the end, it's the same bias. But the motivation is different and when I've
pointed out this is "p-hunting" most just give a blank stare.

------
solve
Modern machine learning replaced it. Or rather, relegated statistics to a
small subset of a wider field, only to be used for increasingly smaller subset
of tasks.

------
pXMzR2A
Step 1: Establish disciplines that ridicule radical criticism.

Step 2: Hire undergraduate math/stats students who took criticism courses at
gun point and hated them.

Step 3: ...

Step 4: "OMG Ponies and revolution"

Sir, you do not need a revolution, because it has already occured. You simply
need employees who have read critical approaches to statistics AS AN INTEGRAL
PART of their stats education. Those people are usually in

\- feminist studies (did the requested stats revolution)

\- gender studies (ibid, different name)

\- women's studies (ibid, different name)

\- lgbt studies (exploded feminist studies ftw)

\- african american studies (ibid)

\- critical studies (sociology, know of mainstream criticism only)

\- american studies (good with convoluted language)

------
gmcrews
And why we probably won't get one.

------
zootam
first thing we need is increased literacy in statistics.

if people actually understood they need to question the validity of statistics
presented to them before making conclusions and based on those possibly faulty
statistics, we'd be much better off.

------
oakwood
Terribly written article. This guys lives in an academic cave.

