
First results from psychology’s largest reproducibility test - timclark
http://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433
======
jpmattia
It would be interesting to know how reproducible the reproducibility test
turns out to be.

Edit: Just to expand a bit --

Suppose I flip a fair coin 100 times, and publish each result as its own god-
given Truth. Now somebody comes along and questions my 100 Truths, so they
flip a coin 100 times; The expectation value is that 50 of my Truths
reproduce. Naturally there is a sigma (of about 10 I think), so the
reproducibility study shows 50 ± 10.

The linked reproducibility study is right around expection - sigma. I realize
that's numerology, but kinda funny. Nevertheless, my original point: The
reproducibility study should be run several times to understand the random
nature of the results. (Do the same studies reproduce their results? Or is it
a different 50-ish results?)

~~~
capnrefsmmat
What you're asking is basically the concept of statistical power. Assume the
original study found an effect size E, and we take that as the truth. How
likely is the replication attempt to find a statistically significant effect?

The Reproducibility Project calculated the sample sizes necessary in advance,
so if the effects are the size the original researchers claim, they'd have
good power to detect them.

Their power could be worse than they expect, though; pioneering studies tend
to overestimate effect sizes, because their sample sizes are too small and
they filter for statistical significance. I call the problem truth inflation:
[http://www.statisticsdonewrong.com/regression.html#truth-
inf...](http://www.statisticsdonewrong.com/regression.html#truth-inflation)

Anyway: understanding the random nature of results is exactly the job of
statistics, and the Reproducibility Project researchers are being very careful
with their statistics.

------
heimatau
tldr; Efforts to reproduce 100 psychological findings. Only 39 were
reproducible. And 61 were not.

Hal Pashler, “A lot of working scientists assume that if it’s published, it’s
right,” he says. “This makes it hard to dismiss that there are still a lot of
false positives in the literature.”

~~~
innguest
Thanks for that. That's Psychology in a nutshell for you folks, this amazing
"science". Now let's wait for a similar study for economics, the other amazing
"science".

~~~
danmaz74
How do you define "science"?

~~~
DanBC
Not like this "scientist" at a reputable university.
[http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm](http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm)

> · Recent hand-wringing over failed replications in social psychology is
> largely pointless, because unsuccessful experiments have no meaningful
> scientific value.

> · Because experiments can be undermined by a vast number of practical
> mistakes, the likeliest explanation for any failed replication will always
> be that the replicator bungled something along the way. Unless direct
> replications are conducted by flawless experimenters, nothing interesting
> can be learned from them.

{edit: read the link. It's scary to read this stuff coming from a science
PhD.}

Other psychologists have realised that there's a "crisis of replication" in
psychology. But it's not just psychology, it affects other sciences (albeit to
varying degrees) too.

[http://slatestarcodex.com/2014/04/28/the-control-group-is-
ou...](http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-
control/)

~~~
danmaz74
Ok, but one thing is to say that there are bad scientists in the field of
psychology, and another one is to say psychology is not a science at all. It's
not (and it can't be) an "exact" science, but it's science all the same.

------
chuckcode
Great to see psychology going after this in an open forum. The current
publishing systems are likely to generate some false positives even assuming
no bad actors. Replication by independent 3rd parties is a great way to
confirm important results. I wish the nutrition community would do this for
diets and nutrition before changing the guidelines all the time.

------
Retric
IMO, Publishing results before peer review seems like a much worse issue than
just reproducibility problems.

~~~
capnrefsmmat
They've been transparent from the beginning, by posting study protocols and
analysis plans. You can think of this like posting a physics preprint on the
arXiv before peer review. Nothing wrong with that.

