

Most Published Research Findings Are False–But Little Replication Goes Long Way - gwern
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0040028#close

======
bmh100
I would like to see replication handled with the introduction of a new type of
journal: one that only publishes studies that have been previously published
but are now backed up by additional, confirming studies. Maybe two
replications would be enough.

It would be even better when paired with a journal of studies failing to
reject the null hypothesis. The first journal could then base the publishing
threshold on statistical power of the group, rather than a simple replication
count.

------
anton_tarasenko
It's a bit trickier than the title says. By a "false" finding, they mean "a
finding fails to yield the same result in at least one other study." But
there's nothing wrong with failing a replication. See cancer risk factors
research papers: they regularly yield results with opposite signs.

Rather, we should talk about "more research is needed to get a representative
sample of studies of a phenomena."

~~~
gwern
> See cancer risk factors research papers: they regularly yield results with
> opposite signs.

You'll have to expand on that - when I think of 'cancer risk factors', I think
of data-dredging epidemiological analyses which are predictably wrong by the
logic of hypothesis-testing, or I think of biochemical studies which blatantly
overfit by failing to cross-validate or keep holdout samples, and lack of
replication seems like it has the usual meaning.

