
Why Most Published Research Findings Are False (2005) - NoB4Mouth
https://fermatslibrary.com/s/why-most-published-research-findings-are-false
======
majidazimi
The problem I can see right now with scientific community is that, no one pays
you to re-implement the same idea just to confirm that it is correct. Every
professor wants shiny new innovation from her Ph.D student. No one wants you
to experiment the currently published ideas.

I'm about to finish my master thesis. I implemented a couple of ideas
collected from multiple papers in my thesis, and I can say all of those
absolutely stunning results were just intelligently crafted experiments that
are not applicable elsewhere.

Basically, my whole thesis is just to show that some paper was wrong and the
idea is not applicable in another experiment.

~~~
chriskanan
I often have my PhD students re-implement ideas to gain skills and to verify a
method works. It is possible to publish these efforts, but it isn't easy. It
typically involves comparing multiple methods on datasets they haven't been
tested on before to see how well the results generalize beyond the original
paper. It is hard to publish in prestigious venues with this approach, but we
have had some success. Replication and comarison makes for a good MS or early
PhD project, but later PhD students have to showcase an ability to create new
algorithms. In my lab, we try very hard to not cherry pick and to do good work
that generalizes.

~~~
gus_massa
From your profile, you are working in Deep Learning. In other fields it's much
more difficult to be sure that the replication is accurate.

Do you have the correct variety of rats? Are they receiving the same kind of
food? Does they get the same illumination during the day? ... Theoretically al
the details should be clear from the published paper, but most of the times
the paper is full of "underspecified"[0] parts.

Also, in many fields even if the paper is only about calculations in a
computer, the programs are not published (or are a mess (or an unpublished
mess)) and the data are not published (or are a mess (or an unpublished
mess)). So it's more difficult to even make a direct copy of the results of
the paper.

Also, there is a lot of informal replication, essentially what you do but
without the final publication step. Just replicate somewhat similar to the
original paper, but then publish a version with some extension or tweak.

[0] aka "missing"

------
tensor
It's worth noting that this is talking about research that involves sampling
groups of people, or other similar sampling approaches. It is not talking
about mathematical results which you can often find in computer science,
physics, and elsewhere. Not is it necessarily true all of the physical
sciences.

It also shouldn't be read as "science is broken and wrong so therefore my
opinion should be considered equally." There is definitely a problem with
accuracy in many scientific fields that needs to be addressed, but the baby
doesn't need to be thrown out with the bath water.

~~~
epistasis
It's not even talking about biology research, or even most medically related
research.

While it was a great point to make at the time, far too much has been made of
it. Yes, don't p-hack, but it's also better to publish data than to withold it
just because there wasn't a positive result. We need better publication
mechanisms for data that doesn't have any significant findings.

~~~
astazangasta
Err, this 100% applies to biology research, where there is absolutely a
problem of small sample sizes, small effects, etc. Our studies (my field is
cancer) are routinely vastly underpowered because of the high variability of
the datasets we are studying, the low availability of samples, and the large
number of variables we collect on these sample.

Also, to a good approximation, everybody p-hacks. Furthermore, the habit of
publishing "noteworthy" results (true in every journal, especially true for
large impact factor journals) is essentially p-hacking across the entire
field. This is a huge problem.

~~~
epistasis
How many of your papers consist of a single statistical test, without
validation experiments? Because that's the setup that's described in order for
"most published research findings are false."

In reality, I've never seen a biology paper without several lab techniques and
orthogonal verifications, with p-values on some but not all of those
experiments.

------
reasonattlm
This is why one has to read papers in context. You can't just read a paper:
you have to then go off and read some reviews that cover the surrounding
science. Then read some other papers on the topic of interest. Pay attention
to dates on papers, as opinions change with time. Note the disagreements.

Reading the scientific literature is reading the moment to moment output of a
noisy search algorithm. Any given publication is near meaningless on its own.

This business of reading the context for a specific item of research isn't
that hard. If as a layperson you feel up to having an opinion on a specific
paper, then you are certainly equipped to do more reading in the field.

Start with review papers, which tend to be a gentler uphill slope, and then
fit other papers into what you see there. Take note of the disagreements
between reviews, the different emphasis placed on different aspects of the
topic. In most scientific fields review papers are usually pretty good at
explicitly covering the unknowns and debates of interest. The subtexts and
unwritten stuff, such as funding-driven conflicts of research strategy, take
longer to figure out. But one has to start somewhere.

[https://www.fightaging.org/archives/2009/05/how-to-read-
the-...](https://www.fightaging.org/archives/2009/05/how-to-read-the-output-
of-the-scientific-method/)

"The scientific community doesn't produce an output of nice, neat tablets of
truth, pronouncements come down from the mountain. It produces theories that
are then backed by varying weights of evidence: a theory with a lot of support
stands until deposed by new results. But it's not that neat in practice
either. The array of theories presently in the making is a vastly complex and
shifting edifice of debate, contradictory research results, and opinion. You
might compare the output of the scientific community in this sense with the
output of a financial market: a staggeringly varied torrent of data that is
confusing and overwhelming to the layperson, but which - when considered in
aggregate - more clearly shows the way to someone who has learned to read the
ticker tape."

~~~
TeMPOraL
An interesting perspective, but it doesn't really dissolve the big problems -
that the "noise" in the output of this "search algorithm" is _absurdly high_ ,
and that single papers often drive real-world decisionmaking.

------
serghiou
Super important paper, the implications of which have been corroborated
repeatedly within the most prestigious publications in psychology, social
sciences/economics and cancer biology. If you'd like to read more about such
issues and actually work on doing something about it, you may want to check
out the Reddit community
([https://www.reddit.com/r/metaresearch/](https://www.reddit.com/r/metaresearch/))
and a recent initiative at Stanford
([http://reproduciblescience.stanford.edu/](http://reproduciblescience.stanford.edu/)).

------
xte
The problem have a name: business.

If research is public, well founded by government, and universities are public
entity research is accurate and effective, otherwise it's only a matter of
making money quickly and moving on.

------
mindfulplay
This only adds credence to the fact that social survey is not a science. Most
of it is glorified door to door salespeople work. It's not even research.

We have to draw a clear line soon to prevent good-intentioned people from
being lumped into shitty science.

------
swagatkonchada
Maybe not in mathematics

~~~
duckerude
>Anecdotal evidence suggests that as many as a third of all papers published
in mathematical journals contain mistakes - not just minor errors, but
incorrect theorems and proofs…

[http://www.gwern.net/The-Existential-Risk-of-Mathematical-
Er...](http://www.gwern.net/The-Existential-Risk-of-Mathematical-Error)

------
stochastic_monk
Please add a [2005] tag.

~~~
sctb
Done. Thanks!

~~~
joe_the_user
It's a reprint of an article in PlosMed, where the context is obvious in the
original but not here, so the title might need changing too.

