
Just 11% of 53 cancer research papers were reproducible - vog
http://www.nature.com/nature/journal/v485/n7396/full/485041e.html
======
jballanc
I came to the biological sciences trained as a chemist. As I began working on
my Ph.D. and variously encountered papers on cell signaling research (the
field that much "cancer research" would fall into) it was blindingly apparent
to me, perhaps because of my chemistry background, what was going on...

Cell Signaling (and much of Biology) research is in its Alchemy phase. Now,
alchemy often gets a bad rap as being completely worthless, snake-oil type
stuff, but this was not the case at all. Rather, individuals were working on
an area about which almost nothing was known, and (more importantly) for which
the key _central organizing laws_ were not yet revealed (for Alchemy: the
atomic theory of matter and chemical bonding, for Cell Signaling: how
individual proteins and small molecules interact). Furthermore, the goals were
lofty and almost certainly unattainable (for Alchemy: turn base metals into
gold, for Cell Signaling: cure cancer), driving people to do "rush" work. It's
not that the results are completely invalid, or that the experiments are
useless. It's just that everyone feels like they're _so close_ to a solution
(Alchemists were way off with Phlogiston, and I'll bet Cell Signaling
researchers are similarly clueless as to what _really_ matters) that no one
takes the time to step back and synthesize the results in an attempt to
understand the forest from the trees.

Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri
Mendeleev.

From experience, the fact that 89% of these "cancer research" papers are not
reproducible almost definitely has less to do with fraudulent data and so much
more to do with crazy complex experimental setups that end up probing a half-
dozen experimental variables all at once (without the researchers even
grasping that this is going on).

Yeah, publish or perish sucks, but what sucks more is the death of basic
science research. The Alchemists eventually became Chemists because they
refocused on core principles (atomic theory, bonding) and forgot about the
lofty goals (turn lead into gold)...

...but try telling any politician that they should decrease cancer research
funding and refocus on genetics, structural biology, and evolution research.
I'd love to know how they respond.

~~~
zeteo
>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri
Mendeleev.

That's very optimistic, but not necessarily true (even ignoring the question
of _when_ ). Priestley succeeded so well because he developed simple,
reproducible experiments (expose mercury oxide to sunlight to produce oxygen
etc.) that he was happy to share for no financial gain (even traveling to
others' labs to help them reproduce his work [1]). The current environment of
paywalls and competitive grants seems hardly conducive to the rise of similar
figures.

[1]
[http://en.wikipedia.org/wiki/Experiments_and_Observations_on...](http://en.wikipedia.org/wiki/Experiments_and_Observations_on_Different_Kinds_of_Air#Discovery_of_oxygen)

~~~
anologwintermut
What do paywalls possible have to do with the reproducibility of results?
Anyone who has access to a lab capable of even attempting to reproduce such
research is either 1) at an institution that has access to those journals or
2) can use google to find the papers anyway. Hopefully both.

The fact that tax-payer funded research is put behind paywalls is a travesty ,
but it seems wrong to claim it actually hinders researchers from doing
anything given.

~~~
kanzure

        > What do paywalls possible have to do with the reproducibility
        > of results? Anyone who has access to a lab
    

Actually, there's no institution subscribed to every journal. There are, in
fact, many papers inaccessible to even the most funded labs. Spotty coverage.

~~~
trentlott
I'm published in a highly-ranked journal that our university doesn't have
access to.

Sorta funny.

~~~
anologwintermut
Can you find the paper online by searching for it?

------
zipdog
What is more troubling is not that so few of these results results are
reproducible, but that it appears almost no-one is trying to reproduce the
results of earlier studies. The ability of the scientists who wrote the paper
to even get access to the resources necessary to try and reproduce the results
is limited.

I'm reminded of Feymann's Cargo Cult Science:

"I was shocked to hear of an experiment done at the big accelerator at the
National Accelerator Laboratory, where a person used deuterium. In order to
compare his heavy hydrogen results to what might happen with light hydrogen he
had to use data from someone else's experiment on light hydrogen, which was
done on different apparatus. When asked why, he said it was because he
couldn't get time on the program (because there's so little time and it's such
expensive apparatus) to do the experiment with light hydrogen on this
apparatus because there wouldn't be any new result. And so the men in charge
of programs at NAL are so anxious for new results, in order to get more money
to keep the thing going for public relations purposes, they are destroying--
possibly--the value of the experiments themselves, which is the whole purpose
of the thing. It is often hard for the experimenters there to complete their
work as their scientific integrity demands."

For anyone who hasn't read it, the whole thing is excellent
<http://www.lhup.edu/~DSIMANEK/cargocul.htm>

~~~
jerf
If I had to sum up science in one phrase, I wouldn't say anything about the
"scientific process" or anything like that. I would say: "Look for reasons why
you are _wrong_ , not reasons why you are right." You can _always_ find
reasons why you're right. You can always do an astrology forecast and find
someone for whom it was dead-on accurate. That's not the problem with
astrology, the problem lies in how often it is wrong. It's wrong so often it's
useless. But if you only examine the positive evidence in favor of it, you
will never come to that conclusion.

The theories that are powerful and worthwhile are the ones that are rarely or
never wrong. Can't always get "never". It's a complicated world and we aren't
all physicists. But we at least ought to be able to get "rarely", and if you
can't, well, I guess it's not science then. That's OK. Unfortunately, not
everything is amenable to science, though you can still approach it in this
spirit of trying to see how you might be wrong rather than proving yourself
right.

Once you start looking around with that standard, it's not hard to see how
little science is really being done. Why are we publishing these dubious
studies? Because for all the scientific trappings we claim, with statistics
and p-values and carefully-written recordings of their putative procedure
written in precisely the right way to make it sound like everything was
recorded (while still leaving out an arbitrary number of relevant details),
we've created a system where we are telling people to look for reasons why
they are right... or we won't publish their results. Guess what kind of
results we get with that?

If you start from the idea that you need to look for why you are wrong, the
scientific method will fall out of that, along with any local adjustments and
elaborations you may need, and every discipline, sub-discipline, and indeed at
times even individual experiments need adjustments. If you start with "The
Scientific Method", but you don't understand where it came from, how to use
it, or what it is really telling you, you'll never get true science, just...
noise.

~~~
specialist
_"Look for reasons why you are wrong, not reasons why you are right."_

I share your worldview. Makes intuitive sense to me. It's intellectually
honest. And if I'm wrong about something and no one corrects me, I get kinda
grumpy.

Maybe everyone else already knows, but I learned relatively late that it's
called Popperian.

[http://en.wikipedia.org/wiki/Karl_Popper#Philosophy_of_scien...](http://en.wikipedia.org/wiki/Karl_Popper#Philosophy_of_science)
<http://en.wikipedia.org/wiki/Falsifiability>

~~~
jerf
Falsifiability is one natural landing point, but it is also somewhat
controversial. What I'm advocating isn't so much a philosophy as a state of
mind, one hopefully less controversial than trying to declare a "definition of
science". I think of it more like a mind hack you can perform on yourself. (So
much discipline boils down to figuring out how your conscious brain can fool
your subconscious brain.)

------
campnic
I have been in discussions about this with one of my friends working in
academic materials research. Its amazing the amount of work today done by
scientist at universities writing code without very basic software development
tools.

I'm talking opening their code in notepad, 'versioning' files by sending
around zip files with numbers manually added to the end of the file name, etc.

This doesn't even begin to scratch the surface of the 'reproducible results'
problem. Often times, the software I've seen is 'rough' to be kind. Most times
its not even possible to get the software running (missing some really
specific library or some changes to a dependency which haven't been
distributed) or its built for a super specific environment and makes huge
assumptions on what can 'be assumed about the system.' This same software
produces results which end up being published in journals.

If any of these places had money to spend, I think there could be a valuable
business in teaching science types how to better manage their software. Its
really unfortunate that outside of a few core libraries (numpy, etc.) the
default method is for each researcher to rebuild the components they need.

I'm surprised about only 11% of results being reproducible. It seems lower
then I'd expect. I agree we don't want to optimize for reproducibility, but
obviously there is some problem here that needs to be addressed.

~~~
epistasis
This is an entirely different issue than code; code mostly does the same thing
when you run it twice. There's no such guarantee in biology. A cancer cell
line growing in one lab may behave differently than descendants of those cells
in a different lab. This may be due to slight differences in the timings
between feeding the cells and the experiments, stochastic responses built into
the biology, slight variations between batches of input materials for the
cells, mutations in the genomes as the cell line grows, or even mistaking one
cell line for another.

Reproducibility of software is a truly trivial problem in comparison.

~~~
campnic
Oh, I agree. Biological experiment reproducibility is an incredibly hard
problem. You are probably right that it is 'trivial' by comparison in the same
way that landing on mars is trivial to landing on Alpha Centauri.

------
api
After decades of publish-or-perish sweatshop science I'm sure the great
shining archive of scientific truth is sort of like an inbox with no spam
filter.

~~~
evoloution
Such a powerful sentence.

Lately science looks like olympic games without anti-doping control...

~~~
api
I'm becoming pretty convinced that the human future depends on the disruption
and obsolescence of bureaucracies.

~~~
spikels
I think you are on to something. When science became so big that it first
needed then eventually was run by "managers" I bet it lost much of its
effectiveness. But how do we change this? For starters there is a lot of money
involved for the institutions conducting this (apparently shoddy) research.

~~~
kanzure
> When science became so big that it first needed then eventually was run by
> "managers" I bet it lost much of its effectiveness.

You can just do projects on your own, you know. There's nothing about science
that absolutely requires an institution.

<http://groups.google.com/group/diybio>

------
pcrh
I work in biomedical research, and this finding has been discussed quite
broadly. Most researchers don't believe it.

Exactly reproducing novel findings usually requires a significant investment
into the underlying procedures, which most places will not undertake.
Reproduction instead usually occurs as part of an extension of the initial
findings. The complexity of biology means that the first findings are
frequently not reproduced _exactly_ the same way as the original, but this
does not detract from the "direction" of the initial findings.

For example, the first finding might be that protein A promotes tumor growth
by modifying protein B. An extension of these findings might be Protein A
_sometimes_ modifies protein B, and when it does tumor growth is stimulated,
but it _mostly_ does not modify protein B, it instead modifies protein C,
which suppresses tumor growth. In this case, were the first results
replicated? Yes, and no.

This is how most biomedical research proceeds...

~~~
rdfi
Really?

Meaning that either there's another unaccounted variable(s) that controls the
effect of A in B and C, meaning that we cannot conclude anything from the
experiment.

Yes and no is not acceptable...

~~~
klibertp
You're wrong. What you say - that we conclude anything if not everything was
taken into consideration - is undoubtedly true for mathematical meaning of
"conclude". Mathematicians, computer scientists and a few others, like
theoretic physicists, have a luxury of using law of excluded middle, which
Sherlock Holmes also used. The famous detective states that if you eliminate
everything that is impossible, whatever remains is true, even if it sounds
improbable.

All the people who use this law do so because their work is about some kind of
model, which can be wholly known. However, if you have a misfortune of working
in the real world - like nearly everyone - then you can't apply this law. This
means that you won't ever get a proof in a mathematical sense. You won't ever
be completely certain - you can be convinced beyond reasonable doubt, but
that's all.

So when we talk about how biologists are just grant hunters because someone
couldn't reproduce their experiments we need to take this into account. I
don't know, but if I had to guess I'd say that nobody ever expected these
experiments to be 100% accurate, 100% reproducible or 100% true. I think they
are treated as a data point, some input to think of, and not definite truth.

But I may be completely wrong here, of course.

------
narrator
I read a lot of medical research papers and I'd say about 1 in 10 is about
cancer. They usually go something like this: We found this receptor on a
cancer cell and it caused it to grow or shrink. We therefore propose making a
drug to inhibit or act as an agonist for this receptor site which also affects
an enormous amounts of completely un-releated stuff around the body in order
to fight cancer.

Truth is, they may have just gotten a funny little mutant of a cancer cell
that happens to heavily express this one receptor as part of its unique
mutated biology. For all we know a lot of cancer research has been about
exploring the peculiarities of the genome of Henrietta Lacks.

------
brudgers
Perhaps we are looking for the wrong measure of success. This is not inclined
plane ball rolling in fifth period Physics with Mr. Johannes. Reproducibility
in cutting edge experiment is success. It means that what is being measured -
even if the wrong thing - is within the control of the experimenter.

It is absurd, but practical, to publish experimental results with the
implication that they are reproducible based on a peer review of the results.
"Well they look reproducible to me," when uttered by a peer reviewer is the
current standard. All we are seeing is that the _a priori_ reasoning of
experts is not a substitute for empirical investigation of scientific claims.

Let us not forget that a journal which finds no worthy articles this quarter
has only two options, publish unworthy ones or publish no articles. I know how
I react when a magazine subscription does not come.

~~~
atrus
I wonder if it would be possible to produce a journal that only released an
issue when it has, say, 10 articles? No monthly/quarterly schedule, you could
have an issue release next week, or next year.

~~~
brudgers
As a digital document it is easy. Because printing requires substantial
coordination, it is less practical. The subscription model common to the
scientific journal industry makes it contractually problematic.

I wonder what portion of academic research is equivalent to content farming.

------
codeulike
_Most published scientific research papers are wrong, according to a new
analysis._

[http://www.newscientist.com/article/dn7915-most-
scientific-p...](http://www.newscientist.com/article/dn7915-most-scientific-
papers-are-probably-wrong.html)

 _Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS
Med 2(8): e124. doi:10.1371/journal.pmed.0020124_

<http://dx.doi.org/10.1371/journal.pmed.0020124>

~~~
kailuowang
Is this analysis a published scientific research?

~~~
return0
Joking aside, the authors of these studies have highlighted the low
statistical power of research in a number of fields (see also
<http://www.nature.com/nrn/journal/v14/n5/abs/nrn3475.html>), which is a
serious issue that often leads to over-interpretation. It's as if researches
willing to publish more are rushing hasty studies out the door.

It's a good thing that people can measure these things and ring the alarm

------
RyanMcGreal
The analysis in question:

[http://www.nature.com/nature/journal/v483/n7391/full/483531a...](http://www.nature.com/nature/journal/v483/n7391/full/483531a.html)

~~~
micro_cam
This is a really insightful article that contains a number of points that
qualify the result and present lessons for any one interested in data science.

A few points that stand out to me:

!) The 11% number comes not from a randomized sample of cancer papers but from
attempts made by AmGen to reproduce results that might be useful in drug
development. This means that they made good effort to reproduce these results
but there is also a strong selection bias that needs to be acknowledged.

2) They point out that using survival time as the measurement complicates
things. I've done a lot of machine learning and statistics on cancer and
medical data and, in my experience, this can not be overemphasized. and There
are loads of confounding factors that contribute to survival time. I expect
that big break throughs will come as we develop rigorous ways for measuring
the behavior of a tumor (does it metastasize? how does it feed itself?) and
use those as the targets of our regression. (Currently these thins are
measured by human visual inspection if at all.)

3) They point out that the studies that were repeatable were the ones that
were careful about using blinded controls, and eliminating investigator bias.
This is basic stuff but easy to overlook. In terms applicable to a startup, a
data scientist needs to me motivated to vet existing and proposed practices
and identify flawed ones as much or more then they are motivated to maximize
gain.

~~~
pcrh
With regards to 1), it would be ironic if this study was itself not
statistically valid...

------
pg
I feel obliged to point out that these other journals are merely copying the
real pioneer in this field: <http://jir.com>.

~~~
irollboozers
Negative results vary from field to field. In math or physics, a conclusive
negative can be very important in deciding what to study next or disproving
established theory. In biology, a negative usually just means "I haven't found
what I'm looking for yet".

~~~
omnisci
And that is the exact problem with certain researchers. I just had to explain
this to my undergrad who's project resulted in negative results (his 1'st
study). I congratulated him, as he discovered that the treatment he gave did
NOT effect the genes he looked into. He didn't understand, so I had to explain
that his study (with the correct controls), provided information that is
useful to science. Negative data = data. Data = good :)

------
gordaco
Wow, that's sad. Are we really so blind when it comes to cancer research?

I recall that oncology journals usually have a ludicrously high impact index
(as ludicrously as 5 digit IIRC); that means there are a lot of citations,
which is an indicator that there is a lot of research going on. And with a lot
of research going on, well, you can expect a lot of false positives. So, I'm
wondering, could this be a case of cherry-picking or some kind of selection
bias? It wouldn't be difficult to select a lot of bogus-sounding research and
test it.

~~~
atrus
But who is going to pay to reproduce that research? What if that research took
years to do?

~~~
jerf
Why do the research in the first place if the only outcome is a number,
floating in space, disconnected from anything and unreproducible?

You're still sort of operating on the idea that the unreproducible papers have
some sort of abstract value to them, and therefore we shouldn't slow the flow
of them lest we ruin their value. But they don't. They're worthless. They're
_worse_ than worthless. They're of negative value. It would be far better to
slow down and verify that what we think we know is actually true, because in
the end that would actually both faster and a more efficient use of resources.
Basically, instead of learning worse than nothing (thinking we know something
but actually being _wrong_ ), we'd learn something. That's a pretty decent
upgrade.

------
ilikejam
An article claiming that only 11% of findings could be reproduced doesn't cite
its sources, thus rendering the findings of the article unreproducible. Nice
work!

~~~
Ensorceled
Nature wrote an editorial justifying the situation, you can read it here
[http://www.nature.com/nature/journal/v485/n7396/full/485041e...](http://www.nature.com/nature/journal/v485/n7396/full/485041e.html)

~~~
return0
> "those authors required them to sign an agreement that they would not
> disclose their findings about specific papers"

I don't know why this doesn't cause a huge stir in the scientific community.
Seems like everyone is fine with people sweeping their negative data under the
rug. Shameful is a very mild word for that.

~~~
ars
How were those authors able to enforce this request? To be a proper paper it
should not be necessary to contact the author in order to reproduce it.

If you had to ask the author for more details then their paper was incomplete
- perhaps that's why it could not be reproduced.

~~~
aidenn0
If you want to maximize your chances of reproducing the results, you will use
identical apparatus and methods to what the original study used. That is often
not possible to do just from reading the paper.

~~~
ars
If the identical apparatus is essential to reproducing the result, then the
description of that apparatus is part of the experiment. Omitting that means
it's not a proper paper.

~~~
spikels
If only it were so. Often important parts of the protocol exist only in a hand
written entries in a lab notebook. Because very few people are reproducing
experiments and it is lots of extra work and jornals have limited space all
the required details are rarely included. This is a big part of the problem.
The typical work around is to contact the original reaseach to fill in the
missing details.

------
bjterry
It seems like a big win for science could be had by centralizing some aspects
of research within academic and research institutions. There should be well-
funded software czars and statistics czars at each university that facilitate
the efforts of the individual researchers to try to reduce the amount of
shoddy science that gets published. Researchers are really terrible at this,
and presumably slower than dedicated people even who are working on many
different projects, so the division of labor should make the whole apparatus
vastly more efficient, able to write and win more grants, etc. Of course, many
researchers would fight against this because if you can't publish shoddy
research, you can't publish as much research, but I would think there should
be some way to get there from here.

------
mchusma
This is a great issue to highlight. However, I'm unconvinced we need to
optimize around creating research that can be reproducible, rather that the
process of science generates the best possible results.

Both the scientific and lay public should read this (appropriately) as: one
study does not prove a result, there is an 89% chance that a single research
paper is simply wrong.

This mentality seems to be the bigger gap, although generating high quality
research is a big part of the equation (mostly I think for the wasting of
resources to validate, process, and reproduce claims).

~~~
ttrreeww
Cold fusion was reproducible. What are you suggesting we do here?

------
tokenadult
This editorial commentary and the article on which it is based are part of an
ongoing effort to improve the quality of scientific publication in a number of
disciplines. The Retraction Watch group blog

<http://retractionwatch.wordpress.com/>

by two experienced science journalists picks up many--but not all--of the
cases of peer-reviewed research papers being retracted later from science
journals.

Psychology as a discipline has been especially stung by papers that cannot be
reproduced and indeed in many cases have simply been made up.

[http://www.nytimes.com/2013/04/28/magazine/diederik-
stapels-...](http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-
audacious-academic-fraud.html?pagewanted=all&_r=0)

That has prompted statistically astute psychologists such as Jelte Wicherts

<http://wicherts.socsci.uva.nl/>

and Uri Simonsohn

<http://opim.wharton.upenn.edu/~uws/>

to call for better general research standards that can be practiced as
checklists by researchers and journal editors so that errors are prevented.

Jelte Wicherts writing in Frontiers of Computational Neuroscience (an open-
access journal) provides a set of general suggestions

Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting
the daylight in: reviewing the reviewers and other ways to maximize
transparency in science. Front. Comput. Neurosci., 03 April 2012 doi:
10.3389/fncom.2012.00020

[http://www.frontiersin.org/Computational_Neuroscience/10.338...](http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full)

on how to make the peer-review process in scientific publishing more reliable.
Wicherts does a lot of research on this issue to try to reduce the number of
dubious publications in his main discipline, the psychology of human
intelligence.

"With the emergence of online publishing, opportunities to maximize
transparency of scientific research have grown considerably. However, these
possibilities are still only marginally used. We argue for the implementation
of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and
(3) online data publication. First, peer-reviewed peer review entails a
community-wide review system in which reviews are published online and rated
by peers. This ensures accountability of reviewers, thereby increasing
academic quality of reviews. Second, reviewers who write many highly regarded
reviews may move to higher editorial positions. Third, online publication of
data ensures the possibility of independent verification of inferential claims
in published papers. This counters statistical errors and overly positive
reporting of statistical results. We illustrate the benefits of these
strategies by discussing an example in which the classical publication system
has gone awry, namely controversial IQ research. We argue that this case would
have likely been avoided using more transparent publication practices. We
argue that the proposed system leads to better reviews, meritocratic editorial
hierarchies, and a higher degree of replicability of statistical analyses."

Uri Simonsohn provides an abstract (which links to a full, free download of a
funny, thought-provoking paper)

<http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588>

with a "twenty-one word solution" to some of the practices most likely to make
psychology research papers unreliable. He has a whole site devoted to avoiding
"p-hacking,"

<http://www.p-curve.com/>

an all too common practice in science that can be detected by statistical
tests. He also has a paper posted just a few days ago

<http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879>

on evaluating replication results (the issue discussed in the commentary
submitted to open this thread) with more specific tips on that issue.

"Abstract: "When does a replication attempt fail? The most common standard is:
when it obtains p>.05. I begin here by evaluating this standard in the context
of three published replication attempts, involving investigations of the
embodiment of morality, the endowment effect, and weather effects on life
satisfaction, concluding the standard has unacceptable problems. I then
describe similarly unacceptable problems associated with standards that rely
on effect-size comparisons between original and replication results. Finally,
I propose a new standard: Replication attempts fail when their results
indicate that the effect, if it exists at all, is too small to have been
detected by the original study. This new standard (1) circumvents the problems
associated with existing standards, (2) arrives at intuitively compelling
interpretations of existing replication results, and (3) suggests a simple
sample size requirement for replication attempts: 2.5 times the original
sample."

The writers of scientific papers have a responsibility to do better. And the
readers of scientific papers that haven't been replicated (or, worse, press
releases about findings that haven't even been published yet) also have a
responsibility not to be too credulous. That's why my all-time favorite link
to share in comments on HN is the essay "Warning Signs in Experimental Design
and Interpretation" by Peter Norvig, LISP hacker and director of research at
Google, on how to interpret scientific research.

<http://norvig.com/experiment-design.html>

Check each submission to Hacker News you read for how many of the important
issues in interpreting research are NOT discussed in the submission.

~~~
kirk21
The pressure on academics is indeed an issue. They have to teach, guide PhD's,
research and publish (without mentioning controlling internal affairs).

On a side note. Has anyone found a good alternative for Mendeley? I heard
<http://bohr.launchrock.com/> is working on sth cool.

~~~
jballanc
Have you considered Papers? <http://papersapp.com/>

------
elizabethiorns
We are attempting to solve this by identifying and rewarding reproducible
research (www.reproducibilityinitiative.org). However, it has been incredibly
difficult to get funding to conduct the validation studies due to the
obsession of funding "novel" results, rather than funding the replication
studies required to identify reproducible research. Funders need to step up to
solve this problem.

------
wheaties
This is one of the reasons I went into math and software. In math there is
only black and white, provably true and patently wrong. Still, part of the
reason this is so is that to show any results everything must be laid on the
table. CS and Math papers are therefore open by design and the world has
benefited. I hope the rest of science will move in this direction.

~~~
rajeevk
CS has similar problem, specially in AI fields. Researches often claim the
accuracy rate more than 95%, but in reality that is much lesser (in fact less
than 10% in most of the claims).

~~~
gordaco
Having worked in computer vision and robotics, I can attest to this.

~~~
jacobparker
Yes. Of the fields I've implemented algorithms from papers in, computer vision
seems to be particularly adverse to discussing the (often critical) downsides
of their algorithms. Often its something like "Camera calibration isn't
exactly perfect? Well this scene reconstruction technique simply wont work."

------
japhyr
This is one of the reasons I am happy to see what is going on with the Center
for Open Science [0]. The goal of the project is to open up the entire
research process, not just create open access to published results. They aim
to make open tools supporting the entire research process as well, through the
Open Science Framework [1].

One of the benefits of this approach is to provide peer review at each step of
the research process. There is also an emphasis on encouraging validation of
prior results, rather than having everyone focus on creating new research.
This is the focus of the Reproducibility Project [2].

The Center is just getting started, and I sure hope it takes off.

[0] - <http://centerforopenscience.org/>

[1] - <http://openscienceframework.org/>

[2] - <http://openscienceframework.org/project/EZcUj/wiki/home>

------
codex
This can be explained without bringing in accusations of fraud or incompetence
[1].

Basically, if you have a bazillion experiments running and only publish the
results which are significant, you still have a huge predictability problem,
because most of those statistically significant results will be due to chance
alone.

For example, assume that only 1% of cancer experiments produce something valid
and interesting, and that standard for statistical significance is the
industry standard 95%, or 1 in 20. If 100,000 hypothesis are tested, you'll
get 100,000 * 1% = 1000 valid results and 100,000 * 5% = 5000 results from
chance alone. In this example, only 16% of the published results will be
reproducible, because only 1000 / 6000 are valid.

[1] [http://sciencehouse.wordpress.com/2009/05/08/why-most-
publis...](http://sciencehouse.wordpress.com/2009/05/08/why-most-published-
results-are-false/)

~~~
dkural
this basic fact is understood by scientists and children alike. it is still
either incompetence (lack of middle school statistics) or fraud (ignoring the
above fact).

------
nogoodnik
The most interesting thing about all this is the fact that the researchers
were required by paper authors to sign non-disclosure agreements and agree not
to identify papers with non-reproducible results. This says a lot about the
confidence authors have in their papers, doesn't it?

Sorry but this isn't science anymore.

~~~
ttrreeww
It's good old capitalism making money from "treating" sickness. The longer the
better. A lifetime of "treatment" means a lifetime of profits.

Who cares about the cure.

------
baldfat
Pediatric Cancer gets only 4% of Federal funding and less than 1% of Private
Drug company research. Also American Cancer Society less than 5 cents on the
dollar goes to research.

Pediatric Cancer is the leading cause of death for 15 and under.

So to see this funding for cancer get thrown away by fraud is extremely
upsetting.

~~~
refurb
Those are some pretty misleading statistics.

First off, pediatric cancer is NOT the leading cause of death for those 15 and
under[1]. It's up there, but "unintentional injury" takes twice as many lives.

Second of all, not many children die (1200 per year), so even if something is
the leading cause of death, it doesn't mean it happens very often. Cancer
kills 10 times as many people between the ages of 35-44. If you add up all
adult cancers it's probably 20-30x the number of deaths.

Also, just because money goes to adult cancer, doesn't mean it can't help.
Most drug are developed for adults first and then tested in children. It's not
like people are ignoring pediatric cancer.

Before you get so upset you should really sit down and think these things
through.

[1][http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_A...](http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_Age_Group_2010-a.pdf)

~~~
Alex3917
"Most drug are developed for adults first and then tested in children."

It was also be unethical to develop drugs for children without first testing
them on adults, since children can't give informed consent.

------
rosser
Having read through these comments for the last several minutes, I'm left with
the overwhelming conclusion that programmers, on the whole, have about as much
to contribute to a discussion on the biological sciences as they do one on
economics.

------
pessimizer
There's an awful lot of money in cancer (research funding/patents), an awful
lot of diet/lifestyle handwaving in cancer research conclusions (that aren't
about treatments), and very little progress in making cancer more survivable.

Just those facts would lead me to assume that most of the published research
is bad. The great part is that 11% of the papers are reproducible, so the
science isn't at a standstill.

~~~
Mz
That isn't limited to cancer. Western medicine is driven by a lot of bad
mental and social models. Doctors only make money if you are sick. That
strikes me as a seriously flawed model right there.

------
psaintla
There are many reasons so many papers are not reproducible.

1.) The prospective data collection methods used in a lot of studies are
deeply flawed and worse aren't documented.

2.) The retrospective data used in many studies is poorly validated and the
quality is rarely a concern for most universities.

3.) The people who publish these papers are very intelligent, most far more
intelligent than I am, but you'd be surprised how many of them don't have a
very good handle on basic statistics. Not because they aren't smart enough to
learn but mostly because they have no interest in it.

4.) There is a lot of pressure put on researchers to publish papers even if
they aren't ready and as a result mistakes are made, there really needs to be
less of an emphasis on the quantity of papers that are published and more
emphasis on quality.

For anyone who might be interested in such things, if you ever find yourself
locked in a dungeon with nothing but a computer loaded with every journal
article ever published you can find articles involving similar cohorts from
the exact same universities that have different results.

------
pesenti
The problem is not that research is not reproducible. Researchers are humans
and full of biases. The problem is too much emphasis is put on novelty vs.
reproducibility. It took a long time for software engineers to recognize the
value of testing/validating vs. creating new code. In the same way the
research world needs to put new emphasis on reproducing/validating results.

~~~
return0
The problem is that too much emphasis is put on prestige and reputation than
on actual impact of the work. Coming from a physics background I always felt
that life scientists tend to be overly audacious in their claims, am I alone
in this?

~~~
pesenti
Is it only in life sciences? Remember the cold fusion and the faster than
light particle fiascos?

------
drorweiss
So maybe we need rename Life Sciences to Life Arts?

------
brandon_wirtz
This has been going around along with the Chemo Therapy doesn't work thread.

The problem with people like the author is they think all cancer is the same.
They try to apply information about Hodgkins to non-hodgkins. They think
"Breast cancer" is a disease, not a symptom. There over 40 different cancer
causes for breast cancer. And the treatments that work are primarily based on
the cause not the visible symptom.

This makes studies hard. Most people never learn the cause of their cancer. We
are not fortunate enough to always have a single known cause that says "yes
you worked in a nuclear waste plant for 6 years" and know that was the cause.

Chemo for example in Hodgkin's Lymphoma increase your 5 year survival chances
by 45%. But if you have Smoking related Lung Cancer it is less than 2%
difference.

Nature.com rarely produces articles that are informed. They push an agenda of
Holistic medicine at the expense of scientific research. I am all for non-
traditional medicine, but I don't discount the advances from University and
Clinical research.

------
eavc
There's nuance here that's going to be totally lost on the general public as
this becomes an anti-science talking point.

------
CatMtKing
Does reproducible in this context mean:

The paper did not document their steps sufficiently, so the authors were
unable to reproduce the experiment.

\-- or --

The paper relied on circumstantial evidence that was not well-understood
enough to be reproduced.

\-- or --

The authors were able to perform the experiment as documented in the paper,
but were unable to reproduce the results.

or something else?

------
JDDunn9
Perhaps part of this is statistical errors and selection bias. Most studies
are done with a 95% confidence interval, but only studies that reject the null
hypothesis are published. If we assume only 20% of all experiments reject the
null hypothesis, then we should expect ~25% of those to be fasle positives.
Journals might also be biased to include more sensationalistic results, which
are more likely to be statistical errors.

My guess is that if you included all test results, that reject the null
hypothesis or not, and are published or not, then the reproducibility rate
would be closer to 90%-95%.

~~~
gwern
> My guess is that if you included all test results, that reject the null
> hypothesis or not, and are published or not, then the reproducibility rate
> would be closer to 90%-95%.

No; you need to bring in power, not just assume an alpha of 0.05 and a base-
rate of real results of 20%. (Consider the recent neuroscience paper
estimating the experiments average a power of like 0.3...)

------
zmmmmm
Cancer is such a weird beast that I'm actually not as alarmed by this as it
might seem reasonable to be.

Only in the last year or so people have realized that how heterogeneous cancer
is. A patient biopsied 10 times from different parts of the cancer turns out
to have 10 different (but related) cancers. While such major findings are
still occurring about the underlying nature of cancer, it is not surprising
that studies are hard to produce; they don't even know which parameters need
to be controlled yet to make them reproducible.

------
dkural
I found a company to precisely fix this problem. Every dataset, every
analysis, every computation fully transparent, reproducible, accessible. You
can even embed it in your blog. For instance check out
<https://igor.sbgenomics.com/lab/public/pipelines/> if someone uses one of
those, and share their task, there is full transparency + instant replication.

------
teeja
The tradition in science publishing has been to leave out the nitty-gritty
details. Most specialists will know (or quickly deduce) most available methods
(unless the results are a radical improvement). As for the details, that's
what conferences and telephones and joining the department and gossip are for.

------
kailuowang
Scientific publications have a history of having papers contradictory to each
other for a long time. In medical research, findings are often based on
statics results from limited samples. From that perspective, it will be a big
surprise if they are more accurate than public polls.

------
viggity
After hearing similar things about psychology papers, this is rather
disconcerting. This is why I am a climate change skeptic, I don't know whether
it is happening due to CO2 or not, but I am confident that science can't be
confident when they can't do control experiments. You can't control for any
variable when it comes to the climate, let alone all the reasonable ones. We
have infinitely more capabilities to control variables when researching cancer
than we do with the climate of something as massive and complex as the earths
climate, and it turns out our confidence is cancer research may kind of suck.

one interested in the subject should check out feynman's discussion on the
psychological effect on scientific research. Millikan use bad assumptions for
his oil drop experiment to determine the charge of an electron, but nobody
would publish results that differed too much.
<http://en.m.wikipedia.org/wiki/Oil_drop_experiment>

~~~
kailuowang
The possibility of CO2 causing global warming isn't 100% but it isn't very low
either. It is the risk we are talking about, the risk of doing nothing and let
the man made green house (if there is one) turn earth into an irreversible
disastrous environment. Most CO2 emitting energy sources are not sustainable
anyway and many of them emit other proved pollution as well. There is really
nothing lost to going green energy, other than the cost of having some CO2
emitting energy reserved there.

~~~
vixen99
Without qualification, I'm afraid what you're saying is parlously close to
mere verbiage. Even the most ardent climate sceptic does not object to solar,
wind or water energy under certain conditions. What is 'Going Green' then? In
the UK it currently means, for instance, paying wind farms a million pounds
sterling a day to produce no energy at all. Meanwhile poor people die because
they can't afford to keep themselves warm thanks to horrendous energy bills
punped up by huge subsidies to the likes of windmill owners and owners of
land, hosting windmills.

~~~
kailuowang
"Going Green" does not equal to switch to renewable energy at the cost of poor
people's lives for gods sake. It simply means that recognizing the global
warming and have a plan to switch to renewable energy, for example, take some
of the hundreds of billions of dollars of the oil companies' profits to invest
on renewable energy.

~~~
viggity
proposed cap and tax carbon regimes disagree.

------
podperson
A scientific writer should state this result as 6 of 53, not 11%.

------
smsm42
So ironic that the paper about these findings is not reproducible itself
because the initial data set is not available for legal reasons.

------
kdazzle
It would have been 13% of papers reproducible if this blasted paper hadn't
ruined things and included itself

------
arrowgunz
Why not just say 6 papers!

------
podperson
This seems like an awesome example of Sturgeon's Law: 90% of everything is
crap.

------
languagehacker
What the hell? That's 5.83 papers. How do you reproduce .83 of something?

~~~
drharris
Um, 6 papers out of 53, significant digits give you 11%. Not that difficult.

~~~
ndr
I guess saying "11% of 53 papers" was _cooler_ than saying "6 out of 53". Meh.

~~~
drharris
Yeah, my guess is they used that form to imply 11% of all cancer research is
unreproducible. But in ironic fashion, they can't show their own data and thus
we can make no broad claims about it.

------
ttrreeww
Also, this is why evidence base medicine is flawed. What we want is science
based medicine.

------
ttrreeww
NaturalNews brought this back up recently.

