
Amgen publishes failures to replicate high-profile science - chriskanan
http://www.nature.com/news/biotech-giant-publishes-failures-to-confirm-high-profile-science-1.19269?WT.mc_id=FBK_NA_1601_NEWSBIOTECHFAILDATA_PORTFOLIO
======
skybrian
Publishing failures to replicate without blame seems like a good way forward,
similar to the practice of publishing postmortems in the computer industry.

~~~
will_pseudonym
I would say that this is more important than postmortems. This is new science
that's being published. It's overturning knowledge assumed to be true.

~~~
_delirium
Well, casting doubt on knowledge assumed to be true, which may lead to
overturning, but is still provisional when you're at the point where you have
one published experiment claiming a result, and one published failure to
replicate that result. Even correct results are not always successfully
replicated on the first replication attempt, including in "hard" areas like
physics/chemistry. Lots of possible reasons. One frequent case is that the
original results were, even if broadly correct, underspecified, i.e. there was
some key component of the original experimental apparatus or method that
wasn't sufficiently described in the paper, either due to an oversight or
because the original authors didn't know it actually mattered to the outcome.
Sometimes figuring out what that was makes it possible to replicate the
result, although other times it turns out to undermine the original results
(e.g. the entire result turns out to be due to some specific impurity in a
specific supplier's equipment). But in any case still very useful.

~~~
mirimir
Yes! Been there, done that. So much of lab technique is hard to explain in
words. People do postdocs and sabbaticals in order to learn technique. And
then there are uncontrolled variables. Classic example:

> When a student had difficulty in crystallizing a compound [professors] would
> simply shake their beards over the flask containing the offending substance.
> Then, lo and behold, after nucleation had done its job, crystallization set
> in. Gerhard believed that the beard of an Adolf von Baeyer or an Otto
> Wallach could indeed be a source of crystals of every conceivable space
> group.

[http://www.improbable.com/2011/02/09/legend-of-
chrystallogra...](http://www.improbable.com/2011/02/09/legend-of-
chrystallographers-beards/)

[https://www.tes.com/teaching-resource/are-bearded-
chemists-b...](https://www.tes.com/teaching-resource/are-bearded-chemists-
better-at-crystallization-6397626)

[http://tdl.libra.titech.ac.jp/hkshi/xc/contents/pdf/11709748...](http://tdl.libra.titech.ac.jp/hkshi/xc/contents/pdf/11709748X/5)

------
nonbel
>'"I believe the main risk of a publication venue like the F1000 channel is
that it becomes a place for “bashing" good science, because biological
experiments are complex and beset by many variables that are hard to control.'

Funny how rare it is to see mention of this in original papers and press
releases. When is the last time you read a biomed article where the authors
said something along the lines of "we excitedly await the independent
verification of our findings".

------
pcrh
While I applaud this effort, it does concern me that they have chosen to
publish only three of the claimed 47 failures to replicate.

The Begley that spawned in part this debate was seriously flawed for two
reasons: the first is that the experiments they performed were not described
in _any_ detail, and the second is that the experiments they choose to repeat
were clearly not selected at random.

Indeed, the limitations of their initial study may underlie the skepticism
that greeted their claim that 89% of studies were not reproducible. Most
scientists expected a certain proportion of studies to be difficult to
reproduce, but nothing approaching 89%.

So, perhaps publishing details of ~6% of their study is an improvement, but
again, ironically, it is not a statistically relevant sample.

~~~
capnrefsmmat
According to the article, these three articles were not part of the original
set:

> In 2012, Amgen researchers made headlines when they declared that they had
> been unable to reproduce the findings in 47 of 53 'landmark' cancer papers.
> Those papers were never identified — partly because of confidentiality
> concerns — and there are no plans to release details now either, says Kamb,
> who was not involved with that publication. He says that he prefers to focus
> on more-recent publications.

~~~
pcrh
The Begley paper and the whole debate over reproducibility spawned a
worthwhile introspection on behalf of the journals, several of which have
adapted the information required from authors in response.

However, I am distinctly unimpressed by Amgen's failure to live up to the
standards it expects of others; not releasing the information required to
assess their original claims being their worst offense. In this series, one of
the studies, as mentioned in the article, is even a repetition of 4 [1,2,3,4]
earlier failed attempts to reproduce the original claim that bexarotene caused
a reduction of amyloid in mice [5]; while worthwhile, it is not an impressive
contribution.

[1]
[https://www.ncbi.nlm.nih.gov/pubmed/23704555](https://www.ncbi.nlm.nih.gov/pubmed/23704555)
[2]
[https://www.ncbi.nlm.nih.gov/pubmed/23704554](https://www.ncbi.nlm.nih.gov/pubmed/23704554)
[3]
[https://www.ncbi.nlm.nih.gov/pubmed/23704553](https://www.ncbi.nlm.nih.gov/pubmed/23704553)
[4]
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4086452/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4086452/)
[5]
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3651582/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3651582/)

------
EvanPlaice
If only we could prevent the news from making discoveries public before
they've had a chance to be verified. Even if the news media publishes a
retraction, it usually happens long after the initial results are already
spread and reinforced as common knowledge.

I'm only just an armchair opinion but I get the impression that a some people
are gaming the system for profit rather at the expense of intellectual
honesty. The prospect of the field of science incentivizing 'junk science' is
really unsettling.

------
zymhan
There was a Planet Money episode about this recently -
[http://www.npr.org/sections/money/2016/01/15/463237871/episo...](http://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-
experiment-experiment)

I loved the part where one of the researchers who couldn't replicate other
experimental results, had some of his experiments found to be not-
reproducible.

------
nonbel
Is there a simpler version of this site for hosting the data:
[https://osf.io/sqb2x/](https://osf.io/sqb2x/)

