
Plan to replicate 50 high-impact cancer papers shrinks to just 18 - nonbel
http://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact-cancer-papers-shrinks-just-18
======
sytelus
Every academic cries about this but no one wants to accept an _obvious_
solution: Add reprodicibility criteria for peer review. Right now even Tier 1
conferences do not have this criteria in their peer review system. The
reviewer simply can't reject paper because authors didn't provided
reproducibility information!

This is truely a plague in research right now. I've came across quite a few
instances where authors told me that their experiments weren't reproducible
even for them! They do not note this in paper because ultimately everyone
needs to show some result for the funding they received (i.e. "published
paper"). They obviously never want to share any code, data files, hardware
etc. On one instance, author wrote me back that they can't share code with me
because they lost all the code because their hard drive crashed!
Reproducibility is a fundamental tenant of doing scientific work and this is
actively and completely ignored in current peer review system.

I think conference chairs needs to take stand on this. We get now 4X to 8X
papers in tier 1 conferences. Reproducibility could be a great filter when
area chairs are scrambling to find reasons to reject papers. Sure, there will
be papers where very specialized hardware or internal infrastructure of 10,000
computers were used. But those papers would be great for Tier 2 conferences.

~~~
ThePhysicist
I think this is a great idea at least for fields where reproducibility is
comparatively easy to achieve, e.g. computer science: Include your data, code
and run environment (using a suitable technology) with the paper to give
reviewers a chance to "play" with your experiments (if they actually have the
time for this).

In other fields such as experimental physics or biology it might be more
difficult to achieve the same effect though, as experiments are quite hard to
repeat in general: For example, in my former field (experimental quantum
computing), building the setup and fabricating the sample required for a given
experiment could take years of effort, making it almost impossible to "just"
reproduce someone's work for the sake of verification.

That said, in experimental quantum physics the exciting results tended to get
replicated within a couple of years by different teams anyway, not because
these teams wanted to verify the results but rather because they wanted to
build their own experiments on top of them (I imagine this is similar in other
fields). Another natural way of exchanging knowledge and improving
reproducibility was via the exchange of PostDocs and PhD students: If you do
good work in one group, another group will usually be very eager to give you a
position so that you can help them to set up the experiments there as well.
I'd even argue that this is one of the main mode of knowledge dissemination in
experimental science today, as most research papers are just extremely hard to
reproduce without the specific -and often not encoded- knowledge of the
individuals that ran the experiments.

I'm not sure though if it is practical to document everything in a research
paper in such a way so that any person can reproduce a given experiment, as
many of the techniques are very specialized and a lot of the equipment in the
labs (at least in physics) like sample holders, electronics, chip design
templates and fabrication recipes is custom-built, so documenting down to the
last detail would take years of effort.

That's why (IMHO) written PhD theses are so important, because that's kind of
the only place where you can write 200-400 pages about your work, and where
you can include minute details such as your chip fabrication recipe, a
description of your custom-built measurement software and a detailed summary
of the experimental techniques used in your work. In that sense, PhD theses
are probably more important to reproducibility than short papers.

~~~
dekhn
Your description of experimental quantum physics exactly matches the work done
in state of the art biology. It basically takes 10-20 years of training and
excellent brains to reproduce your competitor's paper in a way that lets you
build on top of it. Many people who complain that papers aren't reproducible
just don't have the skill to move the state of the art, because it's become so
esoteric and challenging to run the critical experiments.

I used to be a "everything must be replicable by even the simplest of people"
person, but I changed to "for progress to be made there must be <X> competent
people who can reproduce challenging experiments and run new ones".

~~~
BoiledCabbage
> because it's become so esoteric and challenging to run the critical
> experiments.

Maybe more people would be able to run the experiments if papers detailed how
to do so. The knowledge would no longer be esoteric.

------
buchanae
I have been working on reproducing computational biology papers from the
cancer field lately. I am very frustrated. When the inputs and outputs are
machine-readable data, there's no excuse for not making your work
reproducible, in my opinion. Often the problem is plain laziness and
disorganization.

One major problem is that there's not much real incentive to make your work
reproducible. Money granting organizations favor researchers breaking new and
exciting ground, not those rehashing an already published method. Publishers
don't require reproducible methods, and reviewers don't have the time, desire,
nor expertise to do an in-depth methods review.

Wet lab experiments are 1-2 orders of magnitude more expensive and difficult
to reproduce, that's true, but we're not even getting the basics right!

~~~
buchanae
And another thing! Why should a project be done when it gets published?! What
kind of software project would make one release and call it good forever?
Nonsense.

~~~
fmap
In an ideal world you would be right, but that's not how the funding
structures for academic research are set up.

Think of a research lab as a company that gets paid per prototype and then has
to market the concept for the next prototype in an infinite loop. If you can't
package up what you're doing into a sequence of small prototypes then you're
not getting paid.

~~~
ajuc
The requirement for publishing in a scientific journalshould be opensourcing
all the code used.

You can't expect the results to be reproductible 20 years later otherways.

~~~
skummetmaelk
20 year old code that has not been maintained most likely would not run on a
modern system anyway, unless it is extremely simple.

~~~
azernik
Research labs operate on a very slow tech upgrade cycle anyway; since code is
handed down from assistant to post-doc to grad student, complete rewrites
would take up a significant fraction of a person's time at any given lab, and
so codebases are often as long-lived as the labs in which they live. We're
talking decades-old FORTRAN here. Running twenty-year-old software is a
barrier for some labs, but not all.

~~~
_Wintermute
My experience is the opposite, that a lot of labs are running R code that
won't work 6 months later, and no one actually recorded the package version
numbers that were used.

~~~
azernik
TBF, the experience I have is with physics and mechanical/civil engineering
labs, where there historically hasn't been much of a reliance on R. And in any
case said experience is several years out of date.

Speaking of, I haven't played with R - what are its standard methods for
handling dependencies? I'm particularly enamored of the pip and npm way of
doing it, where you create a version-controlled artifact (requirements.txt and
packages.json, respectively) that defines your dependencies. Does R not have a
similar system, or do people just not use it?

~~~
_Wintermute
R isn't fantastic for handling dependencies. If your code is bundled up as a
package then you can specify version numbers for your dependencies, but I
don't know of any equivalent to `pip freeze` to actually list these.
Installing anything other than the latest version of a package is a bit of a
pain, and setting up environments for separate projects is pretty much unheard
of.

I'm a bit bitter about the whole "writing reproducible code in R", as I'm
currently wasting a lot of time trying to get R code I wrote at the start of
my PhD to run again now I'm writing up.

~~~
nonbel
Theres tools for this:

[https://rstudio.github.io/packrat/](https://rstudio.github.io/packrat/)

[https://stackoverflow.com/questions/43018752/version-
control...](https://stackoverflow.com/questions/43018752/version-control-
alternatives-to-packrat)

------
gnusci
This is a well known problem of scientific publications now-a-days:

[https://www.nature.com/news/1-500-scientists-lift-the-lid-
on...](https://www.nature.com/news/1-500-scientists-lift-the-lid-on-
reproducibility-1.19970)

[https://en.wikipedia.org/wiki/Replication_crisis](https://en.wikipedia.org/wiki/Replication_crisis)

Papers have become a target for success, so scientist need publications for
better status and remuneration:

[http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-
rev...](http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-reviewed-
papers-go-global)

A new scientific "mafia" is in place around the world:

[https://www.technologyreview.com/s/608266/the-truth-about-
ch...](https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash-
for-publication-policy/) [https://retractionwatch.com/2017/08/10/paid-publish-
not-just...](https://retractionwatch.com/2017/08/10/paid-publish-not-just-
china/)

So papers have become a good business, no the way to disseminate outstanding
research results.

~~~
lsh
> So papers have become a good business, no the way to disseminate outstanding
> research results.

That's awfully cynical and over-broad, but I agree to a point. Greedy and
unscrupulous publishers are part of the problem, but so are lax or
unprincipled scientists eager for prestige and a career-making publication in
a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy
way to cut it. Perhaps more emphasis on replication post-publication? Perhaps
a reputation system for unethical publishers or scientists?

~~~
timr
_" Greedy and unscrupulous publishers are part of the problem, but so are lax
or unprincipled scientists eager for prestige and a career-making publication
in a top tier journal."_

That's just incredibly unfair. There are some fields and methodologies where
p-hacking and cherry-picking have been a problem, but the _primary_ reason
that papers aren't reproducible is just noise and basic statistics.

As a scientist, you control for what you can think of, but there are often way
too many variables to control completely, and it's probable that you miss
some. Those variables come to light when someone else tries to work with your
method and can't reproduce it locally. However, _real scientists_ don't stop
and accuse the original authors of being "unprincipled" \-- nine-point-nine
times out of ten, they work with the original authors to discover the
discrepancies.

It isn't surprising at all to actual, working scientists that most papers are
impossible to reproduce from a clean room, using only the original paper. It's
the expected state of affairs when you're working with imperfect, noisy
techniques, and trying tease out subtle phenomena.

~~~
michaelt

      There are some fields and methodologies where p-hacking
      and cherry-picking have been a problem, but the primary
      reason that papers aren't reproducible is just noise and
      basic statistics.
    

It's possible to imagine a version of academia where results that can be
attributed to noise don't get published.

~~~
wyattpeak
Is it? That doesn't seem at all obvious to me. In fact it seems decidedly
impossible.

Almost any result could in principle be attributable to noise; where are you
planning to source all of the funding to run large enough studies to minimise
that? And no matter how large your experiments or how many you run, you're
still going to end up with some published results attributable to noise since,
as GP says, that's the nature of statistics. By its nature, you cannot tell
whether a result is noise. You only have odds.

I'm not saying there aren't problems with reproducability in many fields, but
to suggest that you can eliminate it entirely is naive.

No, not naive - wrong.

~~~
TeMPOraL
> _Almost any result could in principle be attributable to noise; where are
> you planning to source all of the funding to run large enough studies to
> minimise that? By its nature, you cannot tell whether a result is noise. You
> only have odds._

Well, with a single paper the odds indeed are that it's noise. That's why we
need reproduction. Now of course a paper needs to be published for it to be
replicated later. But the paper (and/or supplemental material) should contain
_all possible things_ the research team can think of that are relevant to
reproducing it - otherwise it's setting itself up to be unverifiable in
practice. Papers that are unverifiable in practice should not be publishable
at all, because a) they won't be reproduced and thus it'll be forever
indistinguishable from noise, and b) there's no way to determine whether it's
real research, or a cleverly crafted bullshit.

~~~
wyattpeak
I don't disagree with any of that, although I'd stick a big _citation needed_
on the implicit suggestion that there's a large group of scientists who aren't
making a good-faith effort to ensure that their successors will have the
information they need to reproduce (that is, after all, what a paper is).

My issue is the flippant and silly claim that "[i]t's possible to imagine a
version of academia where results that can be attributed to noise don't get
published".

~~~
saalweachter
I think this is actually something that can be experimentally examined.

Take a sampling of a large number of papers, give them some sort of rating
based on whether they provide enough information to reproduce, how clear their
experimental and analytical methodology was, whether their primary data and
scripts are available, etc, and then look at that rating versus their
citations.

Hopefully, better papers get more attention and more citations.

(And yeah, "peer review" as it is done before a paper is published is not
supposed to establish a paper as _correct_ , it is supposed to validate it as
_interesting_. Poor peer review ultimately makes a journal uninteresting,
which means it might as well not exist.)

~~~
wyattpeak
That sounds like a very interesting idea. At the least, it would be
interesting see the major classes of reproducibility problems. And there may
well be a lot of low-hanging fruit, as the comments on this page suggest about
data corpuses in computational fields.

------
kayhi
I wrote my concerns about replicating these studies a couple of years ago [1].
In short, most papers don't have enough details to allow for a third party to
replicate them.

Problem areas include detailed protocols and reagents used. If you don't what
someone did exactly and what they used then replicating it is going to be very
difficult.

[1] [https://www.linkedin.com/pulse/scientific-reproducibility-
re...](https://www.linkedin.com/pulse/scientific-reproducibility-reagent-
variability-sean-seaver/)

~~~
drb91
> If you don't what someone did exactly and what they used then replicating it
> is going to be very difficult.

Why do journals accept these papers? If you can just make shit up, it kind of
makes the whole value of a paid journal moot. What value do they add if not
reviewing the damn paper?

~~~
karthikb
What is repeatability? 10% of the time? 50% of the time? 99% of the time?
99.9999% of the time?

Even medications made in high volume processes have rejected units (and
sometimes, batches). Process drifted, machine failed, raw ingredients had some
issue.

A scientific paper at the very early stage is meant to show that something is
possible if the conditions are _just right_. It may be actually outside the
capability of the lab to document (or measure, or even know) every variable
that makes up these conditions. One of the reasons there is such a chasm
between lab and practice is that as people dive in for deeper review and
productization, they find the process is too finicky, the applicability too
narrow, etc. But the scientific paper is the first step in this process of
discovery.

That said, it's critical that authors document whatever they _do_ know or have
control over. Electronic publishing, tools like Github, etc, all make the
barrier for disclosure much lower.

~~~
stonemetal
Given that P <= 0.05 is the usual standard, I would hazard that if it can't be
reproduced at least 95% of the time then you miscalculated P.

~~~
kgwgk
This is no what a p-value means.

~~~
stonemetal
It isn't but it kind of is. It is a 95% chance that you should reject the null
hypothesis. For example if we flip a coin a 5 times and get heads every time
that gives you a p value .03 for rejecting the null hypothesis that the coin
is fair. If I run the five coin flip experiment a 100 times and never once
reproduce a significant result is it because gee science is hard, or is it
because you hit one of those 3% of cases. The same goes for real science if
you can't reproduce it more often than not then you might be close but you
haven't figured it out yet.

~~~
kgwgk
Ok, so we can agree that if you got positive a result with p<0.05 and you
repeat the experiment 100 times you expect to reproduce a positive result
(i.e. getting a new p-value below 0.05)

a) if the null hypothesis is true, 5 times

b) if the null hypothesis is not true, anywhere between 0 and 100 times
(depending on what is the true alternative)

This is quite different from your previous assertion that the expected number
of successful replications would be 95 out of 100.

------
Palomides
>says total costs for the 18 completed studies averaged about $60,000,
including two high-priced

that seems really cheap to me based on my brief experience with lab research.
replicating 18 studies is a great achievement.

~~~
jonlucc
It's not terrible for in vitro studies, but that would be a stretch for in
vivo studies at a CRO, especially for oncology, where lots of studies require
tumor grafts. It's not my area, but I think those mice can run $1000 and up
per animal. The cheap, more common strains are ~$25 each. I haven't looked at
the papers they're trying to replicate, but in my lab, it's not unusual to
have 60 mice in a study, so you can see how just getting animals into your
facility can be expensive.

~~~
dingo_bat
$1000 per mouse? Seems really expensive.

~~~
amelius
Annual cost of small mammal ownership: $300

Source:
[https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5](https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5)

Add to that expenses for the experiment, and $1000 seems not far fetched.

~~~
mattkrause
That’s for a pet.

While labs have huge economies of scale—-I worked somewhere with a quarter
million mice—-the mice aren’t just random field mice. Some of the genetically
engineered ones are hundreds of dollars a pair (and possibly more if custom,
raised under unusual conditions, etc). Xenografts need skilled labor and very
clean conditions for the immunocompromised mice. $1000 seems a little high,
but not much, especially if the work isn’t being done by an underpaid grad
student.

------
DoctorOetker
For the price of 10% of subsidy, you could sponsor reproducing ~10% of
discoveries (proportional to the subsidy they took). Just some simple public
provably random commitment protocol.

While you are doing the experiment, everyone knows theres a 10% probability of
eventually being selected, so healthy pressure to make sure everything is
properly documented, and everyone looks out to detect fraud by collaborators.

I don't see the problem, unless its the 10% price hike... if 10% is too much,
just do 90% of the usual number of projects. I'd prefer 10% less projects if
it ensures much higher reproducibility rates...

~~~
Q6T46nT668w6i3m
I don’t understand your math. Typically, for one paper, if you’re trying to
reproduce the experiment (rather than just the analysis) the cost is closer to
110% because, for many researchers, their papers build off of their previous
papers so they already have expertise with a particular assay, protocol,
technology, etc.

~~~
DoctorOetker
The math stays the same, even if the numbers change: with your number of 110%
of the original project price for reproduction by others we get the following
price hike 110% * 1/10 = 11%

big deal, 11% more expensive science, but 1 out of 10 results get reproduced,
stoichiocratically, so you don't know if it will be reproduced until after
publication...

Also: your comment about reproducibility details being scattered over the
previous work of the original authors... as I said, in a world where we use my
system, you are incentivized to put all details for reproduction within the
paper, since you wouldn't want to risk possible reproduction by others to fail
simply because they didn't read your previous papers...

------
lsh
A direct link to the results:
[https://elifesciences.org/collections/9b1e83d1/reproducibili...](https://elifesciences.org/collections/9b1e83d1/reproducibility-
project-cancer-biology)

There have been several news items on Hacker News recently about academic
publishing, reproducibility in general and pre-print servers and I would love
it if eLife got some more attention, for better or worse, for the excellent
work they are doing and helping to fund.

To those biology researchers in the comments, please submit your work! even
failed results! Even if it's software heavy. Editors are listening closely to
your feedback.

edit: disclosure, I work for eLife

~~~
chmaynard
Does eLife have an RSS feed? I can't find one listed on your website.

~~~
rossmounce
Yes, various RSS feeds.

Alert options explained here:
[https://elifesciences.org/alerts](https://elifesciences.org/alerts)

early versions:
[https://elifesciences.org/rss/ahead.xml](https://elifesciences.org/rss/ahead.xml)
final published versions:
[https://elifesciences.org/rss/recent.xml](https://elifesciences.org/rss/recent.xml)

------
textmode
There is an unwritten rule in biology that if you publish a paper that refers
to uses of certain reagents that are not commercially available, then you are
obligated to provide those reagents to other investigators who read the paper
and request them. There can also be an expected obligation that the other
researchers will share any data they generate using the reagents with the
original authors.

Outside of biology, I have seen many "academic" papers published on computer-
related topics that refer to software programs developed by the papers'
authors that are crucial to the research but not publicly available. Is there
any similar unwritten rule to that in biology where another researcher reading
these papers can request a copy of these programs from the authors?

Obviously, in many cases other researchers cannot replicate and verify
findings without access to the same research tools used in the published
papers.

~~~
your-nanny
I've frequently asked for researchers code. most are excited someone is
interested in their work, if a little nervous, or embarrassed by their coding
umm style (neuroscience)

------
Alex3917
> organizers realized they needed more information and materials from the
> original authors;

Then the study already isn't replicable by definition, so why waste time
asking the original authors? Just mark it down as 'not replicable' and move
on.

~~~
jonlucc
I'm not so sure this is the right way to go about it. They give the example of
cell densities, but there are also tons of things that just are never reported
in in vivo papers, including the light cycle in animal rooms (mice and rats
are nocturnal, so some labs make their rooms dark during the day), the
preparation of treatment compounds, information about animal ages, handling
procedures, etc. In addition, many papers have length limits, so you can only
include so much.

~~~
Alex3917
Sure, all of which make the research non replicable. That stuff is all
important, you can't just discount it because it's the norm not to include it.

~~~
flukus
They should be replicable despite a lot of that, if they aren't then then the
original study is garbage.

~~~
Alex3917
Getting the same result doesn’t mean a study is replicable or a result is
reproduceable. If you don’t use the same methodology it’s meaningless.

~~~
irq11
No.

Nobody ever reproduces papers exactly, because you can’t. There are too many
variables, and even though you try to control as many as possible, you can
still be blindsided by the random variate that you didn’t anticipate.

Scientific results that are robust to random variation are the important ones.
The ones that can only be reproduced exactly as specified are most likely to
be “meaningless”.

------
nonbel
I was mentioning this in another thread and came across the new info I hadn't
heard about yet. Here are some previous articles about this:

March 2012:
[https://www.nature.com/articles/483531a](https://www.nature.com/articles/483531a)

June 2015:
[http://www.sciencemag.org/content/348/6242/1411](http://www.sciencemag.org/content/348/6242/1411)

Dec 2015: [https://www.nature.com/news/cancer-reproducibility-
project-s...](https://www.nature.com/news/cancer-reproducibility-project-
scales-back-ambitions-1.18938)

Jan 2017: [https://www.nature.com/news/cancer-reproducibility-
project-r...](https://www.nature.com/news/cancer-reproducibility-project-
releases-first-results-1.21304)

------
rrock
Wait, they thought they could replicate 50 papers at the cost of $25k each,
and all in the span of one year?

Not a chance. I don’t think these guys have any idea what they’re doing. I
feel sorry for the funders.

~~~
LyndsySimon
They thought that because they’d already done 100 in another field:
[https://osf.io/ezcuj/](https://osf.io/ezcuj/)

I worked there for three years :)

~~~
rrock
I'm going to guess that psychology research is far less expensive. To give you
a sense of perspective here, $25k will get you something like 4 months of a
full time employee at the "fresh out of college" experience level, including
benefits and research consumables. That also assumes that you have all of the
equipment in place.

If all there doing is running a few western blots, because that's all they can
do on this budget, well then ok. But I'll venture to guess that that's a far
cry from reproducing the most important results in these papers.

~~~
LyndsySimon
As far as I know, the problem wasn't primarily one of underestimating cost or
effort - it was one of funding. The non-profit that did this has downsized
quite a bit since the RP:CB began and has been seeking other funders.

------
adiusmus
Initiate Rant.

Most science isn’t.

Most of it is a set of nearly fanatical beliefs about reality only vaguely
related to facts agreed through consensus by authority. Look at how strongly
doctors fought against germ theory. They saw no reason to wash their hands for
delivering a new born after having spent their morning cutting open cadavers.
Their response was anything but scientific even in the face of easily tested
claims and results.

If research can’t be reproduced it’s just a story with interesting data. I
doubt most scientists using statistics would be able to provide the alpha / p
value / confidence interval etc for their hypothesis.

When the bar is set so low should we be surprised by low quality?

Rant complete.

Initiate beer.

~~~
southern_cross
Hand washing may still be a problem. I saw a blurb recently which stated that
when doctors know that they're being monitored for using proper hand washing
protocols, compliance was near the top end of the scale. But when they thought
they were no longer being watched, compliance fell by something like two-
thirds. So in other words, doctors (and no doubt others) probably still aren't
anywhere close to doing it right.

~~~
Engineering-MD
That would be because “doing it right” is completely over the top. You are
suppose to wash your hands before and after each patient interaction, which
includes touching their notes, or coming into a patient environment without
touching anything. This is impractical, as it would slow everything down so
much, and would make carrying anything impossible, not to mention the negative
impact of doctor-patient interactions. It also doesn’t match how patients then
move around, touching objects around hospital and nullify the whole process.
Doctors and nurses tend to common sense check when a hand wash/alcohol gel is
needed, which I think is a good thing. I think some go too far the other way,
but there will always be variation.

~~~
fabricexpert
How can you common sense check when handwashing is required? You can't see
germs, it's not possible.

The handwashing protocol is there to stop people getting sick, circumventing
it because you're a doctor and you know better is not going to help.

~~~
adiusmus
Commonsense is easy to develop. Just invite anyone who doesn’t believe in
germs/handwashing to eat a meal using a public toilet seat as a plate.

It’s amazing how real something becomes when beliefs intersect with
consequences, especially those involving self-preservation.

------
fabian2k
One big issue with making papers reproducible is that the incentives to do
that simply don't exist in many cases. It is often not rewarded if you put
additional effort there, and usually also not punished even if you don't do
the minimum the journals require in their official rules.

The official rules are slowly changing, and the funding agencies tend to
require that scientists make raw data available, and put effort into making
their experiments reproducible. But the reality is changing much more slowly
than the rules.

~~~
danieltillett
Actually the incentives encourage non-reproducibility. Any effort put into
reproducibility increases the chance that the results won't be publishable.
The worst outcome is not publishing crap, it is not publishing at all. This
needs to change.

------
timr
Please don't ignore this very important quote from the article:

 _" In fact, many of the initial 50 papers have been confirmed by other
groups, as some of the RP:CB’s critics have pointed out."_

This article is staying assiduously neutral, but one perfectly valid
interpretation is that _the original initiative was flawed_ in a way that was
predicted by the initiative's critics. Science is _routinely_ reproduced --
just not by labs working in isolation, using publications as clean-room
instruction books. This is the sort of thing that _programmers_ believe about
science, not something that scientists believe themselves.

There are a great many serious, legitimate scientists who believe that this
"reproducibility crisis" is verging on irrational, and it's important to
consider their arguments. It's particularly scary to me that so many comments
here are dovetailing with the sort of nonsense you encounter on anti-vax and
global warming denial forums. We're literally gaslighting the process that has
done more to advance society than any other in human history:

[http://www.pnas.org/content/115/20/5042](http://www.pnas.org/content/115/20/5042)

 _" The discovery that an experiment does not replicate is not a lack of
success but an opportunity. Many of the current concerns about reproducibility
overlook the dynamic, iterative nature of the process of discovery where
discordant results are essential to producing more integrated accounts and
(eventually) translation. A failure to reproduce is only the first step in
scientific inquiry. In many ways, how science responds to these failures is
what determines whether it succeeds."_

~~~
southern_cross
And the attitude being described above seems to me to border on religious
"just trust us" dogma. I mean, one of the big things that's supposed to
distinguish real science from pseudoscience and such is the ability to fairly
easily and routinely reproduce its results. And if that's not happening, and
in fact it's being actively discouraged, then that's a real problem!

 _" In fact, many of the initial 50 papers have been confirmed by other
groups, as some of the RP:CB’s critics have pointed out."_

I don't know that I would be willing to take a statement like that at face
value. I once read of a scientist who claimed that there was absolutely no
need to try and independently reproduce his work, since it was being routinely
reproduced in college labs everywhere as part of other experiments - or
something like that. To me that meant that it should then be trivial to
reproduce these results when attempted by trained professionals who were
specifically trying to do that very thing. So why the reluctance to allow that
to happen?

~~~
timr
_" And the attitude being described above seems to me to border on religious
"just trust us" dogma. I mean, one of the big things that's supposed to
distinguish real science from pseudoscience and such is the ability to fairly
easily and routinely reproduce its results."_

Nowhere is it written that you should be able to "easily and routinely"
reproduce a scientific result. It's damned hard for skilled scientists to
reproduce most scientific results. And yes, it's utterly impossible for
amateurs. I'm sorry if that makes you uncomfortable, but it's the truth.

There are many things in life that you routinely accept on authority: When you
fly in a plane, that it isn't going to crash into the earth. When you turn on
the tap, that clean, safe water comes out. When you go to your doctor, that
she is prescribing you medication that will help you, not hurt you. When you
vaccinate your children, that you are protecting them from horrible diseases.
I'm sorry that it bothers you, but "science" is just one more thing in the
world that you're going to have to accept, because nobody -- scientist or
otherwise -- can independently verify all of human knowledge.

If it makes you feel better, you can look around, realize that we're no
starving from famine, or dying from minor cuts, or poisoning ourselves with
lead, or dying of Polio or Smallpox, and you can try to convince yourself that
in the long run, the _method works_. Because _that 's_ the argument you're
missing. You don't have to take my word for it, or anyone else's. It's a
system. The system works. You just don't fully understand _why_ it works, and
you're not willing to read when a group of scientists write a long article
that tries to explain it to you. Because you'd rather believe that explanation
is "dogma".

 _" I don't know that I would be willing to take a statement like that at face
value."_

Nobody is stopping you from digging into it. If you're really so bothered by
it, I encourage you to follow up.

 _" So why the reluctance to allow that to happen?"_

There is no reluctance, whatsoever. Nobody is trying to stop these people.
They're just beginning to realize that their approach is a lot damned harder
than they originally expected, and those of us who _knew it would be_ are
pointing it out.

~~~
southern_cross
I understand where you're coming from here so I don't want to belabor the
point, but what you're actually describing is more of a faith-based system
rather than one based on science! And I for one am simply no longer willing to
take most of what (for example) shows up in the peer-reviewed literature these
days based primarily on faith, and I haven't been for several decades now. The
fact that you and so many others seem so readily willing to do so makes me
seriously question your critical thinking skills.

And I don't blindly trust that "the system works" really at all to the extent
that's so it often claimed to. An awful lot of what gets published these days
tends to just not hold up to close scrutiny, and often what appears to be
groundbreaking research may just kind of up and disappear with no further
trace soon enough, without ever directly or even indirectly leading to a
practical new product or a new drug or whatever. I understand that this kind
of thing is going to happen, and it may happen quite a bit if you're doing
_really_ groundbreaking stuff, but in general this should probably be more the
exception than the rule.

But yes, I do understand how the _method_ works, and how the system that's
supposedly based on that method works (when it actually does work), and also
when it doesn't work and why, and what the motivations might be to try and
pretend that it works even in situations when it actually doesn't. And rather
than blindly defend it, maybe you try a bit harder to understand how it
actually "works", too.

------
atdt
Why, apart from expedience, do designers of experiments share an institutional
affiliation with the experimenters themselves? I wonder what would happen if
the two groups were far apart, and if communication between them restricted to
the transmission of an experimental protocol and the reception of raw
observations.

~~~
nonbel
This is a really good idea. I bet it is not as hard as some of the "real
scientists" claim.

------
Avshalom
turns out we can't raise the money to replicate researchers who barely had the
money to do what they did.

~~~
nonbel
I'd honestly say if the study isn't worth replicating then don't fund it to
begin with. Its like funding it enough to get some animals or cells into your
lab but then not enough for gloves and pipets and computers.

~~~
Avshalom
That sounds great in abstract but every study is subject to that logic,
including studies studying studies. Which means that every study should be
matched by an infinite regression of studies, which is untenable to put it
mildly and mathematically fucking ridiculous to put it conservatively.

~~~
nonbel
There is no infinite regression, what are you talking about?

Its that at least two groups try to measure the same thing under the same
conditions (as similar as you can get them).

~~~
Avshalom
how do you know that those two groups aren't collaborating for the same
result? Or biased in the same direction of the result?

you don't. You either add a 3rd party (4th... 5th...6th... Xth)or accept the
conclusion. If you add a third party how do you know they are trust worthy:
infinite regression.

~~~
nonbel
Its not about perfection, its about taking the simple, common sense, and time-
tested precautions that gave us the technological marvels we enjoy today.

But of course the more independent, even adversarial, the two groups are the
better. Like in this story it mentions the "critics" of this project claim
other labs (their friends?) have already replicated these studies, apparently
using the super secret protocol they couldn't explain to this group.

I'd compare it to using sms 2FA. Is it perfect, no. But it is far, far better
than no 2FA at all. And there's nothing stopping you from putting your banking
app on the same phone you use for 2FA.

------
doener
[https://news.ycombinator.com/item?id=17662185](https://news.ycombinator.com/item?id=17662185)

~~~
jwilk
(same article, 0 comments)

------
anticensor
It turned out remaining 32 are unworthy to replicate.

------
Gatsky
Replicating cancer papers: This is one of those ideas which although pointless
and impractical, is seemingly impossible to criticize.

Now the leaders of this project are basically saying - we can't do difficult
experiments, and can't replicate studies which have already been replicated by
others.

What have we learned exactly, except that experimental biology is difficult?

~~~
pietroglyph
> pointless and impractical

These papers underlie a lot of modern research and treatments. If they aren't
reproducible, the papers' results, and other papers that use their results
come into question. This is a serious, documented problem [0].

> What have we learned exactly, except that experimental biology is difficult?

The problem wasn't that the project couldn't do difficult experiments, it was
that not enough information was provided by the original papers. _That 's_
what we learned here.

[0]:
[https://en.wikipedia.org/wiki/Replication_crisis](https://en.wikipedia.org/wiki/Replication_crisis)

~~~
Gatsky
I disagree with your characterisation of the problem and the proposed
solution.

Cancer research is not an edifice constructed on foundational results. It is
not like physics in this regard, where for example, accurate calculation of
the gravitational constant is vitally important, and over time repeat
measurements are generally more precise but not wildly different, or do not
dispute the importance of the gravitational constant.

Take one of the papers they reproduced - Transcriptional amplification in
tumor cells with elevated c-Myc. This is an interesting result conducted in
highly contrived experimental conditions. Nobody is going to base their
career, a drug development program, treat a patient or even start a PhD based
on this result alone. I say this is a translational cancer scientist and
medical oncologist. The contribution of this paper is to our knowledge of the
biology of Myc, which has a multitude of actions which are context dependent.
Myc is studied in a variety of different ways using many different methods. If
this paper were being repeated today, the technologies and techniques would be
quite different. The result of this paper is not plugged into the central
dogma of cancer biology, setting us down an erroneous path for the next
thousand years. So to return to my original question, what have we learned in
trying to replicate this study, that we didn't already know? The money would
have been better spent on orthogonal validation/extension of the result using
modern techniques - another name for this is 'science'. The replication crisis
suggests that this routine extension/validation process is somehow less
important than going back and repeating the original experiment, which is I
think a complete misunderstanding. You also seem to be saying that we should
ask researchers to document in excruciating detail all experimental conditions
such that a pastry chef or meteorologist could walk into a lab and
successfully reproduce the experiment - this is an impossibly high bar to set
for scientists who are already working under very difficult conditions, and is
not the solution.

~~~
nonbel
>"what have we learned in trying to replicate this study, that we didn't
already know?"

It sounds like you think it doesn't matter if that result was published vs
"Transcriptional amplification in tumor cells with depressed c-Myc".

If I misread that title replace it with whatever is the opposite result in
this case. Anyway, like I said elsewhere if it isn't worth trying to
replicate, then the original study should have never been funded. How many of
these studies just exist as a "jobs program"?

It sounds like you think it is most of them, in which case great. We can then
easily cut out 90% plus of funding current going towards jobs program stuff
and devote it to the <10% that is worthwhile...

~~~
Gatsky
The effects of Myc have been tested in many different ways since that study.
Just type Myc and transcription into pubmed and you can see for yourself.

I'm not sure what you are trying to achieve by making outlandish comments
about cutting funding or jobs programs (the idea that cancer research is a
jobs program is utterly hilarious!!), but it doesn't really seem you read my
comment.

~~~
nonbel
I don't see whats confused you about my post.

If no one cares whether the authors got it all wrong and "Transcriptional
amplification in tumor cells with _elevated_ c-Myc under conditions xyz"
should actually be "Transcriptional amplification in tumor cells with
_depressed_ c-Myc under conditions xyz", then why was this funded?

If someone does care then it should be replicated.

~~~
Gatsky
> If someone does care then it should be replicated.

It is tested in other forms, but isn't generally replicated in the sense you
seem to think is paramount. Nor should it be. I'll give you a silly example -
would you support a project to go back and replicate electromagnetism
experiments performed at the start of the century? Say we do repeat Millikan's
oil drop experiment and get a different result (which is actually what
happened) - does this mean there is a reproducibility crisis in physics? If we
don't repeat the exact experiment, does that mean that Millikan shouldn't have
received funding? Why is it that replicating the result the way Millikan did
it more useful than doing other related experiments with more sophisticated or
different apparatus? The latter is actually MORE useful.

~~~
nonbel
>"would you support a project to go back and replicate electromagnetism
experiments performed at the start of the century?"

Yes, of course! That is a great idea. Everyone should be doing this experiment
in high school or undergrad science class by now. In fact that seems to be a
thing:

[https://hepweb.ucsd.edu/2dl/pasco/Millikans%20Oil%20Drop%20M...](https://hepweb.ucsd.edu/2dl/pasco/Millikans%20Oil%20Drop%20Manual%20\(AP-8210\).pdf)

[https://www.ucd.ie/t4cms/EXP18%20millikan.pdf](https://www.ucd.ie/t4cms/EXP18%20millikan.pdf)

