

Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies - djkn0x
http://blog.scienceexchange.com/2013/10/reproducibility-initiative-receives-1-3m-grant-to-validate-50-landmark-cancer-studies/

======
napoleoncomplex
Reproducibility in science is something that badly needs this push. It's an
incredibly difficult "sell" to anyone with funds for research, and I'm
extremely happy that they've found capital for it.

The foundations of our scientific knowledge need to be solidified, and from
all the science news and developments, this one is the one that makes me by
far the most excited for the future of science.

Next on the list, open source repositories for protocols of experiments! Maybe
someone surprises me with a link to an existing solution :).

~~~
Alex3917
Assuming there are about 3 dozen common scientific mistakes in medical journal
articles, each appearing in on average 10-20% of papers with a relatively high
degree of independence, there isn't really much foundation for medical science
to begin with. (And I think these assumptions are probably fairly accurate.)

~~~
tokenadult
You haven't validated any of those assumptions. Moreover a "common scientific
mistake," while regrettable and something to reduce, may not have a lot of
impact on actual clinical practice based on what physicians read in published
medical journal articles. You'll have to connect a lot more dots to show that
there is a high-impact problem here.

Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is
still rising throughout the developed countries of the world,[1] so at least
the trend lines look good for continued improvements in health as medical
research is incrementally improved. Girls born since 2000 in the developed
world are more likely than not to reach the age of 100, with boys likely to
enjoy lifespans almost as long. The article "The Biodemography of Human
Ageing"[2] by James Vaupel, originally published in the journal Nature in
2010, is a good current reference on the subject. Vaupel is one of the leading
scholars on the demography of aging and how to adjust for time trends in life
expectancy. His striking finding is "Humans are living longer than ever
before. In fact, newborn children in high-income countries can expect to live
to more than 100 years. Starting in the mid-1800s, human longevity has
increased dramatically and life expectancy is increasing by an average of six
hours a day."[3] An article in a series on Slate, "Why Are You Not Dead Yet?
Life expectancy doubled in past 150 years. Here’s why"[4] makes clear that
it's progress here, and progress there, and a lot of little things adding up
that have resulted in the dramatic increases in healthy lifespan over the last
few generations.

[1]
[http://www.scientificamerican.com/article.cfm?id=longevity-w...](http://www.scientificamerican.com/article.cfm?id=longevity-
why-we-die-global-life-expectancy)

[2] [http://www.demographic-
challenge.com/files/downloads/2eb51e2...](http://www.demographic-
challenge.com/files/downloads/2eb51e2860ef54d218ce5ce19abe6a59/dc_biodemography_of_human_ageing_nature_2010_vaupel.pdf)

[3]
[http://www.prb.org/Journalists/Webcasts/2010/humanlongevity....](http://www.prb.org/Journalists/Webcasts/2010/humanlongevity.aspx)

[4]
[http://www.slate.com/articles/health_and_science/science_of_...](http://www.slate.com/articles/health_and_science/science_of_longevity/2013/09/life_expectancy_history_public_health_and_medical_advances_that_lead_to.html)

~~~
Alex3917
> You haven't validated any of those assumptions.

There's an entire field of researchers researching researchers, so it's
trivially easy to put together a list of common research errors and their
prevalence. The only difficult thing is calculating the degree of
independence. But there isn't any evidence, so far as I know, that these
errors are highly correlated. Nor is there much reason to beleive that they
are, since presumably the more errors a study has, the less likely it is to be
published. Also, we already know from Ioannnidis et al. that the percentage of
medical studies that can't be replicated is (iirc) in the 80-90% range. And
just because a study can be replicated doesn't mean it's accurate, so 10-20%
is really an upper bound on accuracy.

> Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is
> still rising throughout the developed countries of the world

But as I've posted previously, and as you can find trivially easily via
Google, of the 30 year increase in life expectancy in the US, a _minimum_ of
25 of those years have come from public health improvements rather than from
modern medicine.

~~~
cowsandmilk
> But as I've posted previously, and as you can find trivially easily via
> Google, of the 30 year increase in life expectancy in the US, a minimum of
> 25 of those years have come from public health improvements rather than from
> modern medicine.

Assuming you are getting the 25 years from the CDC[1], you are being obtuse.
Immunizations is the first thing listed. Vaccines are a component of modern
medicine. Control of Infectious diseases includes antibiotics, which are
medicine. Clicking on "Declines in Deaths from Heart Disease and Stroke" shows
that it includes "an increase in the percentage of persons with hypertension
who have the condition treated and controlled" using pharmaceuticals,
definitively modern medicine.

In summary, these 25 years aren't coming from just getting people to quit
smoking. There's a large component that is modern medicine.

[1]
[http://www.cdc.gov/about/history/tengpha.htm](http://www.cdc.gov/about/history/tengpha.htm)

~~~
Alex3917
"Vaccines are a component of modern medicine."

Given that the argument was about the reliability of modern medical studies,
it would probably be only reasonable to include studies that came onto the
market after the era of plabeco controlled RCTs. But many if not most of the
most impactful vaccines came out before then. Of the remaining important
vaccines, they account for only a tiny fraction of the literature.

But more importantly, it's not even a valid argument to begin with since the
increase in life expectancy has nothing to do with the percentage of medical
papers that are flawed.

------
Osmium
Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course,
verification should be cheaper than the original research since you know
exactly what to look for and how to find it.

~~~
mathattack
If $1.3M can cover 50, they should fund reproducing 500, or even 5000.

I suspect that many pure science studies are relatively easier to reproduce
since it's more about basic properties. If you're getting into medicine, it's
more expensive because you may need to reproduce trials. Psychology even more
so because the tests are so soft. But of course psych and medical studies are
the ones most in need of replication.

~~~
Osmium
> I suspect that many pure science studies are relatively easier to reproduce
> since it's more about basic properties.

Not necessarily true at all actually. To use an example, the field I work in
requires that we grow our own semiconductor crystals. Different growth
reactors in different universities producing nominally _the exact same
substance_ will result in sometimes vastly different properties of the end
result when studied (usually due to minute trace impurities, or marginally
different temperatures or flow rates during growth, or a host of other
things–all of which need to be analysed and vetted–and this happens even if
they're using growth reactors made _by the same manufacturer_!). It makes
replication very difficult and often results in slow progress. And it's
expensive too. So it's not just medicine that has that problem.

Edit: I should add that obviously in the end it works out because these
materials eventually make it into industry and into millions of devices. So
it's ultimately reproducible. But that gap between first discovering a process
or a mechanism or just _something_ interesting and making it reproducible
consistently enough and at a large enough scale for industry is just huge.

~~~
mathattack
Indeed - thank you for clarifying. This does sound expensive, though all the
more necessary.

------
irollboozers
Science Exchange is leading and pushing ahead with this very important work.
They are addressing what the public funders and private industry can't and
won't do, but at scale this becomes really powerful. Great stuff.

~~~
Create
They did it publicly, ie

Why Most Published Research Findings Are False. PLoS Med 2(8): e124.
doi:10.1371/journal.pmed.0020124

and privately: the matter of fact is, pharma companies are unhappy, because
according to them, over 70% is irreproducible (euphemism for BS).

This is why ie. BMJ will only accept papers from January on that publish
evidence ie. data also. One wonders what happened before...

BMJ 2013; 347 doi:
[http://dx.doi.org/10.1136/bmj.f5975](http://dx.doi.org/10.1136/bmj.f5975)

------
DaveWalk
I'd be interested to know _which_ studies they are targeting. Is SciEx testing
a key figure from an expansive publication, or the entire methodology from
discoveries with few tests? To me, this seems to be a conceptually difficult
decision to make...most discoveries do not discuss the number of years (or
failed attempts) that goes by before obtaining the quantifiable result.

And where is the peer review in this process? I suppose as soon as something
turns up unreproducible we will find out.

------
gabemart
I don't know very much about how reproducibility validation works. Is it the
case that, if we assume p= ~0.05 and all 50 original studies are perfect, we
would expect the first iteration of reproducibility validation to fail for ~2
of the 50 studies?

~~~
mjn
Not necessarily 5%, no. The p-value is controlling the false-positive rate,
i.e. it constrains the likelihood that a seemingly-significant result was
really just by chance. Therefore, if we were now given the real ground truth,
it's likely that ~5% of the original positive results (if the studies were
properly conducted) would turn out to be unsubstantiated after all.

But you seem to be positing the reverse: a hypothetical case where we assume
that the results of the original studies were in fact correct (i.e. ground
truth), and we want a sort of false-negative rate, the chance that a fact
would _not_ be confirmed by a replication study, despite actually being true.
That depends on several things about the replication attempt, such as sample
size and methodology, summarized into the concept of statistical power:
[http://en.wikipedia.org/wiki/Statistical_power](http://en.wikipedia.org/wiki/Statistical_power)

The overall failure-to-replicate rate would involve both aspects, with four
possible outcomes at varying likelihood: 1) a correct result that was
confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect
result that was confirmed anyway; and 4) an incorrect result that failed to be
confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2
and #3.

~~~
gabemart
Thanks, very interesting. It was silly of me to assume that the p-value could
be used in that way when, as you explain, it's not measuring or indicating
that at all.

~~~
gwern
Don't feel too bad. It's a _very_ common mistake; see some of the citations in
[https://en.wikipedia.org/wiki/P-value#Misunderstandings](https://en.wikipedia.org/wiki/P-value#Misunderstandings)
and
[http://lesswrong.com/lw/g13/against_nhst/](http://lesswrong.com/lw/g13/against_nhst/)

------
doctoboggan
This is awesome. I met some of the people behind The Center For Open Science
at SciPy this year. They seemed very passionate. I hope the idea of
reproducing experiments as a matter of course becomes more common. Maybe in
the future to be a reputable scientist you will have had to reproduce many of
the current experiments of the time.

------
dnautics
this is really phenomenal. Congratulations, SE.

------
ypandit
Congratulations SciEx !

------
brianbreslin
awesome congrats guys!

