
Preregistration of clinical trials causes medicines to stop working - stared
http://chrisblattman.com/2016/03/01/13719/
======
JangoSteve
In case it's not totally obvious, basically...

* In 2000, a rule was passed that clinical trials must include their hypotheses (i.e. clinical outcomes) in their initial registration before the trial is conducted.

* From the graph, the number and proportion of clinical trials through NLHBI with positive outcomes (i.e. an improvement in some health condition) is far lower after 2000 than before.

It was previously easy for researchers to conduct a trial, and then find
positive trends in the data to form positive outcomes by framing the data from
the trial in a different manner, as if the trial were originally intended to
test that positive outcome. This is a problem related to A Priori Probability,
where you're essentially turning research, which could have been statistically
applied to a larger population given some statistical confidence, into
deductive reasoning which doesn't necessarily apply beyond the finite sample
size [1] (also see @entee's comment and links for better explanation of why
this is important).

This article is stating that the graph implies that, because there are fewer
positive outcomes after this new rule in 2000 than before, most of the
positive outcomes prior to 2000 (which likely lead to new FDA-approved drugs
on the market) must have been illegitimate.

I would point out though, that at least from that graph, the last positive
outcome shown prior to 2000 was in 1996, so it's not like it obviously
continued right up until 2000 and then suddenly stopped. Even when that rule
was imposed in 2000 (I'm not necessarily sure how the ruling was phased in),
there was a 4-year gap since the last positive outcome before then, and I
don't see another 4-year gap prior to that since before around 1979. I'm not
saying I disagree with the article (obviously), but just that while the
graphic used seems to imply a trend, it seems like more data would be needed
to conclusively show that trend.

[1]
[http://www.investopedia.com/terms/a/apriori.asp](http://www.investopedia.com/terms/a/apriori.asp)

~~~
akiselev
The 90s is also around the time when we pretty much ran out of small molecule
blockbuster drugs (the very profitable but low hanging fruit) and started
shifting over to much more complex multi-drug therapies like those for HIV and
cancer. There are still a few such small molecule candidates left but the
major markets like blood pressure, heart disease, arthritis, etc. are very
saturated. As the clinic trials start to target more complex diseases with
more complicated drugs and therapies it adds a large confounding factor to the
decrease in the number of successful trials.

~~~
acveilleux
I think you make a very salient point.

From the abstract:

> We identified all large NHLBI supported RCTs between 1970 and 2012
> evaluating drugs or dietary supplements for the treatment or prevention of
> cardiovascular disease.

So this is for a narrow field where there are effective drugs already and
where all the low hangoing fruits are fully picked because the business
opportunity is huge.

------
entee
There's a huge amount of concern in the medical profession that lots of
studies in the area are statistically compromised precisely because the
authors gamed the statistics after the fact, or because improper metrics were
used in the first place.

See these (kind of old but still quite appropriate) links:

[http://journals.plos.org/plosmedicine/article?id=10.1371/jou...](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)

[http://www.theatlantic.com/magazine/archive/2010/11/lies-
dam...](http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-
and-medical-science/308269/)

It's this kind of work that led to pressure to pre-declare statistical
objectives, and now we see the results. Biology is hard, medicine is hard, we
must be quite humble about what we understand in these fields and therefore
how easy it is to be misled by early promising results.

------
chopin
Would someone care to explain what the graphic shown means exactly? I have a
hard time understanding what the vertical axis tells me.

~~~
gech
When experimenters are required to state their targets beforehand, they fail
to meet them. If they don't have to state the outcomes beforehand, then its
much easier to spin what they do find as beneficial!

------
Alex3917
If clinical trials were required to be funded and conducted by the government
then you'd see the same thing again.

------
biot
In other words, declaring your lottery number picks prior to the drawing
results in fewer winners than being able to pick your numbers afterwards.
Ergo, pre-declaring your picks causes lotteries to stop working.

I really hope the article is satire.

~~~
mizzao
It's not satire; that's just the sad state of scientific experimentation. A
similar finding was widely publicized for psychology studies:

[http://www.nature.com/news/over-half-of-psychology-
studies-f...](http://www.nature.com/news/over-half-of-psychology-studies-fail-
reproducibility-test-1.18248)

------
NoMoreNicksLeft
My god. Can we please stop requiring this? I need my placebos to work.

