
Case Studies Where Phase 2 and Phase 3 Trials had Divergent Results [pdf] - aaavl2821
https://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/UCM535780.pdf
======
aaavl2821
A more recent, and very high profile, example of this is Incyte and Merck's
failed study of Incyte's IDO inhibitor in combo with Merck's PD-1 inhibitor
[0]

Immuno-oncology has been the most active area in biotech VC for years. THe
excitement has been driven by PD-1 and PD-L1 studies showing dramatic benefit
in many types of cancer. However only 20-50% of patients respond to these
drugs, so VCs and pharma companies have been spending billions of dollars
studying "combo therapies", basically regimens where you add drug x to a PD-1
inhibitor to improve effectiveness and response rate

IDO was one of the most promising combo drugs. Bristol myers bought a leading
IDO startup for $1B 18 months after series a. after this study pretty much all
IDO programs were canned across industry. beyond that, many investors /
companies now want to see that a drug works as a standalone agent, not just in
combo w another drug

as the paper says, there are lots of valid reasons to use innovative study
designs, but proper phase 3 RCTs are importnat

[0]
[https://www.businesswire.com/news/home/20180406005141/en/Inc...](https://www.businesswire.com/news/home/20180406005141/en/Incyte-
Merck-Provide-Update-Phase-3-Study)

------
Gatsky
I was surprised when I first got involved with drug development, how rushed
the process is. Phase I and Phase II trials are initiated by pharma with
strikingly little understanding of how the drug works. A classic example is
iniparib, which made it to a phase 3 trial but turned out not actually hit the
target it was supposed to hit (PARP).

The reason for this is I think economics. There is a large advantage to being
first to market, especially if the disease has no other treatments. This means
that you can set your price, there is zero competition, and you disincentivize
all other companies from developing similar drugs. Under these conditions, you
can recoup development and trial costs in 1 - 2 years. So it is no wonder that
drugs with any sort of signal are pushed quickly through early phase trials.

~~~
lekanwang
Yes, this is a very good point -- you're on the clock as soon as you file your
patent, so sponsors have an incredibly strong incentive to take risks with
trial design and use surrogate endpoints rather than direct endpoints for the
primary to cut down on time. Then it's really an educated guess (albeit a
highly educated guess) for your Phase III trial design to balance out your
market vs chances of proving efficacy.

~~~
refurb
Patent life is an important factor, but the other one is that it’s sometimes
really hard to validate how a drug works. Either you don’t have the
technology, money or time to do it. And in the end all that matters is it
works in humans.

My favorite story is about gabapentin. Designed to block the GABAase enzyme.
It made it all the way to market before they realized it actually works
through a different mechanism.

------
arachnophobe
As someone on a phase 2 trial an interesting note is that I know I'm on the
trial drug (it's not a double blind trial), I also know that the drug worked
in animal and vitro trials.

Furthermore I know, from the trial literature, that a very similar drug works
(the trial drug is an anti-PD-1 immunotherapy) well with the particular
genetic markers on my tumours.

The trial is to assess the efficacy of this treatment, not run a comparison to
other treatments, but having burnt through 2 rounds of chemo in short order to
no effect it is more a comparison against that.

Whether the placebo effect is relevant is an interesting question because
under the trial the treatment is, by necessity, more personalised.

Previous studies have shown that, essentially, human contact/interest adds to
the placebo effect.

------
patorjk
This happened to Otonomy about a year ago and their stock fell off a cliff.
The interesting thing with them was that they were doing 2 sets of phase 3
trials: one in the US, and one in Europe. A little while after the US one
failed, the data from the European one came out and showed statistical
significance. I remember reading that some of the scientists at Otonomy
believed the problem with the US study was that some of the doctors who were
administering the drug were telling the patients that it really worked,
because the success for placebo patents was higher in the US than it was in
Europe.

~~~
refurb
A lot of people underestimate the challenge of running a clinical trial.
Clinical trials are often run across multiple sites/countries where medical
standards differ. The placebo effect can be brutal particularly in the more
subjective trial endpoints.

People crap all over the anti-depressant clinical trial data, but guess what?
It's a poster boy for the placebo effect and trying to development objective
endpoints you can measure.

------
kgwgk
Here is a recent example where the same trial has and has not been a success,
depending on how you look at it:
[https://www.forbes.com/sites/matthewherper/2018/07/05/biogen...](https://www.forbes.com/sites/matthewherper/2018/07/05/biogen-
and-eisai-say-alzheimers-drug-a-success-reversing-earlier-
result/#4933a86628b4)

------
mjevans
After looking at the first few examples I skimmed the summary header, ALL of
the (examples of) phase 3 failures include lack of efficacy (some also lack of
safety) as the main failure.

Since medial trails are very expensive and time consuming, I can't help but
wonder if identifying this would have been possible from prior phases,
possibly with better study design of at least the phase 2 version of the
study.

~~~
aaavl2821
The most commonly cited study of success rates across trials lists Phase 3
probability of success, in aggregate, at 70% [0]

Phase 2 success rate is 34%.

Phase 3 studies are more expensive than Phase 2, but the higher failure rate
in Phase 2 makes Phase 2 failure the largest driver of the cost of drug
development.

Because predicting Phase 2 success from earlier stage data is a daunting task,
the industry has started tackling this issue by "failing faster / cheaper",
getting to Phase 2 data quickly and cheaply, to reduce the cost of Phase 2
failure. However, companies don't want to fail -- so they sometimes design
Phase 1/2 proof of concept studies to increase chance of success rather than
increase truth finding. This may end up causing the Phase 3 failure rate to
increase, though there isnt yet data to prove this. The IDO-PD1 failure from
Merck / Incyte could be interpreted as an example of this phenomenon

I wrote a blog post that in part discusses ways companies are trying to reduce
cost of Phase 2 failure:
[https://newbio.tech/blog/bio_startup_ideas.html](https://newbio.tech/blog/bio_startup_ideas.html)

[0]
[https://www.ncbi.nlm.nih.gov/pubmed/20168317](https://www.ncbi.nlm.nih.gov/pubmed/20168317)

------
danieltillett
The problem is the marketing department. Most of the big diseases (cancer,
depression, heart disease, dementia, etc) have complex and diverse causes and
so only a subset of patients will respond to any one drug. At Phase 2 the
trial co-ordinators have relatively strict inclusion criteria targeting the
likely responders (i.e. the subset that have the version of the disease the
drug will work on).

When it gets to Phase 3 marketing gets involved in the inclusion criteria and
they want the criteria to be as broad as possible so they can market to the
widest number of patients. The aim is to have a criteria that is as broad as
possible where the Phase 3 trial just scrapes across the line. This is a
difficult thing to get right and sometime they include too many non-responders
and the trial falls over.

The only real solution is to get the cost of developing a drug down so that
there is not the need to go so broad. Of course this will require taking more
risks (the level of pre-clinical and clinical testing needs to be reduced) so
I am not sure this is viable.

~~~
refurb
As someone who works in the industry, none of this seems accurate.

\- Marketing gets involved before phase 3. Phase 2 results are usually where
the business decision to proceed to phase 3 are made, so commercial needs
input on the trial design.

\- Inclusion/exclusion criteria might be tweaked between phase 2 and 3, but
they don't tend to drastically change as you're just asking for the trial to
fail.

\- Development works with the FDA to craft inclusion criteria that preserves
the scientific rigor of the trials, but also allows for the broadest possible
indication. In general, the FDA grants a much broader label than what is
described in the trial criteria.

~~~
danieltillett
You have basically just summarised what I said :)

I would not say the criteria are just “tweaked” between Phase 2 and 3 with any
drug aimed at a broad market. It is always a balance between broadness and
success and sometimes even the best in the business get it wrong.

The FDA do allow a broader label than the exact trial criteria, but not
massively so. If you restrict the Phase 3 to an identifiable sub-population
good luck getting approval to expand to the larger population without running
post-approval trials.

~~~
refurb
Where we disagree is on three points:(1) marketing gets involved well before
phase 3,(2) most trial criteria don’t change that much between phase 2 and 3
[yes there are exceptions] and (3) the FDA allowing a broader label than
inclusion criteria.

As for the last one, most phase 3 trials have strict criteria: no other
diseases, age limits, disease severity limits, etc. Those typically don’t make
it into the label’s indication.

I do agree that trials are designed to balance getting a positive outcome vs.
the broadest population possible, but marketing isn’t the only one driving
that.

~~~
danieltillett
1\. Yes marketing gets involved before Phase 3, but they tend to not have the
clout they have going into Phase 3 - generally when it is unclear that you
have something that works at all it hard for marketing to argue in favour of
broadening the inclusions criteria.

2\. I think your point 2 is where we disagree the most. We are really arguing
over what is a significant change or not.

3\. The FDA will be looser on some things like age and other diseases (within
reason as you won’t get approval for children if you only trial in adults),
but not on sub-groups. If you make your trial exclude on some disease sub-
group criteria you are not going to get FDA approval to go outside that sub-
group and sell to everyone with the disease.

Who is driving trial broadening other than marketing? If I want the drug to
succeed I want the trial population to be a homogeneous as possible and only
include likely responders.

------
reasonattlm
It is quite possible to succeed and still fail your trial. Which is one of the
many reasons these things should be thrown out in favor of patient choice. You
could spend 1/10 of the cost of later trials on publishing risk assessments
and still be far ahead of the game of risk management.

Another problem is that too many of today's medical technologies are horribly
marginal in their benefits. Marginal benefits have a way of smoothing out to
nothing as the patient population broadens. The industry isn't approaching
medicine for age-related disease in the right way. If you are actually
addressing a useful mechanism, the effect won't be marginal. Science managed
that shift from marginal to effective for infectious disease. There is now a
pending transition for the diseases of aging - from marginal messing with the
disease state to actually addressing the root causes.

Here is a phase 3 that succeeded and failed. Clearly it works. But the study
design is such that they had an unexpected result:

[https://www.gensight-biologics.com/2018/04/03/gensight-
biolo...](https://www.gensight-biologics.com/2018/04/03/gensight-biologics-
announces-topline-results-from-reverse-phase-iii-clinical-trial-of-gs010-in-
patients-with-leber-hereditary-optic-neuropathy-lhon/)

They show a good effect in the treated eye, and their gene therapy also
produces benefits in the untreated eye. Since they don't know how that could
happen, no approval.

~~~
jcranmer
Patient choice is an idea that sounds good in theory but just has no good way
of actually working in practice.

Most people are going to be uninformed about medical science. (Hell, most
medical practitioners struggle to stay informed). Worse, there is a strong
bias towards misinformation--look at the health supplement market, which is
basically a market the seedy snake oil sellers set up to get around FDA
regulation.

On top of this, keep in mind that it is _very_ easy to accidentally design bad
experiment methodology. With respect to clinical trials, it's all too easy to
take a failed trial and say "it actually worked on this subgroup" because
there are often enough subgroups to find one with a spurious correlation. And
when you have monetary pressure to salvage a failed product, the p-hacking is
less likely to be internally questioned. The current regulatory regime
requires that you declare how you're going to define acceptance before you run
the trial to prevent this kind of p-hacking, and if you truly believe that
you're not p-hacking, you're welcome to submit another trial to show
correlation on that subgroup (unsurprisingly, many of these end up coming
negative).

~~~
teslabox
> Worse, there is a strong bias towards misinformation--look at the health
> supplement market, which is basically a market the seedy snake oil sellers
> set up to get around FDA regulation.

The pure food and drug act was passed to deal with the snake oil industry,
which was in the process of reinventing and legitimizing itself into the
pharmaceutical industry. The supplement industry was later exempted from the
requirements for patent medicine producers because there is no way to
monopolize the profits of many treatments based on supplements. Some companies
have patented certain aspects of their supplement's production or delivery
mechanisms, but (I believe) the natural biological molecule itself can't be
patented.

[https://en.wikipedia.org/wiki/Pure_Food_and_Drug_Act](https://en.wikipedia.org/wiki/Pure_Food_and_Drug_Act)
(1908)

[https://en.wikipedia.org/wiki/Federal_Food%2C_Drug%2C_and_Co...](https://en.wikipedia.org/wiki/Federal_Food%2C_Drug%2C_and_Cosmetic_Act)
(1938)

