
Why Are Clinical Trials So Complicated? - yread
https://blogs.sciencemag.org/pipeline/archives/2020/05/01/why-are-clinical-trials-so-complicated
======
gumby
Having designed and run clinical trials and combo manufacturing (drug +
device) in my past I can testify that this is right on. There's a huge effort
(pushed by the FDA back on the manufacturer) to isolate variables and to
ensure that whatever you want to say about your product you have actually
verified to their satisfaction. As a person who also takes medication I'm glad
about it.

I also don't think people should harsh on the FDA because of their strictness.
There's a lot of gaming attempted in dealing with them which makes them harder
to work with, but in my experience they've been pretty decent to deal with. Of
course we had a policy of _no_ attempting to game the system.

~~~
doctoring
On gaming: it's very interesting that, even within all of the byzantine rules
and regulations and guidances how much gray area and leeway there can be. From
a drug/device company perspective, which people in the FDA assigned as
reviewers / contact points for you and how well you are able to articulate &
convince them of your goals can play a huge role in whether you have to do X
type of trial or adhere to Y rules. This is especially true at the boundaries
(new types of testing, devices, technologies) that don't quite exactly fit the
more well-trod paths. Not saying this in a judgmental way -- it's just the
nature of dealing with extremely complex problems while also making sure
people aren't harmed unnecessarily and that at the end when you say
drug/device/tech A improves condition B we actually are sure it is true.

And then there's the more malicious "gaming", fraud! Clinical research is
obviously heavily incentivized toward positive trial outcomes, but for each
site the act of just enrolling patients is incentivized, so you need to be
damn sure the patient data (and the patients themselves!) in your study are
real. And being sure is not easy or cheap.

For fun, here's the FDA's list of people who have been blacklisted from
performing clinical research: [https://www.fda.gov/inspections-compliance-
enforcement-and-c...](https://www.fda.gov/inspections-compliance-enforcement-
and-criminal-investigations/compliance-actions-and-activities/fda-debarment-
list-drug-product-applications)

Their associated "Volume Page" lists (in excruciating bureaucratese) their
"crimes", which may be unrelated illegal narcotics peddling (oops) or
falsifying patient data.

~~~
refurb
I wouldn't really consider "articulate & convince them of your goals" as
really gaming. There are areas that are new for everyone, and you are correct,
if you don't come up with your approach, the FDA will come up with one for
you.

What I've appreciated about the FDA is their scientific focus. The FDA might
say "You need to do X", but if you can come back and convince them with a
scientific argument that you don't have to, they will listen to you!

------
Anthony-G
For someone who never looked into the complexities of clinical trials, I
thought that was a detailed and informative answer. It seems the key point to
take away is, “The amount of noise in human clinical data can defy belief if
you haven’t seen it in person”. :)

~~~
zxexz
I’ve been working in this space for a while now, and this still blows my mind.
Patient-reported data is noisy, doctor’s notes infamously confusing and
inconsistent in formatting. Even fields you can pull from the health system’s
EHR directly will be in disparate formats across different hospitals, or even
different departments.

It would be easy to say it all should be standardized (and it should, for the
most part), but workflow considerations keep standards from being adopted.
What a data standard for oncology data would look like is very different from
what one developed for ophthalmology. Not to mention UX - if you’re a doctor
seeing 20 patients a day, an extra 15 minutes of data entry per patient could
bring that number down to 15 patients or less. UX and data standards really
does have a direct link to patient outcomes in healthcare!

~~~
yread
Indeed, we had to recently work with 500k patients' appointments. Database
table Appointment had a date and "actually met" flag - sounds simple enough,
right? Nope - doctors were "meeting" dead people, different departments had
different rules about whether phone calls were included and what did the
"actually met" flag actually mean.

------
choxi
> The amount of noise in human clinical data can defy belief if you haven’t
> seen it in person

I'm not surprised that patient compliance is a problem, but it still seems
like an expensive burden to find patients physically near the clinical trials
so they can come in to take the medication. It just seems like a logistical
nightmare that software could help with.

Could you build an app that requires patients to record themselves taking
their medication under whatever conditions they're supposed to? You might have
to manually audit all of the recordings, but the number of people that still
fail to comply might be offset but how many more patients you can include in
the study. There might be a lot of other things you could verify remotely
using wearable tech, e.g. heart rate now that Apple's EKG monitoring is FDA
approved.

Great read, I'd like to see more "Chesterton's Fence" arguments for other
highly regulated industries.

~~~
onetimeusename
There are apps in use now that monitor drug use for psychiatric patients by
trying to detect that a drug was ingested using a sensor inside the pill that
can be detected with a patch placed on the abdomen. The patch is connected to
the phone and reports back to doctors.

This itself had to be approved and has a number of ways it can be prone to
errors and blow a trial by misreporting if it were used for one.

Generally, especially under the current circumstances for at-risk patients,
there might be a push to do more electronic reporting for clinical trials.
However, the requirement to have patients near a trial center isn't that bad,
you don't have to be right next to it. But when patients drop out of trials it
is very expensive to the company so reducing travel problems is important in
addition to maintaining consistency.

~~~
DanBC
> There are apps in use now that monitor drug use for psychiatric patients by
> trying to detect that a drug was ingested using a sensor inside the pill
> that can be detected with a patch placed on the abdomen. The patch is
> connected to the phone and reports back to doctors.

Do you have any more information about this please?

~~~
onetimeusename
Yes, take a look at the following. It sends various data to a doctor and in
addition patients can report their own data for monitoring purposes.

[https://www.abilifymycite.com/](https://www.abilifymycite.com/)

~~~
DanBC
Excellent, thank you!

------
Tomte
> Once in a while, they’ll even say “Stop right now – your drug is so damn
> good that it’s unethical not to give it to everyone in the trial”, but years
> go by when that does not happen, anywhere in the industry.

We've had such a case a short time ago (months?) in the news. Does anyone
remember what that was about or even has a link?

~~~
zxexz
A kind of side-note, but for similar reasons, most interventional trials in
the oncology space aren’t double-blinded for that reason. If you have cancer
and sign up for a trial, you’ve likely run out of standard treatment options,
so being given a placebo in a potentially life-saving therapy is not easy to
justify. The different arms of the trial tend to vary in inclusion/exclusion
criteria and doses, but only very rarely is there a placebo involved.

~~~
tialaramex
Yup. I had Hodgkins many years ago. We've been treating Hodgkins for many
decades with considerable success (~95% of young men with Hodgkins eventually
die of something else and that may have improved further). And so it would be
unethical to perform a placebo comparison because we're confident from
anecdotal evidence (some patients refuse treatment) that it'll kill you.

I was considered for (but ultimately did not qualify IIRC) a trial of a new
Hodgkins treatment. If I had qualified and opted in all that would have meant
is basically they toss a coin, heads I get the new treatment, tails I get the
existing "Gold standard".

As well as not using a placebo because it's a death sentence, they also can't
truly blind the trial because the new regime is different drugs, with
different side effects and a different calendar. The staff must know which
they are giving you and even if they didn't just tell you it's not hard to
look up online what the Gold Standard is and tell if you aren't getting it.

So all they have left it's a two arm trial in which they've tried to match the
arms up, and then compare the results.

In my case the eventual trial results show the new regime was slightly worse,
not better as hoped. Slightly more (statistically insignificant but not good
news) deaths and worse side effects. Unfortunate, but there was only one way
to find out.

------
antipaul
The saying in our industry is, "clinical trials are the most complicated thing
in the world".

EDIT: adding details:

You have to mobilize large interdisciplinary teams to come up with products
that are inherently very hard to invent, and which, after all that, can easily
fail a trail and never see light of day, or can then still harm or kill people
even after approved, all while operating in an understandbly extremely
regulated and complicated environment.

One example:

One of the many piles of documents that you need to submit to a national
health authority _just so they consider your product_ (before you can release
to a single "user"!), is thorough documentation of how you _plan_ to
manufacture the product, if it were approved.

Just these text files can weigh _gigabytes_. Think how much effort it would
take just to write something coherent that is so long! And of course, you have
to get other teams to run a bunch of experiments to inform what to write.

And this is even before you are actually producing the product for release.

Oh, and maybe your product won't actually be approved, so all that goes to
waste.

And that's just the beginning.

Because all of that excludes _any_ mention of the real deal, which is whether
the product actually works or is safe. All of that requires its own
experiments and documentation.

And all that assumes that decades of prior research actually led to some
useful prototype where you can start.

And all that excludes any mention of ethical considerations like who pays for
the research, what is a just price for a cancer drug - heck, whether you
should even be trying to make a profit off of lifesaving drugs.

If getting to a point like selling your app called Whats App for $18 billion
(or whatever it was) would require a walk down the street, then getting a drug
to market is like leaving the earth, flying past the furthest planets, and
then trying to reach another galaxy.

~~~
BurningFrog
Much of this is regulatory burden, not inherently scientific problems.

~~~
bsder
Most of those regulations are written in the blood of patients.

In addition, I suspect that you have never debugged a delicate physics
experiment.

A single experiment could fill a dozen notebooks with records of what I tried
and used. There were records of serial numbers, calibration sequences, colors
of wires, etc. I could go on and on.

Why?

Because when something didn't work, I needed to have "ground truth" as to what
_was_ working so I could focus on the (hopefully contained) surface of
uncertainty. And this was for a physics experiment where the outcome is
_KNOWN_.

If you don't actually know the outcome, everything gets worse as you don't
have a definite signal between "Hypothesis is wrong" vs "Experimental setup is
broken". Now, multiply this by _thousands_ of steps for a drug. Brrrrrrrrrr.

To top it off, the physical world isn't like software, you often can't go
backwards. In the physical world, if you didn't record everything on the
forward pass, you may have to go back to start. :(

~~~
BurningFrog
The tough part of these regulations is that patient blood is spilled on both
sides of the tradeoff.

A: If you allow a drug that ends up causing more harm than benefit

B: If you ban a drug that would have caused more benefit than harm.

So making these regulations is about finding the balance between A and B.

The problem is that a type A error causes a huge scandal. Those who took the
drug and died are known by name and there is terrible publicity.

But a type B error only causes people who would have died before to still die.
Nothing new has happened, and there is no outcry.

So the pressure on FDA to err on the side of never having any type A deaths,
while allowing possibly millions of type B deaths is enormous.

~~~
tptacek
I'm noticing that you didn't really address the meat of the parent comment,
about the requisite complexity in documenting even a straightforward physics
experiment.

~~~
BurningFrog
Yeah, I didn't see how that, and especially my personal experience of it, had
any relevance to the issue of government drug regulation.

So I tried to move on to the substance.

It's of course true that science can be very hard. I'm... not really
connecting the dots?

~~~
bsder
> I'm... not really connecting the dots?

No, you're not. And that's why you're missing the point that the details _ARE_
the substance.

The voluminous FDA regulations surrounding drug approval are about tracing
_EVERYTHING_ about that drug so that when (not if) something goes wrong, you
can figure out _what went wrong_.

Theoretical Example: 5 people just went blind from eyedrops of a drug that has
never exhibited that problem before.

Should the drug be pulled from shelves?

If we work backward through the pipeline records and find that the 5 people
all share the same batch of eyedrops, yes. If they don't share the same batch,
no, but then we probably need to call the CDC as we have a new disease.

But, what's a _batch in common_? There is the crate that got delivered to the
pharmacy--did the drug get too hot and melt some of the conatiner into it?
There is the batch that got delivered to the distributor--did a rat or
something urinate on the box? There is the batch that got delivered to the
sterilizing company (probably gamma radiation for eyedrops)--did they not give
it enough dose or did they have a metal obstruction from multiple loads in the
facility.

I can keep going further and further back. Was the water in the process
supposed to be deionized or sterilized in some way? Did somebody mix up two
similar compounds on the line? Did you clean your reactor with the wrong
compound? Some drugs are made by cells and your reactor can "drift" over time
and you may have to purge it and start over when the drift gets too far.

The regulations demand that you record all of that up front before you can
even _BEGIN_ the process of getting a drug approved for use.

(The cleaning the reactor one is not theoretical:
[https://blogs.sciencemag.org/pipeline/archives/2007/07/09/no...](https://blogs.sciencemag.org/pipeline/archives/2007/07/09/now_with_ethyl_mesylate)
)

In addition, I would like to add that the FDA are _actually really helpful_
once you demonstrate that you are not a clown. However, they get a _LOT_ of
clowns on a daily basis and basically have to assume that you are too until
you demonstrate otherwise.

~~~
zozbot234
> But, what's a batch in common? There is the crate that got delivered to the
> pharmacy--did the drug get too hot and melt some of the conatiner into it?
> There is the batch that got delivered to the distributor--did a rat or
> something urinate on the box? There is the batch that got delivered to the
> sterilizing company (probably gamma radiation for eyedrops)--did they not
> give it enough dose or did they have a metal obstruction from multiple loads
> in the facility.

> I can keep going further and further back. Was the water in the process
> supposed to be deionized or sterilized in some way? Did somebody mix up two
> similar compounds on the line? Did you clean your reactor with the wrong
> compound? Some drugs are made by cells and your reactor can "drift" over
> time and you may have to purge it and start over when the drift gets too
> far.

These days we could track every single step in this product supply chain far
more effectively with a crypto-based, blockchain approach. It would be a lot
more efficient.

~~~
cycomanic
The arrogance in the software space (I'm making an assumption here based on
your block chain comment) is often mind blowing. Somehow people who created
some software (or even more common an IT startup) think they got everything
figured out.

I've just seen it recently with a diary company where a friend was working.
Company was build from the ground up by an elderly couple, they sold to an
investment conglomerate when they wanted to retire. These guys brought in a
new CEO, CFO and upper management team all from the tech startup world. So
these guys come say, we must change this, that make everything more agile,
streamline... When the production manager kwho had been there for >15years)
and my friend former logistics and deputy GM said this doesn't work like this,
we need to take things more slowly, do one change at the time, they both were
let go, who likes a naysayer. Well 1 year later the company has lost half its
value, the investors fired the management team and brought in some people with
dairy experience, it's still 50/50 if the company will survive.

The point of this is, the people doing this stuff aren't stupid, there are
often good reasons why things are done in a certain way, so stop and listen
before being a smug knowitall.

------
anthony_doan
Interned at the FDA. They take their stuff seriously.

Clinical trials have 3 phases and it all have to be explained via statistic
(design of experiment) and reproducible. They have standards for everything to
enter the US drug market.

Lots of biostatistics stuff too.

Even when the drug enter the market, pharma companies stop testing the drugs.
It is up to the FDA to keep track of side effects and toxicity that were not
caught through the phases. (example: ambien sleeping drug)

The people at FDA NCTR are wonderful, highly recommend interning there for a
summer.

They have some ML stuff. I've seen a project where they attempt to figure out
what bug is in the food. One of the bug was midly toxic and they wanted a
image classifier to figure it out via picture of a bug part. You can only have
a certain ratio of bug parts in food.

------
Gatsky
Clinical trials are difficult. But when you consider they are a method to
reliably generate new knowledge about complex systems (ie. humans) it doesn’t
seem so bad. Actually they are remarkably efficient knowledge generators,
compared to other areas of science, and have the advantage that a non-result
is still useful up to a certain point.

------
neonate
[https://web.archive.org/web/20200502061205/https://blogs.sci...](https://web.archive.org/web/20200502061205/https://blogs.sciencemag.org/pipeline/archives/2020/05/01/why-
are-clinical-trials-so-complicated)

------
streptomycin
If you like this post, check out the other posts on his blog too. It's
fantastic content.

------
dennis_jeeves
Because the supposed drug/therapy is so marginally effective that it need to
be muddled up with tons of data (noise) and committees.

Often I think it's just bureaucracy, and incompetence. Even if drug/therapy is
massively effective, I'm sure the positive results would be lost in the
bureaucracy.

