
Medical Science Has a Data Problem - juanplusjuan
http://www.newsweek.com/2014/11/21/medical-science-has-data-problem-284066.html
======
conorh
I don't think that anyone outside the field can understand just how bad and
pervasive this problem is, it is _everywhere_ and harming patients every day.
One more example - my wife is a surgeon working in a specialized field and in
this field there are panels of 'experts' that get together and set the
guidelines for how to treat the disease:

\- The panel is stacked with visiting researchers _working_ for the most
senior researcher on the panel.

\- The most senior experts on this panel recommend treating with a particular
drug. It turns out that these experts also happen to receive money from these
drug companies.

\- These researchers then use the results of this panel when applying for NIH
grants as proof that this is where the treatment is heading AND THEN, next
time the panel happens, also use this research as proof that this is where the
treatment is heading. Talk about self interest and circular logic.

What is incredibly sad about this is that the results of consensus panels are
then quoted by doctors every day and used to lead their treatment of patients.

------
tomohawk
It appears that the FDA also has a data problem, as it refuses to respond to
FOIA requests for data that is obviously in the public interest to be
disclosed, and taxpayers end up paying $1.3 billion to stockpile a medicine
that is probably not as effective as advertised.

How does this practice keep us safer?

~~~
Alex3917
They also actually put a gag order on at least one pharma company preventing
from disclosing the fact that antidepressants were not effective in children
and increased the risk of suicide.

The thing about medical studies though is that only the ones the pharma
companies want to get published make it into journals, and there is no
requirement that what they are saying in the journal articles matches the
clinical trial data at all. And the FDA doesn't publish any of the raw
clinical trial data, so you'll never know.

This is especially problematic since pharma companies only need two studies
showing a drug plausibly works to get approval, regardless of how many trials
fail to show any benefit. So usually what you see is one or two journal
articles showing a drug works, which may or may not correspond to the findings
of one or two clinical trials, and the rest of the trial data is buried
forever. And then other authors publish metareviews based on this completely
inaccurate data.

Essentially journal articles should only be ever be viewed as advertisements
for the pharma industry, and should no way ever be taken as science. The one
exception is the largescale trials funded by the NIH.

~~~
ihnorton
There have been some changes [1] in recent years attempting to address this.
Since 2007, all drug, biologic, and device trials in the US must be registered
with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses
safety only - not efficacy), and "results" must be reported. There could
certainly be some loopholes or limitations on the data release that I am
unaware of.

However, it is true that most of the controversial, blockbuster drugs that are
discussed in the original article were approved long before these requirements
came into effect.

[1] [http://clinicaltrials.gov/ct2/manage-
recs/fdaaa](http://clinicaltrials.gov/ct2/manage-recs/fdaaa)

~~~
Alex3917
> Since 2007, all drug, biologic, and device trials in the US must be
> registered with ClinicalTrials.gov after phase 1 (which, at least in
> principle, addresses safety only - not efficacy), and "results" must be
> reported.

\- A substantial percentage of trials haven't reported their results within
the required timeframe.

\- The number of fines the FDA has levied for non-compliance is zero.

\- The data isn't actually published on clinicaltrials.gov, only a brief
summary of the results which doesn't really tell you anything.

\- As you say, the requirements aren't retroactive.

\- There are several large loopholes, e.g. phase IV studies aren't required to
be registered.

------
pron
Medical science does indeed have a data (or, rather, rigorous statistics
problem), but it has a much more serious, intrinsic, problem: dimensionality.
The number of variables is so high that proving something (clinically, when
the biological process is unclear, which is most of the time) to a high degree
of certainty is almost impossible.

~~~
robrenaud
Why doesn't randomization defeat dimensionality?

~~~
lmkg
Randomization lets you _ignore_ dimensions that you're not interested in. It
doesn't do much in the way of letting you _attribute_ an effect to one or more
dimensions from a large set of them.

There's a fundamental lower bound to the number of data-points you need to
make in order to achieve statistical significance. This lower bound grows
exponentially in the number of dimensions, particularly if you cannot make the
simplifying assumption that the dimensions are statistically independent (and
biological systems are often coupled in strange ways). Randomization is a good
way of achieving that lower bound, but it cannot defeat it.

------
mironathetin
I recently listened to an advanced statistics lecture. As an illustration the
Professor said, medical science (still) uses 2 sigma statistics, jet engine
builders use 6 sigma. That's why jet engines never (sic) fail.

Yes, and then there is the influence of the drug makers too.

~~~
tomp
The biggest difference is that jet engine builders can conduct an unlimited
number proper controlled scientific experiments, while the fields that study
humans (psychology, medicine, sociology, economy) can usually only conduct a
limited number of field trials (which, even though they are "controlled" and
possibly "double blind" are nowhere as rigorous as in proper sciences).

~~~
icegreentea
Yes. There was the other article sitting around recently asking why the rate
of blockbuster drugs have slowed down. Getting even the level of confidence
required now is hugely expensive. Requiring higher confidence is an expensive
choice that must be weighed closely.

I think it's becoming clearer each day that many drug side effects, and
general drug efficacy is effected by the genetics of the drug taker. Without
being able to tease out these genetic effects, blindly aiming for broad
population safety or efficacy levels is always going to be troublesome.

This is separate from the data publication issue that the article focuses on.

~~~
fluidcruft
What is a "blockbuster" drug? One that makes a lot of money for the various
stakeholders or one that has a large impact on human health? I'm not sure I
care too much about the former and it's not at all obvious how "level of
confidence" would affect the latter. Yes, as a result of past medical progress
it is more difficult to improve upon the current state of the art.

But new therapies have to compete with existing therapies in the market
anyway. If a new therapy can't outperform existing ones, what actual value
does it have? Perhaps drug companies should think about solving problems that
aren't already solved rather than bemoaning their inability to reap massive
profits while reinventing the wheel or pushing minor formulation tweaks to
renew their patents.

------
Houshalter
Silly question, why do so many drugs have rare side effects? Why does it work
on 999 people and then the 1,000th person takes it and becomes suicidal?

If there is something different biologically different about the 1,000th
person, could it be tested for?

~~~
davecap1
That's not really a silly question :)

Drugs act on proteins which are little machines in the body, the instructions
of which are encoded in our genome. The slight variations in our genetic code
leads to some proteins having slight variations in their structure (and their
function). Most of the time this is harmful, but sometimes it causes them to
react differently when they interact with a drug or with each other. The
manifestation of this can sometimes be harmful. Understanding
"pharmacogenomic" effects will help us mitigate these side-effects going
forward.

------
leoc
Ben Goldacre has a book on the subject [http://www.amazon.com/gp/offer-
listing/0007350740/](http://www.amazon.com/gp/offer-listing/0007350740/) in
addition to that TED talk.

------
radnam
Clinical trials data is one piece of the puzzle and definitely a important
one. One thing which in my opinion would dramatically change patient outcomes
if care provider starts _listening_ to the patient. In an anecdotal
experience, my wife was wrongly diagnosed for almost a year impacting every
day of her life to be filled with pain. Issue as it turned out was a simple
infection and was fixed by an exceptional surgeon with tweezers in few
minutes.

------
tokenadult
I read the article last evening before I turned in in this time zone, and have
just read all the comments posted till now overnight. Readers of the writings
of Dr. John P.A. Ioannidis[1] will already be familiar with many of these
issues, which dog most research on human health and medical treatments.
Several of the previous comments here are very helpful in pointing out that
human physiology is based on much more complicated system interactions than
most man-made machines, and subtle individual variations in genome or personal
environment that are now poorly understood can make a different in how drugs
work for different patients.

That said, an important point that hasn't been made here yet is that
"alternative" treatments to the treatments developed by medical science often
have a much worse data problem, because no one is gathering data about the
alternative treatments for safety or for effectiveness in the first place. In
the United States, "natural supplements" are regulated in a way that exempts
billions of dollars of annual revenue from any review about whether the
supplements do any good at all.[2] Chiropractic manipulation can kill, but
chiropractors don't look into that issue carefully.[3] What you as a person
who desires to stay healthy for many years have to do is turn in the first
instance to medicine that works[4] and to continually ask your doctors for
explanations of what they are doing and what the rationale is for what they
are doing. Patients thinking more will encourage physicians to think more.
Voters telling governments to regulate treatments more rationally will also
help, as will voters supporting more funding of basic medical research.

[1] [https://med.stanford.edu/profiles/john-
ioannidis?tab=publica...](https://med.stanford.edu/profiles/john-
ioannidis?tab=publications)

[2] "Dietary supplement industry says “no” to more information for consumers
(again)" [http://www.sciencebasedmedicine.org/big-supp-resists-
giving-...](http://www.sciencebasedmedicine.org/big-supp-resists-giving-
consumers-safety-and-effectiveness-information/)

[3] "Stroke Death from Chiropractic Neck Manipulation"
[http://www.sciencebasedmedicine.org/stroke-death-from-
chirop...](http://www.sciencebasedmedicine.org/stroke-death-from-chiropractic-
neck-manipulation/)

[4] "There is no such thing as alternative medicine. There is only medicine
that works and medicine that doesn't."
[http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no...](http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no+such+thing+as+alternative+medicine.+There+is+only+medicine+that+works+and+medicine+that+doesn%27t)

