
When doctors, psychologists, and drug makers can't rely on each other's research - nkurz
https://reason.com/archives/2016/01/19/broken-science/print
======
imh
>We are living in a time of technological marvels, with advances like CRISPR
gene-editing being used to bring back extinct mammoths; lithium-air batteries
that store 10 times more energy than conventional lithium-ion batteries;
mitochondrial transfers that create healthy babies who have three genetic
parents; Ebola vaccines that are nearly 100 percent effective; and cars that
drive themselves.

This line was sort of ironic, since we don't really have those things.
Mammoths are still extinct, lithium-air batteries aren't yet practical, three
parent babies aren't legally deemed as safe in most countries, and cars can
only drive themselves in limited ideal conditions (and those are only research
vehicles!). Was that statement was put in as another example of how you can't
believe what you read and it went over my head?

~~~
tim333
I took that line as saying that it was remarkable that those things are
happening at all rather than saying they are safe, practical and fully
functional.

I mean I'm impressed that they have actually inserted mammoth DNA in to
elephant cells even if Jurassic Park will have to wait a bit.
([http://www.livescience.com/50275-bringing-back-woolly-
mammot...](http://www.livescience.com/50275-bringing-back-woolly-mammoth-
dna.html))

------
jwmerrill
> The model he has in mind is the e-print distribution platform arXive.

> Other disciplines are emulating the arXive pre-print model

> ...something like a gigantic open-source version of arXive for all
> scientific research

That's not how you spell arXiv.

------
golergka
What it would take a resarch-funding organization to issue grants for
specifically replicating results of existing papers? Why don't they do it? May
be they do and I'm just unaware?

~~~
Outdoorsman
There are organizations that are attempting to police published research...one
such is Retraction Watch...

It's a 501(c)3, thus a not-for profit...not sure how much grant funding they
receive, but they do actively solicit donations from the public...

I think you'd be surprised at the number of public research papers, including
some that end up being sources for articles in well-known science journals,
that are retracted each year...

A list of papers retracted, which is continually updated, can be found at
their site:

[http://retractionwatch.com/](http://retractionwatch.com/)

------
tokenadult
I like this summary of current research on how to do research better. A
checklist of suggestions for psychology research journal editors[1] gives a
different set of links to current research and current best practice, and is
well worth a read too.

[1] [https://jcoynester.wordpress.com/2016/01/21/ten-
suggestions-...](https://jcoynester.wordpress.com/2016/01/21/ten-suggestions-
to-the-new-associate-editors-of-psychological-science/)

------
mwsherman
This is largely predicted by Daniel Kahneman and Nassim Taleb, would love to
see them cited more often.

------
jessaustin
"HARKing" is a great coinage I had not seen before.

------
fizixer
Ladies and gentlemen, Sturgeon's law at work.

------
Alex3917
Why are people so obsessed with publication bias and statistical power? Those
are just two of the dozens of reasons why published research is unreliable,
and it's not even clear that they're the most important ones.

It seems like the skeptic community just randomly glommed onto those issues in
the 90s or something and haven't updated their worldview since.

~~~
467568985476
What are the other, more important reasons that research is unreliable?

~~~
Alex3917
I have a list of them going here, but it's definitely not comprehensive
either:

[http://www.alexkrupp.com/Citevault.html#pharmaceuticals](http://www.alexkrupp.com/Citevault.html#pharmaceuticals)

Even if you look just at statistical analysis, there are dozens of different
issues. E.g. I seem to remember some articles about how basically 100% of
research involving brain scans is incorrect, because of a mix of statistical
problems and also issues related to the imaging software.

~~~
tcj_phx
This section is particularly useful to me:

[http://www.alexkrupp.com/Citevault.html#mentalhealth](http://www.alexkrupp.com/Citevault.html#mentalhealth)

Thanks!

------
mhkool
In double-blind tests, one deliberately changes only one variable and
compares/observes two groups where individual do not know whether they take a
new drug or a placebo. If the drug has a positive effect, the double-blind
test is an accepted method of proof, but unfortunately this limits our options
severely. If cancer can be cured by changing _two_ variables, we will never
find a cure. NEVER! Remember that we already do research for 50+ years and
have spent trillions, so I tend to think that Einstein was right: it is insane
to repeat the same experiments and to expect a different outcome.

Let's jump to Alzheimer, a feared disease with also a lot of research and not
so good drugs. Dr Bredesen has made a protocol with 35 variables to cure
Alzheimer. And in his first test, he reversed Alzheimer in 9 out of 10
patients. The scientific purists say that there is no double-blind study, so
no proof. I am convinced that Dr Bredesen in on the right path, not only
because of the results but also because of his reasoning. The treatment has no
drug, but a health optimization in all possible ways. And then the body heals
itself in 4 months. Please do not comment with "that cannot be true" unless
you proof that it cannot be true (I do not belief that you can provide proof).

So the state of current research methods is weak. Medical research only
focuses on one variable, one drug, that will cure a disease. The results of
this way of doing medical research are VERY disappointing. Trillions are spent
and no drugs that cure cancer, AIDS or Alzheimer have been found. Dr Bredesen
has chosen a different path and I support him.

~~~
mabcat
Unfortunately you have a fundamental misunderstanding of health experimental
design. Double-blind means that neither the participants nor the experimenters
know which participant is allocated to which group. It doesn't say anything
about how many groups there are or what is done to each group. I haven't heard
of Dr Bredesen, but they could certainly run a double-blind study to validate
their treatment regime. Treatments that haven't been validated in this way
have a high risk of being ineffective but appearing effective due to
experimenter error, placebo effect, demand characteristics, etc. Requiring
convincing evidence in the form of a double-blind trial isn't purism, it's
sensible caution based on experience.

edit: you're also on shaky ground statistically. There's a rule of thumb that
if interactions between several variables are present in a treatment effect,
70% of that effect will show up on the individual variables (as main effects).
It's very unlikely that there exists a treatment where all the effect is on a
35-variable interaction and none of the effect is visible when looking at each
variable singly.

~~~
mhkool
You need to read my comment again. I never stated nor think what you wrongly
assume.

Your reasoning about the work of Dr Bredesen shows that you a re ignoring the
facts. reversal of Alzheimer in 9 of 10 patients is a very significant results
since nobody has ever reversed Alzheimer. The fact that the sample was small
is only a reason to do a followup study with a larger number of patients,
which is exactly what he is doing.

The desire to reduce protocols to only a single variable is based on the wrong
assumption that treatments can be successful with only one variable changed
and excludes all treatments where more variables play a role.

