
Nearly 400 medical devices, procedures and practices found ineffective in study - EndXA
https://www.sciencealert.com/recent-study-finds-400-medical-devices-procedures-or-practices-that-are-ineffective
======
dang
The study was discussed a few days ago at
[https://news.ycombinator.com/item?id=20164809](https://news.ycombinator.com/item?id=20164809)

------
EvanAnderson
I found this one from the table of reversals to be interesting:

Article: Efficacy of infant simulator programmes to prevent teenage pregnancy:
a school-based cluster randomised controlled trial in Western Australia

Summary: The infant simulator is an example of persuasion technology or
captology, where the use is intended to prevent teenage pregnancy.26 Their use
is widespread in developed countries27 and is expanding into low-income and
middle-income countries.28 However, in this study done in Australia, the
infant simulator-based VIP program did not reduce teenage pregnancy. In fact,
girls in the intervention group (n=1,267) were more likely to experience a
birth (8% vs. 4%; HR=1.35; 95% CI=1.06-1.73; p=0.016) or an induced abortion
(9% vs. 6%; HR=1.33 (1.00-1.78; p=0.049) than those in the control group
(n=1,567) before they reached 20 years of age. This is a reversal of the
practice of infant simulator programs to prevent teenage pregnancy.

A school district I contract with purchased some of these "infant simulators"
a few years ago. I shared spooky photos of them entombed in foam in their
storage cases (evoking visions of racks of "hibernating" astronauts from sci-
fi) with my wife. She made a general observation that she thought some
fraction of teenage girls would feel encouraged to have a baby after time w/
the infant simulator. Guess there was some truth to that intuition after all.

~~~
sp332
That would only explain part of the effect. It wouldn't explain why the number
of abortions went up by 50%.

~~~
manicdee
\- dolls stimulate maternal instinct (child “gets clucky”)

\- child gets pregnant

Then

\- parents find out, recommend terminating pregnancy

\- friends find out, recommend terminating pregnancy

\- doctor finds out, recommends terminating pregnancy (“you are not ready for
this”)

\- unpleasant side effects set in, child decides she is not ready to handle
pregnancy, or

\- reality sets in, child decides she is not ready to be a mother/wants to
finish school

\- boyfriend abandons her

Basically increasing the number of pregnancies will necessarily increase the
number of abortions.

------
program_whiz
The majority of every invention / attempt will be wrong in any human pursuit.
Just think, 400 useless apps or pieces of software are probably released DAILY
to the itunes store. The only difference is the impact, but its still
understandable. What if you had to write software that did the following:

* Fixed a bug in an existing "legacy system", deployed remotely without your knowledge. You have no logs or debug information, but you can speculate on cause and try it in small simulated / test environments (e.g. rodents). By introducing this fix you would also be changing the system in unknown ways.

* You only have a vague idea of how the code in the OS for this system works, your fix doesn't even change the code. Your only mechanism of change is injecting some foreign JS into a webform to cause ripple effects on the system for the user, meanwhile the system is setup to try to stop / avoid such "attacks" and will try to undo your fix if it can be detected by the system's "AI".

* if your fix fails in production for even one user, you may lose millions and potentially go jail, that person may die.

* You get one shot to release it to the public, after go-live it basically has to work or everything was for nothing. There is no "push this fix and everything gets better", even one small change is a complete restart (new molecule == new drug).

* If you accidentally release any information about the users, or lose their data, you lose millions and potentially go to jail.

* This software will take 10 years, a handful of scientists, a lab, some lawyers, lobbying, and a few billion dollars to make.

* Every user using your software will be on an entirely unique stack, processing all kinds of data in tandem with your software, and will not follow directions. Some will even have lots of other bugs in the system that are unrelated to your fix.

~~~
martindbp
Have you ever thought that maybe not everything has to be seen through the
lens of software? Not trying to be snarky, but it's something I see so often
here on HN and it kind of bugs me.

~~~
Duladian
It is much easier to relate to a known quantity than an unknown. I would say
the vast majority of people here are either programmers or at least have
enough knowledge in programming to relate to it.

------
EndXA
Abstract:

> The ability to identify medical reversals [practices that have been found,
> through randomized controlled trials, to be no better than a prior or lesser
> standard of care] and other low-value medical practices is an essential
> prerequisite for efforts to reduce spending on such practices. Through an
> analysis of more than 3000 randomized controlled trials (RCTs) published in
> three leading medical journals (the Journal of the American Medical
> Association, the Lancet, and the New England Journal of Medicine), we have
> identified 396 medical reversals. Most of the studies (92%) were conducted
> on populations in high-income countries, cardiovascular disease was the most
> common medical category (20%), and medication was the most common type of
> intervention (33%).

The original study can be found here:
[https://doi.org/10.7554/eLife.45183.001](https://doi.org/10.7554/eLife.45183.001)

~~~
js2
Supplementary File 2 contains the full list of reversals:

[https://elifesciences.org/articles/45183/figures#supp2](https://elifesciences.org/articles/45183/figures#supp2)

I don’t know if this is a permalink due to the hash parameter but I’ll try:

[https://elifesciences.org/download/aHR0cHM6Ly9jZG4uZWxpZmVzY...](https://elifesciences.org/download/aHR0cHM6Ly9jZG4uZWxpZmVzY2llbmNlcy5vcmcvYXJ0aWNsZXMvNDUxODMvZWxpZmUtNDUxODMtc3VwcDItdjEuZG9jeA==/elife-45183-supp2-v1.docx?_hash=X7MxyiHvN4wdoXByVuyzLJxV6%2FSJjil2sR5Sc1ETBzw%3D)

~~~
web007
The b64 of that crazy link directs you to
[https://cdn.elifesciences.org/articles/45183/elife-45183-sup...](https://cdn.elifesciences.org/articles/45183/elife-45183-supp2-v1.docx)
\- maybe slightly more permanent?

------
hirundo
I think it's bad policy for the FDA to screen medicines not just for safety
but also efficacy. From this study it appears to be frequently unjustified but
it gives the drug a seal of approval that has a tendency to replace the
judgement of doctors and patients. It's not that the FDA decision makers are
corrupt or incompetent. These are wildly complex issues that nobody can get
right consistently. Better not to conceal that with centralized decision
making.

Modern medicine can't even agree in broad strokes on the proper ratio of
macronutrients for humans. We give them too much power when we uncritically
accept their judgement on any given treatment. For me, prayer would have no
effect. For others it could be a life saving intervention. The efficacy of a
particular molecule for a particular patient isn't much easier than that to
determine. Therefore decision making should be as decentral and
computationally parallel as we can make it.

~~~
richardthered
Without some kind of efficacy standard, you could sell water as a cancer cure.
That would be very bad for health outcomes.

However, I do agree that it's a difficult topic, in general. Off-label drug
uses is one example.

~~~
nybble41
That would be plain false advertising. The FDA doesn't need to get involved in
such trivia.

I have no problem with the FDA as a certification agency, whether they're
certifying safety or efficacy or both. I do have a problem with their approval
being _mandatory_ in order for the product to be sold. If their stamp of
approval means anything—if _lack_ of FDA approval implies that a product is
unsafe and/or ineffective—then doctors, insurers, and patients will all look
for that approval and avoid products without it. The problem is that "safe and
effective" is neither necessary nor sufficient to obtain FDA approval.

~~~
pas
The lack of safety would affect those who can't afford the "approved" options.
:/

~~~
nybble41
Which is more likely to cause an issue, a treatment with no FDA safety
approval (but also no history of problems, and possibly safety approval from
some other trusted organization) or going without treatment altogether because
you can't afford the one with FDA approval and everything else is prohibited?

The existence of uncertified, or just _differently_ certified, options should
also put some downward pressure on the prices of the certified options
compared to the current monopoly environment. When everyone has to obtain FDA
approval there is no incentive to minimize the cost and plenty of reason for
established incumbents to make approval as expensive as possible to lock out
any potential competition. You can always push for "more safety" above and
beyond the level of risk reasonable people would be willing to accept.

~~~
pas
Mixing up cross-approval (from let's say the EU's EMA) with no history is very
dangerous. Exactly because snake fucking oil salesfolk rely on the latter.

Downward pressure is nice from unapproved/unknown stuff, but any kind of price
stratification _inevitably_ leads to harm for those who are at risk the most.

People already order generics on "ebay" and a lot of those shipments are
seized at customs, etc.

We know why prices are high in the US. And that's not because the need to go
through FDA approved trials, but because of patents and because every other
developed market got their shit together and negotiates in bulk with
suppliers. (Thus the US market is what pays for most of R&D.)

> FDA approval there is no incentive to minimize the cost

Do you mean the cost of going through the approvals? I think it should be
largely subsidized (and ratio should be computed from a reputation score of
the requester - of course using the lower bound of the Wilson score interval,
or something similarly fast converging).

> established incumbents to make approval as expensive as possible

True. But there are quite a lot of newcomers. VC money is pouring into medical
tech. (Though usually small promising research projects [and sometimes the
labs themselves too] are bought up wholesale by bigger companies that then do
the preclinical and clinical trials and the FDA approval.) This of course
incentivizes incumbents to consolidate.

Keeping the approval sane is largely the job of watchdog organizations (NGOs,
other civil advocacy groups, expert groups). Of course filtering the lobbyist
bullshit should be the job of everyone's beloved dear leaders. The incentives
are not stacked in the common man's favor, as we already know it for some
time.

Pushing for safety is of course the usual think of the children argument, but
I'm not pushing for more. I'm pushing for the same amount that is currently
there.

Furthermore, the FDA costs are almost negligible compared to the development
cost of new drugs. (Seriously, it's less than one percent.) And running
clinical trials themselves cost a lot of money. Which is something that cannot
be simply avoided. Because human biology is very complex, and even with these
trials getting the signal from the noise is not easy. Just look at the mess in
psychiatry ( [https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-
rev...](https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/) how
basically targeting one gene is [almost?] completely uselss)

------
peterwwillis
Most comments here are only focusing on medical devices; this is about the
practice of medicine, not shitty companies!

In dentistry/endodontics it was believed for a long time that installing a
post helped keep a crown and tooth intact for longer. But long term studies
showed posts can actually reduce longevity. In another case, a study showed
that a common type of knee surgery for pain is actually less helpful than a
placebo surgery.

The overarching problem is that it's (currently) very hard to evaluate the
real effect of practicing medicine on large swaths of the population, and then
using those results to change what doctors do. By getting doctors into a
continuous improvement cycle, we could identify costly, useless practices, and
improve health.

------
asdf21
Actual study:
[https://elifesciences.org/articles/45183](https://elifesciences.org/articles/45183)

Seems to impact nearly all medical disciplines..
[https://iiif.elifesciences.org/lax:45183%2Felife-45183-fig2-...](https://iiif.elifesciences.org/lax:45183%2Felife-45183-fig2-v1.tif/full/617,/0/default.webp)

------
graycat
Remember, in research it is easy to _conclude_ that some _treatment_ does no
good -- just do sloppy statistics. E.g., collect some data, flip a coin, and
report the results of the coin flip. That will work because the coin flip is,
in probability, _independent_ of everything else including the data and
treatment. Well that's not the only way to do sloppy statistics -- just pick
some measures that are relevant but not very good. And use a small sample
size.

E.g., there are some reports on how much data was needed by the Large Hadron
Collider (LHC) to conclude that the Higgs Boson DOES exist -- the amount of
data needed was beyond belief. Sooooo, anytime well short of that much data
they could have _conluded_ that the Higgs did not exist, that is, so far their
data failed to show that it DID exist. Same for any _treatment_ : If don't
collect enough data, use poor measures, let the data get corrupted, etc.,
i.e., do sloppy work, then will fail to find where the treatment did work and
conclude that it didn't.

------
hurrdurr2
I do remember watching a documentary about medical devices where it describes
a FDA loophole; if the device you are trying to introduce is similar to a
previous already approved product then the scientific rigor involved is
significantly less. I wonder if this has something to do with it.

------
chiefalchemist
I wonder how many of these are still covered by insurance companies. You'd
think they're in ideal (data collection) position to raise questions about
what works and what doesn't. Certainly, the insurance cos have more incentive
to raise red flags than Big Pharma.

~~~
DoctorOetker
upvoted this, but what happens with conflicts of interest? for example, what
if insurance companies insure both patients (to cover medical expenses) and
doctors (to cover malpractice lawsuit costs)? insurance company houses and
systems are typically set up such that the house will statistically win out (a
supposed small tax that pays for running the paper work, and manpower within
an insurance company). In such a setting, where insurance companies insure
both doctors and patients, the net motivation is no longer defending what
works, but increasing throughput of interactions...

------
reallydude
Challenging chemo:
[https://www.icnr.com/articles/ischemotherapyeffective.html](https://www.icnr.com/articles/ischemotherapyeffective.html)

Australia has a study that's pretty damning. Is that one of these practices
listed?

~~~
OldFatCactus
the website you linked reads like homeopathic conspiratorial junk. I would be
wary of anything they cite

~~~
reallydude
The papers cited (eg Morgan G, Ward R, Barton M. The contribution of cytotoxic
chemotherapy to 5-year survival in adult malignancies. Clin Oncol (R Coll
Radiol). 2004;16(8):549-60.) have been instrumental in removing Chemotherapy
as the defacto treatment of many cancers in Australia. The science is
definitive insofar as the inefficacy of the treatment.

The pharmacological industry resistance to these findings is unsurprising.

------
jpmattia
It's hard to see this headline as something other than a result of The
Irreproducibility Crisis.

------
manjana
Can anyone comment on the validity of this publication? I'm sceptical towards
the site given it's "flashy" title and I'm not well enough schooled in
distinguishing good vs. bad publications.

------
dannykwells
Everyone yelling about how the FDA is corrupt or should only screen for safety
are completely ignorant of history and basic facts.

Medicine is not like trying out a new coding framework - in some cases, like
cancer:

i) There is no ability to "try it out for your self" \- you are either all in
or not

ii) There are extreme risks to taking the drug (most cancer drugs are actually
somewhat to extreme toxic...just less toxic than cancer)

iii) You only get a few tries (since, you know, cancer will kill you).

iv) People are very desperate and will try anything

v) Data sharing is very very hard given the sensitivity of the information
(patient data) - making it effectively impossible to have an open data
exchange. (This is also due to the billions of dollars at stake on the pharma
side).

Try designing a system to overcome those challenges...and you will likely end
up with a central governing body that looks like the FDA. Anything else you
are going to get way way more drugs that don't work, which waste time and
money and kill people, which endanger patient privacy, or which are not
satisfactory to pharma or, ...

The fact is, the FDA is full of many of the smartest and hardest working
doctors and scientists in all of medicine. They take their jobs _extremely_
seriously. They are far more careful and rigorous than most academic or even
pharma-level researchers.

Do they get it wrong? Yes, occasionally. Not frequently. Occasionally. But
that also, is science. The FDA follows rigorous statistical procedures, but
even p<0.05 (or <0.001) still means false positives can happen.

~~~
criley2
With respect to cancer drugs, while generally no one can be prescribed
medication that has not passed clinical trials, in the case of terminal
patients who could potentially be saved by a drug currently in trials, there
is a system for allowing them access to the medication.

It's called "Expanded Access" (also known as "Compassionate Use")
[https://www.fda.gov/news-events/public-health-
focus/expanded...](https://www.fda.gov/news-events/public-health-
focus/expanded-access)

The other reason why data sharing is hard is because it costs on average over
$1,000,000,000 (1 billion USD) and 10 years of time to whittle over 10,000
drug candidates into a single new FDA approved NME (new molecular entity).

If you spent a billion dollars and a decade on a project, you will consider
the data you collected to be a trade secret and would protect it with
everything you can, because you have to now take this drug to market and make
up the billion you spent making this drug. If you simply shared all your
results, foreign competitors will simply copy your work without paying you,
and will then offer your product without the cost of R&D and regulation built
in: AKA they will charge 1/10 or less than you and you will struggle to profit
anywhere they are in the market.

~~~
LitFan
This is a good argument for government funded R&D, and government owned drugs.

They cost a ton of money and take forever to create, so the product becomes
very expensive, and rights to it are owned by the creating company. The
government empowers a body to regulate the flow of new drugs on the market,
and once the drug meets regulatory requirements, the government - in many
cases - pays on behalf of the patient.

A company is interested in making a profit on their time investment, whereas
the government could be content to break even (for the good of their
citizens).

If the government instead paid a company to do the R&D, maybe even providing
access to approved facilities where the work is to be done, the government
could then retain ownership of that product and then create agreements with
drug manufacturers to produce those drugs.

This solves a couple of problems, like sole producers of a product arbitrarily
increasing prices (it is expensive and time consuming to set up production for
a new drug, even if it is off patent), as well as supply issues (I don't think
right now the FDA has any rules saying a drug company must be able to supply a
certain amount of a drug to get their license to distribute).

~~~
jschwartzi
A good argument against government funded R&D is that several of our modern
medical challenges have been deeply embroiled in political debates.

AIDS/HIV(originaly called GRID) was originally considered a punishment for gay
men who were seen in the 80's as being immoral and sinful by the
establishment. Government-funded R&D for AIDS drugs would have been unlikely
for decades.

Another example is hormonal birth control. Birth control in general is a huge
political football with portions of the establishment wanting no access to
birth control at all. If R&D were government funded it's possible for these
people to prevent access simply by making funding of certain lines of research
illegal.

I think you have to look beyond the price of treatment to really see the
benefit of a healthy private enterprise here. There are still a ton of ways
that government contributes to new treatments, like grant funding for basic
research and development. But I can see your proposed future becoming a
dystopia, where whoever is in power in government makes it illegal to sell
certain patented drugs to their particular underclass of the week.

------
ptah
>to actively seek out independent, governmental and non-conflicted clinical
research

does this mean private sector pharma studies are to blame?

------
kevinalexbrown
I read the original study. I have a few thoughts. My partner is a physician
and I am an AI researcher working in medicine. I think a lot about doctors as
machine learning models, and RCT results like loss terms in a complicated
objective function.

What is the best learning rate for updating physicians (our models) from the
results of RCT's (part of our loss)?

The authors reviewed all articles in three journals from (generally) between
2003-2017. They didn't, afaict (please point me if they did), review the time-
to-correction (if any correction has been made). It takes some time before the
results of an RCT end up in established practice. I'm actually surprised it's
so small in many cases.

It's not like there's a database where the results of every RCT are
immediately updated and the physician model is retrained overnight on the new
data.

Even if there were, imagine if the learning rate (so to speak) were so high
that every discipline immediately changed their published best practices on
the basis of a single RCT?

Here's a cautionary paragraph from one of the excellent reversal studies they
use:

 _Several limitations of the study warrant discussion. First, because we
enrolled only 26% of eligible patients, our findings must be generalized
cautiously. The most frequent reason that patients declined enrollment was a
strong preference for one treatment or the other. Since patients ' preferences
may be associated with treatment outcome, our trial may be vulnerable to
selection bias. Participating surgeons may not have referred potentially
eligible patients because they were uncomfortable randomly assigning these
patients to treatment; this form of selective enrollment may also create
bias.26 Second, because the trial was conducted in academic referral centers,
the findings should be generalized carefully to community settings. Third, we
did not formally assess the fidelity of the physical therapists or surgeons to
the standard intervention protocols. Finally, our study was not blinded, since
our investigative group did not consider a sham comparison group feasible._
[0]

I'm less concerned about RCT to Best Practice time than from Best Practice to
Typical Physician Practice time. There is a cascaded model connected to the
'complex RCT loss' and it's discipline published practice down to individual
physician treating patients. Compressing the time from RCT to individual
physician is fraught with difficulties, but could be improved.

Finally, RCT is the gold standard, but it's not perfect and it doesn't always
clearly translate to the individual physician's model of practice. Many best
practices weren't established from RCT's either.

And an inconclusive result from an RCT is not the same thing as proving that
there's no difference in outcomes, but a proper statistician can chime in
there.

[0][https://www.nejm.org/doi/full/10.1056/NEJMoa1301408](https://www.nejm.org/doi/full/10.1056/NEJMoa1301408)

------
Moodles
Last Week Tonight with John Oliver did a segment on this recently:
[https://youtu.be/-tIdzNlExrw](https://youtu.be/-tIdzNlExrw)

