
IBM pitched Watson as a revolution in cancer care, but it’s nowhere close - notlob
https://www.statnews.com/2017/09/05/watson-ibm-cancer/
======
code4tee
IBM has some OK offerings but they've really screwed themselves with all this
hyped up marketing around Watson. They've become somewhat of a joke within the
real data science community and business executives are also increasingly
becoming disillusioned when projects based on IBMs mystery black boxes fail to
deliver.

The MD Anderson situation was a debacle of monumental proportions. Such claims
on cancer care are quite unethical in my view given that it gave a lot of very
sick people false hopes and diverted a lot of hospital attention and funding
away from stuff that actually works.

~~~
carlmr
We have a lot of IBM tools at my company, and they're all terrible. I always
heard these great things about Watson. Thank you for setting this straight. My
world view is healed.

~~~
moocowtruck
Lots of IBM where I work too, can't say a single good thing either.. honestly
Watson being a bust is not very surprising! The people who pay the bills get
taken to fancy parties and dinners by IBM, they give us free "trainings"(sales
pitches) where they try to convince us to use something that management has
already bought into... Honestly it's all very terrible :(

My sentiment of IBM these days is they sell software which is purposely made
bad...and people buy it because, well no one ever got fired for going with
IBM.

~~~
fredley
The pattern I've seen: IBM sells things that underdeliver, the solution is
always to _buy more IBM_ to fix the inadequacies of whatever it is you've
bought already. The go-to line is "Oh that's a know problem with X 3.4, if you
upgrade to X 4.5 with Y 2.7 that will fix that problem". This process never
stops.

~~~
kiddico
Which is partly due to their actual problem solving strategy. Instead of fix
the problem with X 3.4, they bought the company that makes Y so they could use
that solution. But Y has it's own problems of course. And even if it didn't X
should have just been developed along with something like Y. Though it's
unlikely that X was even made by IBM anyways.

------
yread
> On a recent morning, the results for a 73-year-old lung cancer patient were
> underwhelming: Watson recommended a chemotherapy regimen the oncologists had
> already flagged.

> “It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s
> treatment suggestion while discussing the case with colleagues.

> He said later that the background information Watson provided, including
> medical journal articles, was helpful, giving him more confidence that using
> a specific chemotherapy was a sound idea. But the system did not directly
> help him make that decision, nor did it tell him anything he didn’t already
> know.

I don't know, that doesn't sound bad at all. Perhaps it's just a case of
overhyping. But if a computer can do recommend the same treatment as an
expensive oncologist with years of training that's actually pretty good!

On the other hand it is quite baffling why doesn't FDA require a clinical
trial or any independent review of its capabilities before allowing its use in
clinic. It's a bit like Boeing certifying its own planes, I guess.

~~~
jessriedel
> why doesn't FDA require a clinical trial or any independent review of its
> capabilities before allowing its use in clinic.

Because it's a tool for informing doctors, not a medical treatment. We don't
have a clinical trial before PubMed rolls out a new search form either.

~~~
dekhn
It's definitely part of medical treatment. Laws are changing to include
"software medical devices" in health regulation. The FDA recently recognized
this as a gap.

BTW I've worked with regulated software and it's not pretty. Just to update
the firmware you have to fill out a ton of paperwork (because somebody died
once after a firmware update and the firmware update was found to be the root
cause). Regulation and review doesn't really make things tremendously safer,
it just provides an audit chain so you can point at something when it does go
pear-shape

~~~
ska

       Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap.
    

This isn't quite correct - FDA has always regulated software medical devices.
It is true however that the FDA is looking a lot closer at software than they
did in previous decades, and is releasing guidance (e.g. recently on
cybersecurity) that signals intentions moving forward. They definitely have
signaled that they don't believe they were focused enough on software.

It's important understand that FDA approval to market is always done in terms
of indications for use. You don't approve a device full stop, you approve it
(and evaluate it) in terms of the indications. Only in this context can you
determine if you have a 510k path (i.e. short cut) or need a PMA (i.e.
clinical trial). It is also important to realize that different panels in the
FDA (i.e. the type of device you are making) have different evaluation
criteria.

So "Watson" is unlikely to ever be cleared per se (it is a tech platform) but
you are likely to see individual Watson enabled systems file 510ks for
particular indications. If I recall correctly this has already happened.

~~~
ltowel
Do you remember what it's already happened for? Do you know what devices
they're equivalent to?

While I get the impression FDA and industry prefer 510ks as much as possible,
personally I don't see how this is close enough to anything out there right
now for that to pass the smell test.

My guess would be that some `Watson` equivalent stuff will be 510k'd in as
part of an MRI, CT or DBT imaging system or a CAD tool working on these
systems. I think it'd take a while though, I don't think the regulatory
science is there yet for actually showing value.

~~~
ska
Off the top of my head, I don't, but I'll have a look if I have a chance
later.

There have been approved CAD systems separate from imaging systems since the
90s.

"Value" is a nebulous term. FDA looks specifically for "safety & efficacy",
and the criterion for each are rather dependent on the intended use.

------
pixelmonkey
> Hospitals pay a per-patient fee for Watson for Oncology and other products
> enabled by the supercomputer. The amount depends on the number of products a
> hospital buys, and ranges between $200 and $1,000 per patient.

Sigh. Let's remember that in medicine (a severely resource-constrained
system), costs operate in a zero-sum environment. $200-$1,000 spent on Watson
is $200-$1,000 not spent on some other treatment or human labor. And, in our
current environment, specialist doctors will _always_ have to re-do the
diagnosis suggested by Watson, because guess who is held legally responsible,
in the case of a malpractice suit? Not IBM, but the doctor co-signing the
"recommendation".

------
danmaz74
TLDR: Watson has been trained for 6 years to build a knowledge base about 7
types of cancer; therapy recommendations are entered by experts from one top
US hospital.

Looks to me like a giant pattern recognition system. The best assessment I
found in the article:

> He said later that the background information Watson provided, including
> medical journal articles, was helpful, giving him more confidence that using
> a specific chemotherapy was a sound idea. But the system did not directly
> help him make that decision, nor did it tell him anything he didn’t already
> know.

~~~
carlmr
>Looks to me like a giant pattern recognition system.

Pretty much all attempts at AI are pattern recognition systems. I think the AI
hype will die down a bit, after we see more of the high profile AI programs
underdelivering on marketing's promises. But we'll get some gems that do
deliver something useful.

I just guess IBM bit off more than they can chew on this one. It's an old,
slow, uninnovative company, trying to do what the cool kids are doing by
putting all of their money on it.

~~~
romaniv
_> Pretty much all attempts at AI are pattern recognition systems._

All attempts in _modern_ AI. Most interesting research from the 60s and 80s
wasn't based on pattern recognition.

------
fredley
Cynical take, having worked in organisations that have bought into IBM: IBM is
not into curing cancer, it's into selling, and in particular _up-selling_ at
enormous scales (government and big business). The primary goal of the Watson
project has not been to cure cancer, or do anything in particular, but to get
existing customers to buy into extremely expensive contracts that include
'Watson' in some way. All the hype, which it is now becoming apparent was
artificially expanded to some extent, is just a means to this end. As with
most (all?) of their other products, it'll be oversold, underdeliver, but lock
organisations into even heftier contractual chains.

------
S_A_P
Its kind of sad to see what has become of IBM. It is now just another
consulting shop that wants to plant a slew of 3-400/hr consultants to run
projects. The big consulting companies best trick was to convince those in the
enterprise that hiring a boutique consulting shop is "risky" and that a big
4/5 consulting company is a "safe" option even though it is orders of
magnitude more expensive and not necessarily better, and in some cases worse
when it comes to specialized systems.

Source: years of working in Energy/Commodity trading systems market and seeing
several botched big 4 implementations that vendor/boutique shops can do at
1/10th the price.

~~~
zwerdlds
Having done big and small consulting work, I think part of the issue is that
waterfall still has quite a large buy-in. Smaller shops tend to go more agile,
but when you have a team of analysts chomping at the bit to document your
processes and say "yes, we can do this" (regardless of if they actually can)
that builds confidence.

And that confidence is what moves the project forward, until the customer sees
dates slipping, at which point it's too late.

------
madenine
I was under the impression "Watson" was just a brand-face for IBM consulting
services.

You're not buying thousands consulting hours and some big data services...
you're buying Watson

~~~
mholmes680
I'm on board with that if it means I don't have to pay for additional PVUs
[0].

0:
[https://www-01.ibm.com/software/passportadvantage/pvu_licens...](https://www-01.ibm.com/software/passportadvantage/pvu_licensing_for_customers.html)

------
notfromhere
As per every thread Watson is brought up in, it's mainly a catch-all term for
its consulting division. IBM's consultants aren't very good, and the good ones
move on to brighter futures.

This is why Watson is a disappointment and every Watson project ends in cost
overruns and failure.

------
arikrak
It's interesting that Watson appears to rely so much on processing text, which
runs into so many difficulties. It would be helpful if (for a given disease)
every patient's medical records were tracked as structured data. Machine
learning would then be able to find patterns of diagnosis much more easily.

(I guess companies like Flatiron Health are working in that direction.)

~~~
carlmr
But then all the NoSQL people working on data have to learn SQL ;).

~~~
equalunique
Funny that you mention NoSQL. MUMPS, the original 1960s database system
designed for medical data was NoSQL:
[https://motherboard.vice.com/en_us/article/3dkmg3/meet-
mumps...](https://motherboard.vice.com/en_us/article/3dkmg3/meet-mumps-the-
archaic-health-care-programming-language-that-predicted-big-data-2)

------
cs702
When results from "AI" don't match the hype nor justify the cost and hassle...
that's how AI Winters begin.

------
program_whiz
There's a bigger issue at hand here. AI generally has not made huge inroads
into disease treatment. I think this is because we haven't figured out a good
way to describe the diseases. Where it does work, we have a good way to
classify the data (i.e. a set of training images or clear metrics to feed it).
As Google/Verily learned the hard way, its not easy to fit digital models to
organic beings.

In the area of cancer treatment, my guess is that the reason specialists are
highly paid/trained is because the analysis and judgement required is highly
manual and varied. Surgical intervention, prescription medications, targeted
radiation treatment machines, patient psychology and counseling. Other than
maybe guessing the prescription based on symptom list, what could the AI
profitably help with?

~~~
tim333
>not made huge inroads into disease treatment. I think this is because we
haven't figured out a good way to describe the diseases.

Perhaps more likely, human doctors are actually quite good and AI has yet to
out do them.

------
rezlov
> On a recent morning, the results for a 73-year-old lung cancer patient were
> underwhelming: Watson recommended a chemotherapy regimen the oncologists had
> already flagged.

> He said later that the background information Watson provided, including
> medical journal articles, was helpful, giving him more confidence that using
> a specific chemotherapy was a sound idea. But the system did not directly
> help him make that decision, nor did it tell him anything he didn’t already
> know.

This type of thinking is reason true ML adoption is still sparse. Analytics
should be viewed as a supplement to decision making, not a replacement.

Manipulative marketing tactics by IBM and others are largely to blame. I'm
confident that over time, as the general population becomes better educated on
the topic, the perspective will shift.

------
hacker_9
> _“It’s been a struggle to update, I’ll be honest,” said Dr. Mark Kris,
> Memorial Sloan Kettering’s lead Watson trainer. He noted that treatment
> guidelines for every metastatic lung cancer patient worldwide recently
> changed in the course of one week after a research presentation at a cancer
> conference. “Changing the system of cognitive computing doesn’t turn around
> on a dime like that,” he said. “You have to put in the literature, you have
> to put in cases.”_

Wow the way they describe Watson is just like a massive book of documentation,
with some NLP functionality layered on top to interface with the doctors.
Keeping the database up to date must be a huge undertaking. Also this:

> _" Given the same clinical scenario, doctors can — and often do — disagree
> about the best course of action, whether to recommend surgery or
> chemotherapy, or another treatment. Those discrepancies are especially wide
> for second- and third-line treatments given after an initial therapy fails,
> where evidence of benefits is slimmer and consensus more elusive."_

Having an expert doctors opinion hard coded into the machine sounds good, but
it won't be as good as a doctor in the room who can pick up on all sorts of
other cues that I'm not sure how'd you even type into a computer. Things like
breathing patterns, reaction to tactile stimuli, fuzzy logic ('that hurts,
that hurts more'), facial expressions and so on.

Seems like this project has a long way to go, to get to the standard the IBM
marketing team are pretending it's already at.

~~~
teknover
"He noted that treatment guidelines for every metastatic lung cancer patient
worldwide recently changed in the course of one week after a research
presentation at a cancer conference."

Isn't this a good thing then that there's a digital assistant to ensure
doctors are up-to-date and provide the right treatment plans?

~~~
rscho
A digital assistant that signals guideline changes does not require advanced
technology. This reflects failure of the medical system, more than anything
else. Plus, the fact that guidelines change suddenly after some conference
should tell you that treatment in the case depicted here is defined entirely
on expert opinion. AI should seek to do better than just following expert
opinion. Conclusion: the Watson system is a failure.

------
dep_b
I guess one day IBM will sue Scott Adams for plagiarism.

------
gexla
IBM is nowhere close, but is there anyone closer?

------
gregman1
Who uses Watson besides IBM?

~~~
arkadiytehgraet
All those poor developers, whose management bought into the standard IBM
marketing lies and false promises.

------
pamqzl
IBM seems to have painted themselves into a bit of a corner by calling a bunch
of largely unrelated research projects by the name Watson.

~~~
ralfn
IBM is doing the smoke and mirros and selling bullshit game for a while now.

And it's intentional! They have a strategy of inventing new words that sound
all scientific (like 'cognitive computing') but there is absolute no
meaningfull research from them. Its like the whole company has become a
parody.

Any companies that allows their employees to talk to IBM has the problem that
afterwards their employees understand less of actual problem domain -- and the
knowledge they think they gained from that interaction .. its all 'IBM and
consultancy buzzwords' rather than the industry-standard terminology fueled by
actual research and science.

------
georgeecollins
I don't like the Watson logo. It's too busy. If you shrink it or squint when
you look at it you just see a blob.

~~~
mholmes680
The old one or the new one? I like the subtle "heart" shape in the new one,
especially the animation, to implicitly persuade me to love IBM.

[http://www.underconsideration.com/brandnew/archives/new_logo...](http://www.underconsideration.com/brandnew/archives/new_logo_and_identity_for_ibm_watson_done_in-
house_with_others.php)

