
Jefferies gives IBM Watson a Wall Street reality check - code4tee
https://techcrunch.com/2017/07/13/jefferies-gives-ibm-watson-a-wall-street-reality-check/
======
throwaway9980
IBM vastly over promises with their marketing. It is so frustrating to have to
answer questions from the CEO about why we don't solve all our problems with
magic beans from IBM's Watson.

I understand that this is what they want. They want to drive executives'
interest in the product, but I believe they do so at the expense of their
goodwill with the tech community.

Am I the only one who cringes when these ads air?

Edit: "magic beans" is harsh and it isn't that I don't think their tools are
good. My point is that they put you in a position where it seems very unlikely
to meet expectations.

~~~
wmccullough
I'm happy to be harsh because I've had years and years of experience dealing
with IBM.

Their tools are trash for the cost they want. The majority of their offerings
are overengineered and prone to failure in a production setting.

They are pros at selling to non-technical management, all the while IT
employees get stuck dealing with the aftermath.

We ended up getting stuck with the IBM version of work item management, called
Jazz Team Concert. From the "source control" to the actual setup and
configuration, the product was a nightmare. The worst part was when we looked
at license renewal. They wanted three quarters of a million dollars for
something that was essentially dog shit wrapped in cat shit.

In case you thought I might be on the fence with IBM, I'll add one final
clarifying point:

Fuck IBM.

~~~
deepGem
What perplexes me is that none of these seem to affect their services
business. 39 billion in services revenues is nothing to scoff at. I mean how
can a company build shitty products like this and be a world leader in
services ? Do these units operate like two separate entities.

~~~
thisisit
It is actually a cycle. Managers in an enterprise are judged based on their
ability to deliver low cost solutions ie "savings". IBM(or any service
business) is very good at selling these low cost solutions to managers.

We are starting a project and even with FTE's within the company, "consultant"
has convinced management to hire IBM resources because of cost. The high
sounding reason within the presentation twisted it to - FTE's are not expert
in this tech so they won't be able to do an efficient job.

But it is always that the end resources from IBM are even more incompetent.
They end up delivering a product which is terrible. Even the final handovers
and KTs are done terribly.

In the end, management has no option but to continue hiring IBM (or any
service business) to support the shoddily done project.

This doesn't mean all implementations from a services business is terrible. It
all comes down to cost and ability to spend. If you pay peanuts, you get
monkeys - this is aptly true.

To answer your last question - Yes there are different entities within IBM.
There is GBS which takes up most of the consulting and up-selling IBM products
and services. Development for example Watson is done by IBM Labs. A list can
be found here:
[http://www-07.ibm.com/in/careers/businessunits.html](http://www-07.ibm.com/in/careers/businessunits.html)

~~~
petra
So OK , you can do this cycle for some time, for a few people.

But doesn't the word get around ? does nobody in most enterprizes think long-
term when investing ?

~~~
sgt101
No, because if you think for the long term you can find that you are not
around after a short term.

------
xienze
> Jefferies pulls from an audit of a partnership between IBM Watson and MD
> Anderson as a case study for IBM’s broader problems scaling Watson. MD
> Anderson cut its ties with IBM after wasting $60 million on a Watson project
> that was ultimately deemed, “not ready for human investigational or clinical
> use.”

Well, can't say I'm surprised. I used to work on that project a few years ago,
basically the idea was that Watson would look at a patient's medical record,
figure out what medications they're on, what symptoms they had, etc. and
cross-reference all that with the medical knowledge it had ingested from vast
amounts of medical literature. In theory, Watson could figure out what
medications the patient should or should not be using, a proper course of
treatment, etc.

There were two major problems:

First, it turns out your medical record is mostly written in narrative form,
i.e., "John Smith is a 45 year old male...", "Patient is taking X mg of Y
twice daily", "Patient was administered X ml of Y on 3/1/2016", etc. In other
words, there's basically no structured data, so just figuring out the
patient's stats, vitals, medications, and treatment dosages was an adventure
in NLP. All that stuff was written in sentence form, and of course how things
were written depended on who wrote it in the first place. It was really,
really hard to make sure Watson actually had correct information about the
patient in the first place.

Second, all that medical literature that was being ingested? Regular old,
don't-know-anything-about-medicine programmers were the ones writing the rules
the manipulating the data extracted via NLP. Well guess what, if you're not a
domain expert you're bound to get things wrong.

Put those two things together and we would frequently get recommendations that
were wildly incorrect, but that's to be expected when you get garbage input
being fed into algorithms written by people who aren't domain experts.

~~~
filereaper
So if I understand correctly, there was dirty/malformed data which was
difficult to interpret, and when sent to a ML algorithm not tuned by a domain
expert led to bad results.

This applies to all ML work, why is Watson exempt from it?

Conversely if this the canonical failure case, why's everyone so harsh on
Watson?

~~~
PKop
Perhaps because they charged $60 million for it?

~~~
semi-extrinsic
What I sincerely don't get about IT is why customers sign contracts where they
fork over the mulah even when _nothing of value is delivered to them_. Why the
actual fuck?

I mean, if Delta ordered a super-advanced new fancy 787 as a tech demo, and
Boeing came back with "sorry, our experts were unable to make anything work,
so we're going to take all your money and not give you anything back", Delta
would rightly tell Boeing to fuck right off and demand that they refund the
money.

~~~
mdpopescu
If you buy a product, like Windows 10, you pay X and get the product, with
whatever it contains. You can return that and get your money back.

If you buy a service, or a customized product, then you pay for the time and
effort it takes to build / customize that product. Returning that time and
effort would be difficult :)

------
throwaway111991
I was at the CogX artificial intelligence summit in London a couple of weeks
ago, and IBM were there in full force.

I made several rounds around all of the stalls, and sat at the bar for a
couple of hours with friends, and the whole time I could see the IBM stall,
with 4-5 people there, WATSON plastered everywhere and nobody talking to them.

So I went over. I got talking to one of their technical people there,

I am highly experienced in Deep Learning so I started talking about Neural
Nets, and he went blank, and admitted he didn't know much about that. I
inquired about WATSON's technology and he couldn't answer telling me he didn't
know.

I asked about the main use cases, and what makes WATSONs offering better than
Deep Learning, he couldn't answer, or even compare on basic levels.

I asked him "What are the coolest uses of WATSON you've seen" and he
immediatly went into a canned response about WATSON diagnosing cancer (a
project I had seen and was familiar with) we spoke a few minutes on that, and
I asked what other cool projects WATSON had been used on ... he had nothing,
and I mean literally nothing.

very disappointing

~~~
jimmaswell
Kind of reminds me of these frustrating anti-nuclear advocates that set up a
table with some posters at my old college one day. They couldn't give me any
real answers why I should support solar wind and hydro over nuclear besides
nuclear being "old technology". They had no response to the fact that this
state isn't ideal for large scale use of any of the green alternatives or that
much more deaths directly result from solar panel installation accidents alone
than nuclear as a whole. They couldn't tell me specifically what was
supposedly dangerous about current waste storage techniques. The worst part
was I googled the organization they were there on behalf of and right on their
webpage was an explanation of the failure of nuclear plants in the state to
stay profitable (I forget why but it seems really odd since you get so much
power from them). I guess they cared more about getting more mileage for less
effort dishonestly swaying the local types who are automatically turned off by
"radiation" and big industrial buildings.

Was that event not geared towards technical people overall or was that stand
an exception?

~~~
Spooky23
It sounds like you trolled some kids passionate about a subject to argue for a
technology and industry that you don't really understand.

Were you prepared to compare the plight of the solar installers who fall off
of roofs and get injured to the construction workers injured on the job during
various nuclear plant construction projects?

Did you count the various victims at Chernobyl? (Epidemiologists project at
least 4,000 deaths)

Were you ready to justify why states like New York needs to subsidize nuclear
plants with $500M in direct savings because they are not economically viable
to operate anymore?

In your case, those kids were being nice.

~~~
jjoonathan
> Were you prepared to compare the plight of the solar installers who fall off
> of roofs and get injured to the construction workers injured on the job
> during various nuclear plant construction projects? > Did you count the
> various victims at Chernobyl? (Epidemiologists project at least 4,000
> deaths)

The Banqiao dam killed 170k, and that's just the largest dam failure, there's
a heavy tail following it. Wind and solar don't have any single incident so
dramatic, but they require so damn much manufacturing, construction, and
maintenance to generate so little energy that they don't do particularly well
in the comparisons I've seen.

[https://en.wikipedia.org/wiki/Energy_accidents](https://en.wikipedia.org/wiki/Energy_accidents)

Admittedly this is a surface level understanding, but you speak with the
confidence of someone who knows better while providing even less concrete
information. I'm hoping you've got deeper knowledge to back your position up
and that you're willing to share.

> Were you ready to justify why states like New York needs to subsidize
> nuclear...

Everyone in the energy industry gets subsidies. Big subsidies. What matters
are costs per kwh, and nuclear seems to win there too.

[http://instituteforenergyresearch.org/analysis/electric-
gene...](http://instituteforenergyresearch.org/analysis/electric-generating-
costs-a-primer/)

Are these numbers wrong? If they're wrong, and fission is actually much more
expensive than it looks, is it because fission is fundamentally hard / risky
or because the updated prices factor in risk due to irrational public
perception?

~~~
Spooky23
I agree that there is plenty of government intervention of energy. But I don't
think there are other electrical generating technologies that require direct
cash infusions of that scale when in a mature operational state just to remain
a going concern.

The costs for fission are high because they must operate at peak capacity to
recoup their capital costs. That was fine when electricity generation was 100%
regulated and demand curves were steady. Now things are changing faster and we
have market pricing for generation. Market forces price peak demand higher
than base demand, and smaller, more nimble and cheaper generators are eating
the lunch of big 1960s nuclear plants.

The risks and technology issues are also serious and very expensive, but we
have pushed out the costs into the future and taxpayers who aren't born yet
will be paying those debts. Nuclear advocates always ignore the costs
associated with securely storing waste products for a period of time
approaching recorded human history. That said, those costs aren't priced into
nuclear energy.

------
pgodzin
The marketing has made it very hard to have a real conversation about IBM
Watson. There is no such singular thing as "Watson". IBM offers a ML solution
for health, for NLP, chatbots, etc. They all have very different capabilities
and require different levels of machine learning. The marketing is BS, but
most of the tech is real - if you give IBM your data, let them train a model
on it, and communicate what you want, you will get an end-to-end custom
solution. It's just not the magic IBM sells in its marketing videos.

Disclaimer: SWE at Watson Health

~~~
shostack
How do they market it to a tech audience who presumably see through this sort
of thing due to a technical understanding of the subject matter?

~~~
pgodzin
For me, Watson Health is actually working on a lot of interesting work in the
healthcare space that will eventually lead to improvement in care and
outcomes. And hospitals are exactly the kind of clients that IBM handles well
- complicated data requirements, government regulations, etc. It's a very
difficult, and often frustrating, space to get into.

------
save_ferris
I don't think this isn't limited to IBM, my partner's PE firm recently hired a
small consulting group touting a "revolutionary, AI-driven" real-estate
analysis product that has zero AI whatsoever. It's basically a custom
spreadsheet tool that they're claiming to be building AI on top of as they
consume company data, but for a few hundred grand per year, they have a basic
CRUD app on Azure with a reporting tool using D3 visualizations. But they
think it's AI.

It's almost like 2016-17 were gold-mine years for marketing buzzwords and some
companies are closing deals with no real execution plan for what they're
selling.

~~~
ballenf
Later this year I predict they'll relaunch the product as being built on the
blockchain.

Problem solved.

~~~
visarga
"Deep Learning Blockchain"

------
RcouF1uZ4gsC
Watson is one of the biggest empty marketing slogans ever. The marketing makes
it almost seems like General AI able to easily solve your pressing problems if
you pay IBM money.

As a non-expert, it seems like the top end researchers are working for
Google(Hinton, Bengio, etc), Facebook(LeCun), Baidu, Uber (ex CMU faculty). I
don't really see a lot of machine learning research coming out of IBM
comparable to the others.

IBM seems to running on the fumes of it's previous greatness while burning the
ship to generate stock market returns.

~~~
chadcmulligan
> IBM seems to running on the fumes of it's previous greatness while burning
> the ship to generate stock market returns.

They have great research, but as always with lumbering behemoths they have to
earn enough from the tech to keep the behemoth going, and with their bread and
butter (hardware and consulting) drying up they're trying to maximise returns
and having to oversell their remaining products

------
verdverm
I quit IBM Watson 5 weeks ago. Here is why IBM is suffering.

[https://verdverm.com/2017/07/a-new-
chapter/](https://verdverm.com/2017/07/a-new-chapter/)

PM me for more ;]

~~~
keithpeter
Not my circus and not my monkeys, but how long did you work for IBM?

Your blog post is short on actual dates (a time frame)

Best of luck

------
batmansmk
IBM offered a day of Watson training in San Francisco about a year and a half
ago. As engineers working with classifications, we were interested to compare
the results of Watson to our algorithms, but also look at the API, the
communication, the community etc.

It was a classroom nightmare. WIFI not working, Bluemix required for all
workshops not working at that time, teachers very new on the topic themselves
(one confessed he only knew Watson for a couple of weeks before the training),
no announcement, no nice moment to socialize or build up a community, no
coupon given to try on our own after, ...

And... the algorithms didn't work at all. The sentiment analysis was
classifying as really positive the sentence: "I wasn't happy at all by the
service" due to 'happy' and 'all' present in the sentence.

------
bradneuberg
I've had two encounters with IBM Watson that left me unimpressed. The first
was using the IBM Watson Speech Transcription service (give an audio file and
get text); the results were pretty bad vs. Google's, for example. The second
was in their recent integration into Star Trek Bridge Command (which is an
amazing game BTW!); the speech recognition results were pretty bad.

------
bischofs
What does IBM even do anymore? Is it some bizarre set of buildings where they
just play with computers and print money? What product do they sell? Whom do
they sell it to?

I am genuinely curious...

I have a comp sci degree and worked in different industries relating to
software and have never even seen or touched any IBM tech except for those old
cash registers.

~~~
lucaspiller
I was working for a UN organisation last year and some of my work overlapped
with another project that was being done by IBM. They were hired to make our
web applications more secure by adding 2FA and user access control middleware,
using their Websphere family of products (IIRC) and building custom
integrations.

I didn't get involved much on the project, but from what I understand the
developers sent to work with us were useless, the products were bad or at
least poorly suited for us, and what they were going to build was a huge waste
of money. (However the last point was more to do with deep organisational
issues where I was working).

The main product IBM sold was 'mininising risk' which the director of our
department loved - as it meant if the project failed he would have someone to
take the blame other than himself. The old saying 'nobody got fired for
choosing IBM' is still true.

------
code4tee
Within the data science community Watson has long been viewed as snake oil.
Glad to see less technical and business folks are finally smelling BS too.

------
hbarka
Lotus Notes. Why this garbage of an email system is still perpetrated by IBM
explains IBM. It worked back in 1999 but pity you if you're in a company still
using it and the CIO still putting upgrade patches to it.

~~~
kranner
Still using it? We went from Gmail (for business) to Notes because IBM Sales
managed to convince the leadership that they could track and file everything
better in Notes. Now we have to deal with this shitshow of an app where it
takes 10 clicks to open a message because you have to file it under a project
first and good luck if you ever need to search for anything.

------
daxfohl
> and let’s be real, things would look much worse if Google, Microsoft and
> Facebook were added to this table

Umm, so add them? And Nvidia, Intel, Baidu, Uber, Tesla? Anybody else? That
single chart would actually be more interesting than the entirety of this
article.

~~~
RobLach
The techcrunch article says that but the Jefferies analyis mentions them in
the notes for those charts saying they're compareable to Amazon.

------
mark_l_watson
In the last 6 weeks, I have been called by two reporters (Wall Street Journal
and Reuters) for background on AI. I talked with the Journal reporter for
about an hour, covering 'everything.' However, the Reuters reporter only
wanted to talk about IBM Watson - we just had a short talk.

I have seen a lot of negative press on Watson, but really, it can be evaluated
like any other API to see if it meets your needs.

~~~
isubkhankulov
the submarine - paul graham

~~~
mark_l_watson
Thanks, interesting read!

------
zitterbewegung
While getting my undergraduate IBM said they were going to give an overview of
the Watson system they used to solve Jeopardy . I skipped it but there were
some professors that went to it. The professors walked out saying that they
were using Watson as some kind of marketing term. They gave no technical
details either . That's how I found out that Watson was a marketing gimmick.

------
zaphod_ibm
Disclosure: I work for IBM.

Watson is not a consumer gadget but the AI platform for real business. Watson
solutions are being built, used, and deployed in more than 45 countries and
across 20 different industries. Take health care alone -- Watson is in
clinical use in the US and 5 other countries, and it has been trained on 8
types of cancers, with plans to add 6 more this year. Watson has now been
trained and released to help support physicians in their treatment of breast,
lung, colorectal, cervical, ovarian, gastric and prostate cancers. By the end
of the year, the technology will be available to support at least 12 cancer
types, representing 80 percent of the global incidence of cancer. Beyond
oncology, Watson is in use by nearly half of the top 25 life sciences
companies, major manufacturers for IoT applications, retail and financial
services firms, and partners like GM, H&R Block and SalesForce.com.

We have invested billions of dollars in the Watson business unit since its
inception in 2014, with more than 15,000 professionals, and more than a third
of IBM's research division is devoted to leading-edge AI research. When you
consider the vast scope of IBM's work in AI, from Watson Health to Watson
Financial Services to the emerging Internet of Things opportunity, it is clear
that no other company is doing AI at the scale of IBM.

By the end of this year, Watson will touch one billion people in some way ·
Watson can “see,” able to describe the contents of an image. For example,
Watson can identify melanoma from skin lesion images with 95 percent accuracy,
according to research with Memorial Sloan Kettering. · Watson can “hear,”
understanding speech including Japanese, Mandarin, Spanish, Portuguese, among
others. · Watson can “read” 9 languages. · Watson can “feel” impulses from
sensors in elevators, buildings, autos and even ball bearings. · At IBM, there
are more than 1,000 researchers focused solely on artificial intelligence

------
sumoboy
IBM should ask Watson how to fix the company first, then it would have some
credibility. They don't treat there employees very well either, but neither
does Oracle so why would anybody waste time working for losers.

+1 "dog shit wrapped in cat shit" .. that is awesome.

------
zhanwei
So IBM Watson can do all the smart and complex stuff but we still need human
to do the dumb stuff like importing excel files where the cost outweighs the
benefit of getting Watson to do the smart stuff.

------
atsaloli
This reminds me of a Linux Journal piece I did 5 years ago on system
administration of the Watson supercomputer (after they got their 15 minutes of
fame on Jeopardy):

[http://www.linuxjournal.com/content/system-administration-
ib...](http://www.linuxjournal.com/content/system-administration-ibm-watson-
supercomputer?page=0,1)

They brought in a sysadmin after they got up to 800 OS instances. Before that,
it was just 3 part-time researchers handling the system administration duties.

------
throwaway91111
This is only the beginning of selling AI as a panacea. People, if it does
something useful, _there is a term for it_. The only reason not to use that
term is to AVOID direct comparisons.

------
dangero
Anything that IBM puts out remember they are a consulting company, so they
want to generate a huge brand name. That allows them to charge the consulting
prices they need to charge to make this business work for them. IBM Watson is
a collection of sort-of-working AI related APIs, but it gets A LOT of press.
If they can create an AI brain, then people will believe they can do anything
for them in the tech arena and that's the goal.

~~~
heisenbit
This is true. However they are also accounting towards AI services. The way it
works is that at the beginning a few existing projects are re-classified under
the new heading. But over time the revenue is expected to grow. It did not.
And other revenue is also not growing - that was why they started this new
initiative in the first place. Unless one switches in time to a new growth
story in a credible manner one has to make it work at some point in time.
Eventually even Wall St. gets a clue and then real problems start.

------
Vijayasankarv
I do work for IBM. Here is my personal take for what it is worth.
[https://andvijaysays.com/2017/07/17/ibm-watson-is-just-
fine-...](https://andvijaysays.com/2017/07/17/ibm-watson-is-just-fine-thank-
you/)

------
rv816
Finally someone publishes what everybody in the industry has long since known,
especially in healthcare.

------
endswapper
[https://www.forbes.com/sites/jasonbloomberg/2017/07/02/is-
ib...](https://www.forbes.com/sites/jasonbloomberg/2017/07/02/is-ibm-watson-a-
joke/)

------
JunkDNA
Still waiting for the peer reviewed publication in a prestigious medical
journal that demonstrates doctors using Watson get better outcomes for their
patients.

------
dmritard96
somewhat off topic but I find their use of 'Watson' to be rather outrageous as
he was a big part of IBMs Jew tracking systems installed in concentration
camps during world war two. I suppose I already looked at IBM as an org that
really does not own this as they should but its particularly bothersome that
they would use his name as a flagship of their marketing efforts.

~~~
ballenf
I had always assumed that Watson was a reference to Sherlock Holmes'
assistant. "Elementary my dear Watson" and such.

Googled "ww2 watson" to find the wiki with the info you reference.

Does IBM make clear which Watson the name's a reference to? I have to think
you're right, but the Sherlock Holmes reference makes so much more sense to me
in the sense of being the assistant while also stroking the customer's ego as
the brilliant detective who hired Watson.

------
snissn
I spent months as a fully qualified lead trying to buy a Watson product and
simply couldn't. Had calls rescheduled, canceled, got on the phone and a
kafka-esque experiences with a sales person. We gave up and just built out
what we wanted to buy..

------
hacksonx
Fundamental software problems here. Probably the reason why software is being
marketed as service more and more. IBM might be moving a little too fast,
especially from a sales perspective but their systems offer features that will
define the future.

~~~
tdb7893
The problem is that their systems aren't very good so if it's just APIs then
it's more than likely a cloud provider would provide similar APIs and eat
their lunch. The only thing they seem to really be able to do in this space is
consulting

~~~
daniellerommel
Did you ever see the comparison between IBM Watson and WolframAlpha?
[http://gizmodo.com/5766124/the-difference-between-watson-
and...](http://gizmodo.com/5766124/the-difference-between-watson-and-
wolframalpha) Worth checking out

------
polm23
Best story I heard from a guy who claimed to have worked at IBM in a bar was
when he went to meet a client and they asked, in all seriousness, where the
talking hologram from the commercial was.

------
komali2
I think it's one of the last great investments - Watson will make IBM an
astonishing amount of money, right up until it and the technology its
spearheading make money irrelevant.

------
fredsanford
IBMs solution? More offshore and H1B and force everyone into the office.

Circling the drain.

------
BucketSort
As soon as I saw "now with Watson" on H&R block's windows, I knew it was over.

------
smegel
> Unfortunately, IBM is struggling to bridge the gap between client needs and
> its own technological capability.

AI in a nutshell.

------
shard972
Only took like 4 years for someone to catch on. I remember seeing right away
when they put Watson on jeopardy it was going to be a giant pr stunt.

------
bitmapbrother
This is simply the media and analysts catching up with what everyone familiar
with Watson already knew - that it was nothing more than marketing bullshit
designed to project IBM as a leader in A.I. Watson is a lot like IBM's cloud
initiative - a service so bad that they don't even use it internally, but have
no problems conning their customers on its value.

------
dpkonofa
This and the comments below are really depressing to me. Watson seemed like
such an exciting piece of tech and something that had the potential to change
the world and now I feel like the shareholder's virus has stagnated it to the
point of it being worthless. I've heard multiple stories where the staff
that's assigned to demo and talk about Watson have no idea what they're
talking about and that the marketing, management, and finance people don't
have any inkling as to what is special about Watson. They only care that it's
not currently making them boatloads of money, despite the fact that it
absolutely could. I guess I'll have to move my excitement to Google and
Apple's machine learning attempts.

~~~
abtinf
Disclosure: I work in developer advocacy at IBM.

If you are looking to build a project using cognitive technologies, you
certainly should investigate your options. Each offering has different
strengths with different workloads and small improvements in quality can go a
long way.

If you want to checkout the services for yourself without all the hype, take a
look at the APIs on
[https://www.ibm.com/watson/developercloud/](https://www.ibm.com/watson/developercloud/).

~~~
sunyataishere
Whoa - did you just provide a link to Watson APIs that include an service with
the name of 'Phd API' and state that this info was without the hype?

You truly are an advocate for developers!!!

Oh IBM... I must thank you for the years of WebShere Commerce misery forced to
downgrade JDKs to 1.4x while everyone else was developing on 7. Sigh

~~~
covener
When Java 7 was released in 2011, there was already a 2-year old WebSphere
Commerce release supporting Java 6. I don't think you can pin those years you
were on a superceded 5-7 year old stack on IBM.

