Hacker News new | past | comments | ask | show | jobs | submit login
Jefferies gives IBM Watson a Wall Street reality check (techcrunch.com)
434 points by code4tee on July 14, 2017 | hide | past | favorite | 278 comments

IBM vastly over promises with their marketing. It is so frustrating to have to answer questions from the CEO about why we don't solve all our problems with magic beans from IBM's Watson.

I understand that this is what they want. They want to drive executives' interest in the product, but I believe they do so at the expense of their goodwill with the tech community.

Am I the only one who cringes when these ads air?

Edit: "magic beans" is harsh and it isn't that I don't think their tools are good. My point is that they put you in a position where it seems very unlikely to meet expectations.

I'm happy to be harsh because I've had years and years of experience dealing with IBM.

Their tools are trash for the cost they want. The majority of their offerings are overengineered and prone to failure in a production setting.

They are pros at selling to non-technical management, all the while IT employees get stuck dealing with the aftermath.

We ended up getting stuck with the IBM version of work item management, called Jazz Team Concert. From the "source control" to the actual setup and configuration, the product was a nightmare. The worst part was when we looked at license renewal. They wanted three quarters of a million dollars for something that was essentially dog shit wrapped in cat shit.

In case you thought I might be on the fence with IBM, I'll add one final clarifying point:

Fuck IBM.

Hey man, you're not alone, even internal teams at IBM using Jazz Team Concert have been calling it "dog shit wrapped in cat shit" for years now.

Don't be so harsh on all of IBM, its like hundreds of companies under one banner. Not all of them have the power to influence other products and often times end up stuck with it and cannot move on to other tools (i.e git).

More often than not, customers have higher priority on bugs than other teams dog-fooding the product. Its insane.

> Don't be so harsh on all of IBM, its like hundreds of companies under one banner.

I think you mean don't be so harsh on the people on the ground at IBM. If the issue is IBM's internal structure is fractured and disorganized then I think it's perfectly fair to blame the company.

> Don't be so harsh on all of IBM, its like hundreds of companies under one banner. Not all of them have the power to influence other products and often times end up stuck with it and cannot move on to other tools (i.e git).

The do this so different divisions can lean on each other and benefit from the IBM brand name. If they want the benefits of this shared branding then the deserve the downsides from it too.

I've noticed many companies recently thinking they can have it both ways (samsung and sony come to mind). Live by the brand, die by the brand!

Yeah, IBM (and Oracle, and SAP and ...) have always been like this.

My own anecdote -- we had a need for a GIS/charting/mapping system. IBM responded to the RFP as did many others.

IBM proposed an entire office automation system with a minor charting system hanging off the side somewhere. Most of the components were unfamiliar to everyone. Though maybe there was Wordperfect or Lotus Notes in there. I remember meetings with IBM always had whole bunch of shiny suits whereas more focussed proposals just had one or two people visit.

After one or two rounds of pointless meetings our fairly technical director told them to go fuck off and never come back.

My first dev job was with a civil engineering consultancy that also built a suite of tools for hydraulic modelling, geo-spatioal and logging and telemetry use cases. We used the IBM ILOG JViews framework for a lot of our software and honestly it was a pain to use and clunky as hell.

The worst thing though that was despite paying an inordinate amount of money to license this software each year and receive support for it they would not fix a bug we found in their mapping modules.

Specifically it was their use of float for X and Y coordinates in the IlvPoint object that you pretty much had to use to make a map work with their tech. In certain parts of the world there is an extra digit on the Y coordinate (projections) and the float they insisted on using couldn't store the whole value, meaning they would just round it up or down causing inconsistencies between our representation of a client's dataand the client's representation. That was an absolute no-go for us and the workarounds IBM suggested to us were absurd and bordering on insulting honestly.

We had a lot of trouble with our affected clients because of this and all we could tell them was that it was an IBM issue.

The absolutely refused to fix their product and eventually sold it to Rogue Wave, washing their hands of it. By that point though we had already decided on a few alternatives lest we start to lose customers (who were essentially massive utility companies from around the globe).

Fuck IBM.

What perplexes me is that none of these seem to affect their services business. 39 billion in services revenues is nothing to scoff at. I mean how can a company build shitty products like this and be a world leader in services ? Do these units operate like two separate entities.

It is actually a cycle. Managers in an enterprise are judged based on their ability to deliver low cost solutions ie "savings". IBM(or any service business) is very good at selling these low cost solutions to managers.

We are starting a project and even with FTE's within the company, "consultant" has convinced management to hire IBM resources because of cost. The high sounding reason within the presentation twisted it to - FTE's are not expert in this tech so they won't be able to do an efficient job.

But it is always that the end resources from IBM are even more incompetent. They end up delivering a product which is terrible. Even the final handovers and KTs are done terribly.

In the end, management has no option but to continue hiring IBM (or any service business) to support the shoddily done project.

This doesn't mean all implementations from a services business is terrible. It all comes down to cost and ability to spend. If you pay peanuts, you get monkeys - this is aptly true.

To answer your last question - Yes there are different entities within IBM. There is GBS which takes up most of the consulting and up-selling IBM products and services. Development for example Watson is done by IBM Labs. A list can be found here: http://www-07.ibm.com/in/careers/businessunits.html

> But it is always that the end resources from IBM are even more incompetent. They end up delivering a product which is terrible. Even the final handovers and KTs are done terribly.

At an old employer who I'm no longer with, I asked why there's a part of the application in Java (the rest has always been in .net). They told me that an army of IBM consultants made that decision and now there's a part of the code in Java and it sticks out like a sore thumb.

So OK , you can do this cycle for some time, for a few people.

But doesn't the word get around ? does nobody in most enterprizes think long-term when investing ?

No, because if you think for the long term you can find that you are not around after a short term.

No one at this level thinks further than their next annual bonus. If the wheels come off after who cares, they've moved on to do the same thing to the next company...

> I mean how can a company build shitty products like this and be a world leader in services ?

IBM is optimized in enterprise (including government) sales. Product is not quite irrelevant, but it's a specialized enough market that often mediocre substantive quality does quite well, because the only competition that matters are the others that are optimized for that market.

They sell products to people who will never personally use them. The executive signing cheques on the golf course has a PA to deal with its day to day use...

c'mon they aren't the IT's crown jewel but you're doing a disservice to them. while products have accrued years of shit, as every old product does, it is extremely frustrating for very large enterprises and governments to talk with system integrators or even startup - most often than not their product is totally not ready to cope with the audit requirements, security requirements, they often don't even have a clue of what enterprises need on that front neither how to handle all the red tape around security certifications, legal requirements, disclosures requests that those clients usually have.

Click that link from inside HN...

I knew what happens :)

> I mean how can a company build shitty products like this and be a world leader in services

Wasn't there a saying that no one got fired for buying IBM.

It's used in real life in a non joking manner.

I've seen it many times.

Yeah, in like 1975.

Or 2017. Things may change, but not as fast as some people hope.

if i ran a company, and the manager/*EO level personnel purchased IBM, i would've fired them.

but you didn't, right?

Traditionnaly at that scale, the people buying the product and the ones ending up using it are completely insulated by layers of bureaucracy. As long as the product can somewhat work, there won't be much repercussions for making shitty choices: the feedback won't go up enough or be ignored (as it is inconvenient)

> insulated by layers of bureaucracy

Misinterpretation is a recursive problem. The best description - by far - of this process is "How a plan becomes policy" (aka, "How Shit Happens").


They optimized for the thing which is currently important: Convince decision-makers that your product is the way to go - by any means necessary. This is as much a reflection on the tech peoples inability to actually influence companies buying decisions as it is on IBM. If companies started to value sound technology IBM (and Oracle and SAP ..) would shift its priorities.

A lot of that is consulting on and reselling of SAP products. There are many companies in the world that want to move to SAP and haven't been burned by IBM yet.

IBM GS also specializes in complexity. They sell their services to management, but then they come in and make the most complicated system they can sustain, which is too complicated for the full time staff to want to deal with.

As the staff back away slowly from the creep, that just makes more space for IBM, and they sell progressively more service hours. It's pretty much their MO.

>> Do these units operate like two separate entities.

Broadly, yes, IBM is run as various different divisions, and the divisions start very high up the organisation, only a couple of levels from the top. They rarely interact in my experience (as a software guy there for some years in the past)

IBM are not alone in getting away with this. Buyers of big brand products need to shape up and talk with their engineers. It's a farce.

Vendor lock-in.

This sounds no different than my experience with some Oracle and SAP products that we had the unfortunate pleasure of supporting.

Oracle's Responsys is definetly dog shit wrapped in cat shit. I hope you'll never had to deal with that dumpster fire.

Even just trying to install Oracle 11g correctly was hard.

I think it expected to be installed in /usr/local and company procedures demanded that it was installed in /opt (or the other way around).

For this it would consequently misplace some permissions on some of the files causing a later stage in the installer to fail and bork the installation.

Back in about 1999, I was working for a big corporate and one of our Oracle admins lost her shit and had a meltdown, think it was around 8i era. She just got up and walked out shouting "fuck this fucking shit, I quit". This was on a freshly delivered HP N-Class HPUX box and they just couldn't get it to run after a month of going back and forth between HP and Oracle.

I bumped into her last year at Ansible Fest London and she was still angry about it 17 years later!

And that's what Oracle does to your soul :)

Edit: that N class and Oracle NEVER got deployed in the end. It just sat there unused, licenses weren't renewed, it got sheepishly powered down after a few months to save electricity and it got chucked in a skip after 5 years. About £200k of expenditure down the pan.

licenses weren't renewed, it got sheepishly powered down after a few months to save electricity [...] About £200k of expenditure down the pan.

I guess you could easily lost even more money. Good thing they stopped it at £200k.

Yeah a big chunk of that was consultancy, support, Oracle licenses. TCO was going to be around £500k for 5 years.

The whole platform got replaced with SQL Server on a clone PC server in the end which cost less than £10k. No fan of Microsoft but there was no competition. And the tooling was far better!

When I get issues were vendors point fingers at each other I three way call them, tell them what's up, hit mute, go get a coffee, and let them figure it out.

We did that. They were still finger pointing two hours later.

Legal guys said it wasn't worth suing them because we'd have to pay up then or lose the support contract.

Oof. You should bill them for your time. I sent HP or Dell (can't recall) an invoice for $800 in my early 20s for my time wasted on a support call where a "support tech" had asked me to reinstall Windows as a way to fix a disk that failed in a RAID array. OS was AIX.

Hilariously Oracle database can't run on Oracle Linux OOTB...

While I'm not a fan of Oracle, I have to correct you: Oracle DB runs just fine on Oracle Linux. It installs exactly the same way that it does on RHEL, same checklist.

That's kinda my point - why do they ship a Linux distribution without the prerequisites for their flagship product preinstalled?

Probably because they don't want to deal with the overhead of maintaining patches to RHEL source.

Then they fully deserve the ridicule. It's like Ford selling you a car but you have to bring your own tyres if you want to drive it away

Just as long as you follow the defaults without changing anything I guess :-)

See it wasn't that hard. Stupid company procedures made it hard-er and even then iff you know what you are doing it is still trivially easy. It is so easy even sysadmins can do it. Next next finish have a cookie back in your gimp cave now easy.

I work for IBM as a Developer Advocate, though my opinions here are my own. One of my goals as a Developer Advocate is to represent the shift in buying power from "non-technical management" (as you described it) to individual developers and other technical staff. If those who have to use or have to integrate with the software are a part of the decision making process, then I believe that this will result in better quality software. IBM is a very large company, with many different offerings. However, I think you'll see this shift towards a developer-focused mindset happening more-and-more within IBM over the next few years.

Non-technicals for most corporations will always make the buying decisions because it's about $$$. There is IBM's fundamental problem, "next few years" which means we're guessing today and guessing tomorrow.

This message has just as little substance as the other IBM employee's comment. Care to share any real examples of what

> However, I think you'll see this shift towards a developer-focused mindset happening more-and-more within IBM over the next few years.

means? In a discussion about not trusting IBM's marketing, you can't simply say "I hear you and it will get better, trust me."

What exactly do you expect him to say? Gimme a break.

I don't work in the same role so I only know things from a 10,000 ft view. But a lot of orgs that IBM works with have inane bureaucratic structures with their in house devs pretty much at the bottom of the food chain, or in an "IT department". All decisions on using tech tools are made by "higher ups" a lot of whom have 0 technical background. We're trying to change this culture to one where devs get to make those decisions AFAIK.

Disclosure: IBM dev, but not a Developer Advocate.

Nothing, it's a corporate identity problem. IBM has alot of organizations and many have good people who advocate developer tools and build products. Then you have the dark side with marketing and global services, where promises are never met and costs are beyond. Someone at the top made the decision the choose IBM, right or wrong, technical or not, they chose. I've been on both sides, and yeah 6 years in IBM global services watching the wheels spin. I've had technical decisions yanked from under me because my CTO was convinced by IBM marketing it was wrong after they failed proof of concepts. Companies have too many choices to build tech and as a developer, advocate or whatever you know there is always a better path. Especially without an IBM product.

IBM was always unique, could sell you sfw, hdw, hosting, resources, and finance the whole thing. Some business as leaders never really adapt to change because management is stupid (blockbuster, radio shack, blackberry, etc.). IBM had so many chances to compete with Amazon and they failed. Watson is no more than a deflection in the marketplace that masks the real problems from within.

> What exactly do you expect him to say? Gimme a break.

If you can't provide any useful comments, then nothing. What they're saying contributes absolutely nothing to the conversation. Maybe say you'll stop selling non-existent features to executives for starters? I don't appreciate HN becoming an advertising ground to name-drop IBM without any real talking points, please provide some sort of real discussion.

> We're trying to change this culture to one where devs get to make those decisions AFAIK.

Ok, but again how? Or I guess in other words, I don't believe you.

It's not really up to IBM whether or not this shift towards more developer influence happens. IBM can either embrace this shift and benefit from it, or it can reject it and pretend it's not happening. Personally, I hope that IBM continues to accept and embrace this shift (which I am seeing many indications of). I hope that IBM continues to become more-and-more relevant to developers, but the proof is in the pudding.

"Nobody ever got fired for buying IBM"

And IBM often tried to get people fired for not buying IBM – a former colleague was once the CIO of the only Fortune 100 company that didn't have a corporate IBM mainframe (one or two of their subsidiaries did, but the conglomerate mother ship did not, only Univac). IBM had quite a few golfing conversations with the CEO trying to get him fired. [This was back in the '70s].

In the late 90s, the startup I worked for made a marketing deal with IBM. We were mostly running Sun hardware, I was in charge of IT purchasing. After I made it clear we would not be migrating to IBM hardware, they went after my job.

The tradition was still going at that point.

Maybe, but then again you can't buy IBM if you're forbidden from buying it in the first place.


I'm so glad people are getting enough of IBM bullshit

They get laid off for being making their department a black hole for money, though.

Please, let this quote be ironic.

It's a prevailing attitude in buying non-cheap ("enterprise") systems. If you buy one of the big names and it fails you can always say "they are a market leader, it was the best decision with the information we had available" - if you take a leap of faith for a not so well known name and the project fails it will probably cost you your job. People behave accordingly.

No, it's historic.

No, it's current at many places.

Canada's Federal Government payroll division would probably agree with you.


> The majority of their offerings are overengineered and prone to failure in a production setting.

What do you mean by 'over engineered' in this context? The usual way I've seen that term used would imply 'reliable but more expensive than necessary' rather than 'unreliable'.

In software it sometimes means overcomplicated for the scope that delivers little value but high cost in usage due to the complexity.

Think J2EE and their xml config files.

Or using hadoop and kubernetes when a sqllite db on digital ocean will do

>They are pros at selling to non-technical management, all the while IT employees get stuck dealing with the aftermath.

I have seen middle management say very clearly that no one was ever fired for choosing IBM.

I remember hearing a lot of great things about Cloudant, which happens to be from IBM, on HN and elsewhere.

Is that just a notable exception or are there better alternatives out there these days for hosted CouchDB?

Didn't IBM buy Cloudant? And wasn't it based off of an Apache application?

Not saying it's not good (I've never used it), but IBM can't really get any credit for this. They just happen to have bought a decent product and now they sell it.

Right, my intention by pointing out Cloudant wasn't to offer it as a counter-example (I personally have 0 experience with it), but rather to see if anyone might have some red flags to share, as I'm considering using Cloudant instead of hosting my own CouchDB instances for a new project I'm speccing out, and especially since nobody has mentioned it as a counter-example in this thread so far.

Disclaimer: this is a throwaway account, and I used to work for IBM.

I like Cloudant .. it is not too bad. In a recent project (after I left IBM), I needed to get a data tier up quickly and I actually used it for the initial POC. And yes, it was an acquisition by IBM. I was a bit worried that bluewashing would have made Cloudant crap but looks like they have managed to avoid it. The caveat I'd suggest is look closely at the pricing model. If you want QoS and scale, be careful of the pricing model. For small projects and projects that need fast completion (what project today doesn't?), it is actually very nice. You asked for caveats .. I think some of the language bindings were a bit out of date .. but since it uses REST calls, it wasn't too hard to bypass.

I've been using Cloudant for several years and it's been extremely reliable. Support is good and they have contributed a ton of open source code and docs back to the community. The communication has been transparent throughout the IBM acquisition process so I feel confident in continuing to start new projects on the platform.

It's really easy (largely thanks to Cloudant) to host your own Couch instance but it's also really affordable to let them do it.

I get to use a transmogrified version of Jazz called IBM Control Desk. Literally dog shit.

At least it's not further wrapped in cat shit, I guess?

Now it is wrapped in "SmartCloud" instead. I am not sure if that is an improvement.

Hey Willie I'm feeling some negative customer feedback here.

Thank you for your feedback. I work for IBM and would like to share the details with our product team so that we can improve. What were your issues with Jazz Team Concert and how do you think we can do better next time?

"Watson" made more sense to me when I realized it was just a branding term for any AI-related product, API, or service offered by IBM, which vary widely in functionality. That fact is not that hidden [1], but their marketing doesn't exactly emphasize it. If you view it that way, "Watson" should just be compared against any other similarly broad AI-as-a-service offering with strengths and weaknesses, not treated as some integrated Intelligence that will solve all your problems.

[1] E.g. they're pretty open about renaming existing things to fit under the Watson brand: https://www.ibm.com/blogs/bluemix/2016/10/predictive-analyti...

Anyone with a passing familiarity for Watson does view it that way.

Eh this is more of a recent development. Originally it was the natural language processing and knowledge extraction engine.

That is a hard sell because requires a truckload of configuration and internal expertise to get a working product out of it, and it ever only solves only some very narrow ranges of solutions.

In typical IBM style, since they can't have a multimillion research division going in red as that would get a bunch of management in trouble, they moved more and more product under the watson name to pad up the division growth and sales numbers.

That's the point where it is now after hoarding anything internal with value. You can see more or less the same with their cloud offer.

They may understand it in theory, but I wonder how many middle-managers still have a gut-feeling that they're hiring that "super-smart AI guy who won Jeopardy" to work for them?

It might be more honest if they said "IBM Data AI Division."

"IBM DAD"? :)

More seriously, do you think Microsoft would have been more honest if they'd called Windows "Microsoft Graphical Multitasking Operating System"? Or if BMW had gone with "BMW Better Performance And A Cooler Badge But It'll Cost You" instead of "BMW M"?

Brand names exist for a reason.

Why not just call it "Watson Systems"?

Because IBM markets to CEOs, not IT staff. They want their customers excitedly talking about Jeopardy-winning, superhuman AI at the dinner table. Technical people won't do that.

Stop, I'm getting fucking flashbacks to our early product design meetings.

You have more interesting meetings than I have..

How about: With All The Shit On Nails

Hmm, I'm having a hard time figuring out the N. Ideas?

"With All The Shit One Needs"?

I recently sat through a sales pitch for a piece of IBM software that has an intensely focused (and better) competitor.

IBM's sales pitch was that 1) this particular piece of software was "central to IBM's [market niche] strategy", and 2) if we didn't go with IBM's product, we would miss out on all the interoperability with IBM's Watson that the future is going to bring. The tone veered toward fire-and-brimstone. I wasn't sold, I was put off. They've demoed other things in the past that I've had my socks knocked off by, but they're nothing that's the sort of "indistinguishable from magic" that they're promising in all the marketing hype.

The thing is, IBM delivers tremendous value, and they have some fantastic products. However, stunts like that leave people like me with a very sour taste in my mouth. If they'd just focus on delivering a better product, as quantified by hard metrics, (which they could easily afford to do), they'd keep making out like bandits for a very long time. There's tremendous good-will toward them in the market: bad experience aside, I'm still favorably disposed toward them.

May I ask whom the competitor was/is?

We are closing on the IBM bandwagon, but I want to show that there is alternatives.

It could out me. I'd like to hang on to my job, and we're not allowed to be critical of IBM.

I'm upvoting you doing The Right Thing, but I'd like to downvote the reality instead:/

It's IBM, isn't it?

> "magic beans" is harsh

IBM sells an image, not a working product, and they deserve some ridicule for it.

I have worked with Watson and a decently large IBM consulting team, and 'magic beans' is a diplomatic way of putting it. 'Obfuscatory horse shit' is more accurate.

This certainly works with the IT department and top executives in my company. I have seen Watson mentioned in a ton of presentations and they are all super excited about it. In the last 12 months I have been trying to find out about anything concrete they are doing but so far nothing. It seems all talk.

Lol. You don't understand. It's a fantastic way to seem like you're doing something innovative when you're not really doing anything at all.

The good ones use it as a smoke screen to cover for doing things that need to be done but that the C-level isn't necessarily going to like, the bad ones use it as a cover for incompetence.

The fact that you're more concerned about the truth is why you'll never be on the take like them. :-)

Funny, you say that. Today I got feedback from someone that my bullshitting skills are not up to snuff and that I still have the naive idea that actual results count in the end :-)

Old saying: "Life is hard, even harder if you're stupid."

We were too stupid to catch on that everybody else is lying.

I recommend you read Dilbert more often.

The sad part is, that my experience with the actual "in the trenches" devs, they're much more realistic about what the various things their tech can actually achieve are; and they genuinely are doing some interesting stuff. But that said, I've only been in contact with them for a couple months, and mostly left to my own devices and given cool hardware to play with, so I'm kind of biased heh.

'AI is the New Electricity' and people are moaning about Watson being over-hyped? Electricity was real and delivered immediate benefits. AI does not even exist yet! Imagine I could give you a true AI. yay! Unfortunately,it has the IQ of a 9 year old. Do you want to license that at great expense to make key business decisions for your company? All that Jeopardy and chess has become real double-edged now. People don't seem to get how thick-skulled pointy-haired bosses are. We are probably a century away from that. We can do some great inference from big data sets that will help invent cures for diseases and many things but it isn't a panacea. It isn't 'finicky' some applications will lend themselves more than others. I wouldn't worry too much about the IBM talent pool either. And yes okay disclosure ex-bluesuit.

The only machine working over at Watson is the PR machine.

It's astonishing how much IBM has spent to achieve so little. Per the Jefferies report, IBM spent $15 billion in Cognitive from 2010 to 2015, not including an additional $5 billion in data acquisitions. Even if Cognitive weren't a marketing buzzword invented by IBM to describe if-then statements, that would still be deeply embarrassing. (You could probably double the number of startups funded in Silicon Valley in a year with $20 billion dollars, and I'll bet you'd get much higher returns.)

When you look at the main open-source AI frameworks, none of them were created by IBM, because IBM was incapable of building them. There are individual startups that have done more in AI, using far fewer resources.

And the real shame is that IBM is convincing rubes to buy their cognitive trash, and the rubes are doing it. And when the rubes' bosses realize that the rubes blew the AI budget and the "AI" delivered nothing, those rubes are getting canned. You can get fired for buying IBM. I've seen it.

There's a Madoff-like quality to IBM's Watson sales strategy. The more they fail, the wider they circle to depend on dumber and dumber money from abroad. They're going after rural Scandinavia. They're selling to 3rd-tier cities in China. Just imagining targeting a buyer with the sophistication of the mayor of Mobile, Alabama. That's where IBM Watson is selling strong.

IBM has been poisoning the well in the AI industry for years, teaching customers to be gun-shy and scared of an important technology, just because they blew their money on a dog to begin with. I don't believe an AI winter will come, but if it did, IBM would be the main culprit.

Even if Watson's many mysterious technologies worked, they would still have a lame business model, because most of AI, and the best of AI tooling, is open source. IBM's trying to sell a secret sauce, even as it declines to participate in competitions that would should how Watson stacks up against other tech. Oren Etzioni has been calling them out for years. [1,2,3]

Anyone wondering how IBM came to this sorry pass should read Robert Cringely's "The Decline and Fall of IBM." It's a company run by a clueless management class, who probably believe the smoke they're blowing up everyone else's ass. That's the scary part: The C-suite at IBM actually bought that hype about Watson, until they started to see the numbers, and the numbers sucked. That's why they don't break them out in the quarterly earnings reports. IBM is not signing up the Watson partners it thought it would. [0]

Watson makes claims to solve everything. But when you dig in with the sales people, the only success story they cite is Memorial Sloan Kettering (MSK). It's suspicious, considering how many other organizations they must have worked with. H&R block is bragging about Watson technology, too, but that's just because they think the marketing gimmick will work for them as well as it's worked for IBM.

The real problem in a situation like this is that very few people have an incentive to get the truth out. IBM doesn't want its stock price to dive. And all the customers are mortified that they failed and wasted their money. And they probably had to sign some non-disparagement clause to boot.

[0] http://www.cringely.com/2016/09/07/carrie-underwoods-america...

[1,2,3] https://twitter.com/etzioni/status/654486809825570817



> main open-source AI frameworks

Do you mean UIMA, GATE? What else? These are frameworks comparable with Watson (at least the original thing, not the PR fuzz).

I was actually thinking of the main deep-learning/RL frameworks: TensorFlow, Keras, Theano, Torch, MxNet, Chainer, Caffe. That's where the action is. And if you're talking about AI today, you need to be talking about deep neural networks.

This seems like another PR hype :-)

It didn't bug me until 60 minutes did a full blown 15 minute commercial for IBM on how Watson is saving the world. No I will not leave a link to that absurd piece.

I'm a SWE at Watson Health. What do you think was so absurd about it? It definitely didn't explain the constraints and limitations well enough (it won't diagnose any problem you can have), but for the cancer scenario the segment centered on, the Watson solution is indeed able to greatly assist doctors with a task that does not scale well for them.

I think the 60 Minutes segment was just marketing copy handed to them by IBM (how else could they talk for 20 minutes about cutting-edge ML without mentioning Google or Amazon at all?). Watson is introduced as "an AI" - singular - designed to play Jeopardy and then, after he wins, nobly re-purposed to medicine. Charlie Rose even says at one point that "they taught Watson to diagnose cancer", implying that it was the same Watson who won Jeopardy. The truth is the original Watson is on display in a glass case somewhere and the new "Watson" shares only the name and some of the underlying ML techniques.

It might be easy to understand what Charlie really meant when you are already familiar with the project, but the non-technical viewer will walk away thinking of Watson as a single, coherent entity sitting in an IBM vault somewhere. Conflating ML-based expert systems with anthropomorphic AI is farcical, supporting the accusations that IBM is overselling its product. It is also irresponsible, because it fuels the idea in the popular mind that machine learning is creating independent intelligences rather than mere tools.

Google and Amazon were presumably left out because the segment was in the context of the healthcare field. I don't think DeepMind's healthcare initiatives were very mature at that point.

I agree the language is problematic for non-technical viewers, but at the same time these are difficult concepts to explain to non-technical people, and explain the details of what it can't do is obviously much less compelling than focusing on what it can do.

> these are difficult concepts to explain to non-technical people

But that's not the problem, is it? It's not that IBM is trying to explain something difficult. Rather, if IBM was trying to be honest, the box that won Jeopardy would be called Watson, and the box they're now trying to sell everyone and their dog would be called "Whitney, Watson's big sister" or whatever. At the very least, "Watson the third".

By calling it the exact same name, you are trying to convince us it's the exact same thing just with more training. Which it's not.

Others mentioned poisoning the well; round here the expression is "pissing in the well", which I think fits much better.

I mean, plenty of software gets rewritten and sold under the same name as an old version - I'm not really sure I see the difference there.

IIRC the segment was followed by a self-driving car piece at Carnegie Mellon. If thats true then it was a clearly a general A.I. / M.L. piece. I've seen this episode twice and its also possible that a month ago it was re-aired alongside other content.

Unfortunately its an emotional argument for me as it led to a 30 minute discussion over the phone with my father on the limitations of AI and the current state of marketing from most technology products.

Theirs at least one post in this thread that goes into detail on the limitations of Watson Health (it mentioned oversold capabilities that led to a medical institution wasting ~$60 million dollars, but you would be right to point out that I have no direct examples from memory on why the 60 minutes coverage itself was 'absurd'. If anything its the effectiveness of Watson marketing (and other companies marketing of AI) that gets to me and how it is even becoming pervasive in cultural icons like 60 minutes.

I think it would be useful to link to some scientific evaluation of Watson Health.

I'd want to see:

0. A baseline measurement.

1. Comparison to the nearest compatible.

2. Comparison against a simple algorithm/expert system.

That's exactly what we do with all our pilot clients before we go into production. I'm not sure that IBM publishes these results after the pilot is done though.

Then it's understandable that people would be suspicious of a puff piece on 60minutes which provides no substantive information.

The people who care about the exact numbers and the audience for the 60 minutes piece are very different groups

There's obviously overlap, because people here have shown a dislike to the 60 minutes show.

Overall that probably helps the brand with some groups, and harms it with others (scientists, engineers) so you may have more work to convince them, though the puff pieces might help sell it at the exec level.

After how much supervised learning driven by highly paid experts? Watson would be more credible if papers/people citing it's efficacy would start showing the actual queries and answers of the system without being constantly mediated by IBM oracles sitting between renaponses and their interpretations

Not sure what you're asking. The hospitals usually need to provided a lot of doctor-annotated data for training. But the custom solution is whatever the clients want it to be, not necessarily a question-answering system like Jeopardy that you seem to be asking about.

> beans

I'm reminded of the horrors of early-2000s enterprise Java.

That was always ibm's business model: convince higher ups to buy it, often in detriment of who uses it

> It is so frustrating to have to answer questions from the CEO about why we don't solve all our problems with magic beans from IBM's Watson.

could be worse, could be magic beans from IBM Blockchain.

gives me some solace to know others find themselves in this position. I remember nearly two years ago, i found some Oracle market brochures directed to BI 'solutions'--courtesy of an Oracle sales rep who sat next to CTO on a 12-hour business class flight--on my desk with a stick note from our CTO "please review, then let's chat". As ordered, i diligently read each pretentious little piece of trivia in the pile, then asked him what specific functionality the Oracle stack offered that our open-source stack did not. He said "360 view of the customer", which i had just read about a dozen times.

"Magic Beans" is accurate. Not sure if you've seen an IBM presentation covering Watson at an investor conference, but it's pretty much entirely hand waving and hocus pocus.

I don't know if magic beans was overharsh considering the degree to which IBM portrays Watson as some sort of universal spanner for all problems.

Disclosure: I work in developer advocacy at IBM.

Thank you for sharing your view. If you'd like to talk more about how we can better message our offerings, please feel free to reach out. I'd love to learn more about the kinds of problems you are trying to solve.

Edit: yes, I know, this is a bland response to the parent. But change needs feedback.

Edit 2: thanks matt4077

Here's some feedback: sentences like "getting feedback is essential toward driving change" are the reason nobody likes IBM.

"Change", as used here, is a noun based on the verb "to change". If your sentence calls for a verb, you can just go back to the original "to change".

Something isn't essential "toward" something else. It is essential "for" something else.

"is essential" is unnecessarily passive. Use active verbs.

"getting feedback" is slightly wrong, because what you need is the feedback, not the act of getting it.

"At the same time" is used here because it sounds smarter than "but". Don't do that.

Try this: "Change needs feedback".

Mostly good advice, except "Change needs feedback." No human talks like that. Change, as the subject of the sentence, "needs feedback?" What will happen when we take the feedback and give it to the change? What will the change do with the feedback? And what does the change need the feedback for?

Not all brevity is the soul of wit.

I do agree that something was lost.

How about "for change to happen, we need your feedback"? I think that conveys the message and request accurately and succinctly.

I loved your dissection of classic corporate-speak!

Ok Watson

i like how you made it more succinct by changing it to "Change needs feedback" but your drawn out explanation makes you sound like a person who doesn't know how to provide feedback

I'm going to pile on to everything else you're receiving here:

> I'd love to learn more about the kinds of problems you are trying to solve.

This doesn't sound genuine. This type of delivery almost never sounds genuine. You wouldn't "love" to learn about those problems, and even if you would, virtually no one will believe you because the same corporate speak delivery is used by people who have no commitment to action.

I know you probably just said this unconsciously, but when you use phrases like "I'd love to learn more about the kinds of problems you are trying to solve", it reeks of a comment written and vetted by a team, including brand/social/marketing/PR/whatever the team is in your org that controls employee disclosures.

I've been on the inside of orgs and watched this PR spin sausage cooked up. Someone wants to respond to a controversy brewing on reddit, oh wait, better run it by the the #social channel and make sure everyone agrees this is our best foot forward...wonderful, now we've essentially guaranteed that people don't feel empathized with whatsoever.

Just solicit feedback like a human being. It's okay to "disclose" that you work at IBM (although: jeez, what a corporate-y way to go about it), but speak to people in a conversational tone that doesn't sound like a customer support Twitter account.

The reason I joined IBM in this role is that I do love hearing about customer problems. I was hired because I started programming when I was 8, spent 15 years in engineering and engineering management roles mostly at startups, and because I really like to talk to people about the problems they face.

I'm sure there are companies, and probably parts of IBM, that have to run their communication by legal, but I'm not part of that. It's just how I talk.

I can't guarantee change on any particular issue, and it would ludicrous for me to act as if I speak for the nearly 400,000 IBM employees, so I temper what I say out of a desire to be honest.

My purpose in this thread is to find people who want to share their experience and I assume, perhaps naively, that is part of the purpose of commenting online.

No matter, the discussion in this thread has already been very valuable and I do appreciate everyone's comments, even throwaways.

IBM has a huge problem with misrepresentation. It goes all the way to the top, with a CEO that promises major growth without a clear plan forward. The employees themselves seem to doubt whether they'll have a job in the near future, given the near-annual mass layoffs and the unclear market strategy.

Talking about market strategy, no one knows what IBM does anymore other than garner support contracts and land body-shop jobs. Everyone knows Watson because it's won Jeopardy and a few other games. But that's not because Watson is ground-breaking or anything, but because IBM hasn't done anything else new in 5-10 years. So Watson seems less like a crowning achievement, and more like debris from the Titanic. The company's treading water, and barely, at that.

Sales, however, swears that Watson can do everything from getting the best tax return to curing hair-loss and replacing 80% of the work force. But anyone who's not a fool knows that current AI still has to be tailored towards an industry to be useful. And that's still going to take a huge amount T&M investment. Which you can see from the well-known failed (and expensive) Watson projects.

Usually the only companies who fall for it are the ones where management is less about technical skills and more about kickbacks and favor trading. Or national/international-level contracts where the most important thing required is someone to blame or sue when things inevitably go wrong. Everyone else either understands the technology is more complicated than the presentation, or they have enough insight into IBM to discount the powerpoints.

I mean, IBM still isn't as bad as Oracle. But, if I had to pick a metaphor for IBM's reputation... I'd go with "IBM is as much an innovator as Steven Seagal is an action star."

My recommendation: Find another job.

Dishonesty is ESSENTIAL to IBM Watson marketing. Specifically, IBM has taken a whole lot of different technologies and rebranded them as "Watson", so as to pretend that the Jeopardy-playing system is actually a big practical deal.

IBM's reputation over the decades earns it the benefit of the doubt on bold technical initiatives ... for a while. But when IBM doesn't deliver, the benefit of the doubt runs out. And that's happened now with Watson.

I think before you ask for help from a community like this, you should either offer a reasonable promise that you can deliver better results, or offer to pay people for their time.

IBM's Watson, for me, has been so much marketing fluff that it's in the Oracle class of "only bought by people who don't know better". A profitable business line, but not a problem solving business line.

I can make a reasonable promise that any feedback I get will be put in front of the right people.

For starters, stop airing commercials on TV that show Watson whispering to the capacitors in a jet airliner, or the grapes in a vineyard.

I've provided this feedback to others at IBM with no apparent change, so let's see how this goes with you. First off, I work for a Fortune 200 firm who's executives are intrigued by the capabilities Watson can provide for our business. We've had three different Watson engagements for three different business units and three different usage scenarios - none of them came to fruition. The problem is IBM is unable to deliver a Watson project template. Project managers live by the 80/50 rule - when 50% of the project's funds are spent you'd better be 80% complete with your project's features. How do you know where you are with regards to the completion of a Watson project? How do you know you're on the right track? How do you know when to scrap what you have and start over?

IBM sends out a team for each of the engagements to work with the business unit leaders to educate them on the capabilities of Watson and how to train Watson. It was all pie-in-the-sky stuff. Since we're an engineering-oriented company our business executives tend to have a strong engineering background (we tend to promote from within) and the whole thing smelled fishy to them, sounded too good to be true. When they asked us IT guys about it we presented the problems mentioned in the first paragraph. They concurred.

So we've never taken the bait. That's the problem you're having with Watson.

I hate to say it but this comment is a lot of problem. It reads like it was generated from Watson, then approved by HR and Legal with minor edits.

I don't want to say that's your fault. I get it, I have a job too.


Honestly I wonder what IBM does these days.

Watson is a professional service Machine Learning thing, which can't be profitable. Onsite engineers enguaging one customer rarely is.

Power8/9 has flopped. Nvidia only appears to be on board to kick sand in Intel's eyes.

Then there's... Mainframes?

I know IBM has a cloud now, but honestly it's prices are far high then AWS or Azure.

This reply all but confirms the parent's comment.

Not sure what part of IBM you're in, but I can speak as someone in an i Series shop.

The problem isn't the message. Everyone already drank the kool-aid. A lot of the problem is just the shoddy product, but really it boils down to the culture.

Put out comprehensive documentation _on the indexable Internet_ and not in PDFs, because for christsake if I have to open another Red Book PDF I think I might kill myself.

Make things that are open source -- and I don't mean Open Source™. Let me learn the platform. Let me hack away at an i Series or whatever on my own time.

Everything is so buried behind licensing and paywalls and PDFs and just general bullshit that I can _never_ invest in _anything_ IBM on a personal level. The vendor lock-in is so heavy that I resist even wasting time learning IBM products because in the back of my mind I'm always thinking "I'll never be able to take this with me to a new job."

The garden walls are too high.

nulagrithom, I sent your feedback to the product team that owns that documentation. Their response is below. Could you please let me know more details about the issue you are running into?


> Put out comprehensive documentation _on the indexable Internet_ and not in PDFs

But IBM does do exactly this--and more than a lot of companies!

For example, IBM Knowledge Center [1] houses the documentation for over 3000 IBM products and services, and offers up more than 13M english pages--without a sign-in even, and we translate our official content into over 30 languages.

IBM KC is very well indexed by Google, too (we take pains to make that happen).

We also offer custom search and links to more freely available IBM documentation. Our Cloud development docs--Bluemix docs, are also quite public, though some, it's true, are behind a cloud product sign-in.

Redbooks, too, are available in HTML, not just PDF. They can be found on the web and are indexed by Google too, as are articles from developerWorks and Technotes from Support.

So I really don't understand the complaint about our documentation. Almost none of it is behind a sign-in, let alone a for pay firewall, and almost all of it is freely available to anyone, not just our customers.

Hope this helps!


[1] ibm.com/support/knowledgecenter

Thanks for the comment. I'm not very familiar with the Power team, but I'll find the right folks and forward your note.

Unless you are a master of corporate-fu: You're going to be so popular! :)

My life is too important to me to sacrifice my purpose for the sake of popularity.

>i Series

i has always been a garden wall and I don't think it's realistic to expect that platform to open up at this stage. That's why Linux on Power exists - tell your boss he/she would have better luck retaining/attracting people if they'd spin up some Linux lpars.

The older IT generations like i just the way it is.

Surely a parody

How about you save the money for the developer advocacy division and invest it into producing software that people do not hate? Literally every piece of IBM software I had to use was utter crap.

Better messaging? Clearly you guys are not getting it.

What makes you think I am trying to solve?

In the jobs I've held, getting asked by the CEO to look into a technology means there is some underlying problem or opportunity.

Sorry abtinf.

My comment was literally the response from the Eliza chatbot when fed your comment. It was an admittedly oblique attempt at humour. My apologies.

Agreed. But the community of problem solvers does not frequently overlap with the community of people willing to listen to salesmen.

Yes, there are opportunities. No, I do not believe IBM could solve mine, because of the overpromise, underdeliver. I would only speak with a sales team, or a PM, if my boss ordered me to.

I'm not in sales, nor is my performance measured on any sales-related metric.

I have zero dog in this fight from a professional standpoint, but you should reflect on why your behavior is frustrating so many people here. A bunch of smart, technical people are venting and saying "IBM says a bunch of shit and doesn't deliver." You then come in and say, "I'm from IBM, tell me your concerns and they will be put in front of the right eyes. Change needs feedback." It completely lacks concreteness, humanity, or even basic detail. It's the same sort of pseudo-sales-y cantrip that most of the people here are complaining about in the first place.

You know who got this right? Fucking Dominos, of all companies. They took a hard look at themselves and said, yeah, we've been making a shit product. We don't want to do that any more. So here's what we're doing.

If you want to get a community like this to respond, you need to put the onus on yourself first. Don't go "hey, explain to us what's wrong." You need to figure that out on your own! At least pitch some possibilities. Show some evidence that you've put in the work before crowdsourcing your five year plan. Otherwise you just look lazy and like you want the community to fix your business. And that isn't their job.

From my perspective (not having a dog in this fight either), it's exactly the other way around. People are hurling undifferentiated abuse at IBM. IBM employee shows up, offering to engage. People are hurling personal abuse at them, refuse to offer specific criticism of the product.

Not exactly a display of maturity from this community.

> You need to figure that out on your own

Even IF an employee has a pretty clear idea themselves what needs to be improved, having a citable CUSTOMER opinion to that effect is going to vastly improve the chances of management listening.

> Jefferies pulls from an audit of a partnership between IBM Watson and MD Anderson as a case study for IBM’s broader problems scaling Watson. MD Anderson cut its ties with IBM after wasting $60 million on a Watson project that was ultimately deemed, “not ready for human investigational or clinical use.”

Well, can't say I'm surprised. I used to work on that project a few years ago, basically the idea was that Watson would look at a patient's medical record, figure out what medications they're on, what symptoms they had, etc. and cross-reference all that with the medical knowledge it had ingested from vast amounts of medical literature. In theory, Watson could figure out what medications the patient should or should not be using, a proper course of treatment, etc.

There were two major problems:

First, it turns out your medical record is mostly written in narrative form, i.e., "John Smith is a 45 year old male...", "Patient is taking X mg of Y twice daily", "Patient was administered X ml of Y on 3/1/2016", etc. In other words, there's basically no structured data, so just figuring out the patient's stats, vitals, medications, and treatment dosages was an adventure in NLP. All that stuff was written in sentence form, and of course how things were written depended on who wrote it in the first place. It was really, really hard to make sure Watson actually had correct information about the patient in the first place.

Second, all that medical literature that was being ingested? Regular old, don't-know-anything-about-medicine programmers were the ones writing the rules the manipulating the data extracted via NLP. Well guess what, if you're not a domain expert you're bound to get things wrong.

Put those two things together and we would frequently get recommendations that were wildly incorrect, but that's to be expected when you get garbage input being fed into algorithms written by people who aren't domain experts.

As I understand it IBM now pulls in a team of domain experts to help tune their algs and even markets this as an advantage over their competitors.

That makes the MD Anderson case more interesting as it mostly means that MD Anderson spent $60 million to help IBM figure out the importance of that.

It begins to seem obvious how IBM stays so impervious despite being so sluggish. If your marketing and sales teams can subsidize the engineering and design side so heavily by wooing the executives that actually sign the contracts, you can spin it as just "where the technology was at that time".

Selling on hype gets you in the door and establishes you as player, and eventually the tech will catch up. Being 1-2 years behind blurs when you're signing 5 year enterprise-scale deals that took a year to organize and negotiate.

And if your products really are a dead end, you've got enough of a war chest to just aquihire an entire branch of your company, like IBM did buying Softlayer to just become IBM Cloud Services.

The acquihire fallback plan might work with other areas, but the valuation of a top AI team/startup is now in the high tens or even hundreds of millions. Almost all single contracts would not be sufficient to pay for that.

Moreover, the acquirer must be willing to pay salaries competitive with Google, Facebook, and the likes to keep the team. From what I heard, IBM is not willing to do that. If anyone has information to the contrary, please feel free to correct me.

This basically sums up my brief stint in healthcare.

Every hospital / doctor wants to talk about the wealth of medical information they are sitting on... as you said most notes are just transcribed ad lib, or input into rigid EMR systems, there is no middle ground.

Also the language spoken by healthcare software is HL7 which is another nightmare in itself.

Please tell me of this HL7.

Its the protocol in which health applications talk to each other. It is extremely complicated to say the least.

So if I understand correctly, there was dirty/malformed data which was difficult to interpret, and when sent to a ML algorithm not tuned by a domain expert led to bad results.

This applies to all ML work, why is Watson exempt from it?

Conversely if this the canonical failure case, why's everyone so harsh on Watson?

Conversely if this the canonical failure case, why's everyone so harsh on Watson?

Because they keep claiming Watson can handle this kind of thing, they charged $60M for it, and failed.

That was $60M which could be spent on actual, real cancer research, or models which actually work.

For example, MSKCC[1] currently has a Kaggle competition[2] to do a roughly analogous task. The prize for that is $15,000, and they'll get something more useful for that than MD Anderson did for $60,000,000. Even taking into account it is probably costing MSKCC double the prize to have Kaggle run the competition it seems like IBM was ripping everyone off.

[1] http://mskcc.org

[2] https://www.kaggle.com/c/msk-redefining-cancer-treatment

Perhaps because they charged $60 million for it?

What I sincerely don't get about IT is why customers sign contracts where they fork over the mulah even when nothing of value is delivered to them. Why the actual fuck?

I mean, if Delta ordered a super-advanced new fancy 787 as a tech demo, and Boeing came back with "sorry, our experts were unable to make anything work, so we're going to take all your money and not give you anything back", Delta would rightly tell Boeing to fuck right off and demand that they refund the money.

If you buy a product, like Windows 10, you pay X and get the product, with whatever it contains. You can return that and get your money back.

If you buy a service, or a customized product, then you pay for the time and effort it takes to build / customize that product. Returning that time and effort would be difficult :)

It is more complicated than "pay X, based upon accomplishing Y". Most IT shops insist upon not just taking on the sustaining operations after the initial rollout, but doing it their way as well. When using and administering the products are nearly as complex as using and administering operating systems, I strongly suspect what is happening is our ecosystems have gotten much more complex than in the past, but our way of delivering them has not kept up. This is why AWS is so popular; those packaged services "only comes in black" forcibly eliminate much of the administration and operations idiosyncrasies many IT organizations insist upon for on-prem deployments, and accommodating those idiosyncrasies comes with costs, one of which a huge blurring of where the vendor leaves off and the IT staff takes over.

A lot of IT products don't stand alone in their own stack, and the wild variance in integration points from one site to the next drives a lot of IT project complexities and costs. It doesn't help that many of these integration points are considered essential by various stakeholders within an IT and business organization, yet are not managed by the organization to a necessary quality level. I've worked with failures that trace back to an LDAP cluster sometimes not synchronizing one of its nodes with a password update, for example. Even after tracing through the entire product stack to an outside service we depend upon that the organization delivers (LDAP in this case), the IT staff insisted it was the product. Client rages about a shitty product, then when I figured out the root cause, it was "oops, never mind", no apologies. The directory services team never fixed the sync issue, and instead I ended up cobbling together my own monitoring and notification to detect the issue and notify the administration team.

The administrator they assigned to take on the product after I rolled it out did not have the skills to diagnose and work around that issue, especially as fast as I worked it (within a few days). Unfortunately, this is a common situation. "Knowledge Transfer" is then leaned upon to fill the gap, which of course will fall short of expectations. It is a separate discussion on "knowledge transfer" commonly being a euphemism to for "give me the TL;DR within 100 pages so I don't have to read the stack of manuals and support notes of this product, but still be good enough to quickly troubleshoot if the product falls down". A lot of this "integration point failures that appear on the surface as product failures" comes down to very sysadmin'y troubleshooting, crossing lots of different specialized IT domains, often working on the fly side-by-side with specialized admins and administrator guides of products you've never seen before. Very hard to find people who are that broad, deep, flexible, and cool under pressure.

What I've seen happen a lot is the product vendor successfully rolls out their product, the organization's staff take over, and the stability and usefulness gradually degrades over time. This makes it particularly difficult to say the product never delivered. Sometimes it happens quickly (within a year), and other times it happens slowly (over a decade). It was a combination of human factors that caused the goal to fall short. A lot traces back to organizations wanting to cut costs, and the quickest way to do that when adopting a product is say, "we'll have our own staff do the day-to-day work, instead of paying an outside organization to come in and do it, or hiring new staff who are already experienced with it".

It's a hot mess, and my hope is eventually the cloud delivery model pushes deeper interoperability across the industry's various ecosystems, and increasingly automate more integration efforts not just in the cloud but in other delivery modes. Or at the very least cloud providers commoditize and stabilize increasing chunks of infrastructure and devops that products rely upon.

> why's everyone so harsh on Watson?

well we're talking a _cognitive_ platform that requires 60M$ of _manual_ tuning before it's useful, am I the only one that finds that weird?

> 60M$ of _manual_ tuning before it's NOT useful

Pedantic but important - the final product wasn't usable even after spending an obscene amount of money customizing it to the task

> Conversely if this the canonical failure case, why's everyone so harsh on Watson?

Because what is being promoted under the “Watson” brand is (or at least includes) examples of this failure case.

> This applies to all ML work, why is Watson exempt from it?

I'm not following, Watson was most assuredly not exempt from it.

> Conversely if this the canonical failure case, why's everyone so harsh on Watson?

Because they were attempting to use Watson to help make potentially life-and-death decisions?

>> This applies to all ML work, why is Watson exempt from it?

>I'm not following, Watson was most assuredly not exempt from it.

Apologies, I meant it as not having clean data with badly tuned models should be obvious root cause of failure to the analysts who are familiar with this industry. As in it may take more work over time to improve the situation, and that this problem occurs commonly in other areas of AI/ML as well.

>> Conversely if this the canonical failure case, why's everyone so harsh on Watson?

>Because they were attempting to use Watson to help make potentially life-and-death decisions?

Touche. I have no arguments here.

When you charge $60M to provide a solution, saying "Hey, it failed because we're not domain experts" is not a suitable response.

I was at the CogX artificial intelligence summit in London a couple of weeks ago, and IBM were there in full force.

I made several rounds around all of the stalls, and sat at the bar for a couple of hours with friends, and the whole time I could see the IBM stall, with 4-5 people there, WATSON plastered everywhere and nobody talking to them.

So I went over. I got talking to one of their technical people there,

I am highly experienced in Deep Learning so I started talking about Neural Nets, and he went blank, and admitted he didn't know much about that. I inquired about WATSON's technology and he couldn't answer telling me he didn't know.

I asked about the main use cases, and what makes WATSONs offering better than Deep Learning, he couldn't answer, or even compare on basic levels.

I asked him "What are the coolest uses of WATSON you've seen" and he immediatly went into a canned response about WATSON diagnosing cancer (a project I had seen and was familiar with) we spoke a few minutes on that, and I asked what other cool projects WATSON had been used on ... he had nothing, and I mean literally nothing.

very disappointing

To be honest, if my choice is "I am highly experienced in Deep Learning so I started talking about Neural Nets" and "IBM Watson representative", I start wondering whether it's possible for the whole building to just go up in flames.

Kind of reminds me of these frustrating anti-nuclear advocates that set up a table with some posters at my old college one day. They couldn't give me any real answers why I should support solar wind and hydro over nuclear besides nuclear being "old technology". They had no response to the fact that this state isn't ideal for large scale use of any of the green alternatives or that much more deaths directly result from solar panel installation accidents alone than nuclear as a whole. They couldn't tell me specifically what was supposedly dangerous about current waste storage techniques. The worst part was I googled the organization they were there on behalf of and right on their webpage was an explanation of the failure of nuclear plants in the state to stay profitable (I forget why but it seems really odd since you get so much power from them). I guess they cared more about getting more mileage for less effort dishonestly swaying the local types who are automatically turned off by "radiation" and big industrial buildings.

Was that event not geared towards technical people overall or was that stand an exception?

> They had no response to the fact that this state isn't ideal for large scale use of any of the green alternatives or that much more deaths directly result from solar panel installation accidents alone than nuclear as a whole.

Given an uninformed student and a misinformed didact, I admit to having a strong bias toward criticizing the latter.

From the Wikipedia page on the human toll of the Chernobyl disaster[1]:

> The [2005 Chernobyl Forum report] says it is impossible to reliably predict the number of fatal cancers arising from the incident as small differences in assumptions can result in large differences in the estimated health costs.

[1] https://en.wikipedia.org/wiki/Chernobyl_disaster#Human_impac...

Chernobyl was a plant design that was already outdated at the time and known to be dangerous, ran in the Soviet Union. It's not representative of anything modernly relevant.

It sounds like you trolled some kids passionate about a subject to argue for a technology and industry that you don't really understand.

Were you prepared to compare the plight of the solar installers who fall off of roofs and get injured to the construction workers injured on the job during various nuclear plant construction projects?

Did you count the various victims at Chernobyl? (Epidemiologists project at least 4,000 deaths)

Were you ready to justify why states like New York needs to subsidize nuclear plants with $500M in direct savings because they are not economically viable to operate anymore?

In your case, those kids were being nice.

> Were you prepared to compare the plight of the solar installers who fall off of roofs and get injured to the construction workers injured on the job during various nuclear plant construction projects? > Did you count the various victims at Chernobyl? (Epidemiologists project at least 4,000 deaths)

The Banqiao dam killed 170k, and that's just the largest dam failure, there's a heavy tail following it. Wind and solar don't have any single incident so dramatic, but they require so damn much manufacturing, construction, and maintenance to generate so little energy that they don't do particularly well in the comparisons I've seen.


Admittedly this is a surface level understanding, but you speak with the confidence of someone who knows better while providing even less concrete information. I'm hoping you've got deeper knowledge to back your position up and that you're willing to share.

> Were you ready to justify why states like New York needs to subsidize nuclear...

Everyone in the energy industry gets subsidies. Big subsidies. What matters are costs per kwh, and nuclear seems to win there too.


Are these numbers wrong? If they're wrong, and fission is actually much more expensive than it looks, is it because fission is fundamentally hard / risky or because the updated prices factor in risk due to irrational public perception?

I agree that there is plenty of government intervention of energy. But I don't think there are other electrical generating technologies that require direct cash infusions of that scale when in a mature operational state just to remain a going concern.

The costs for fission are high because they must operate at peak capacity to recoup their capital costs. That was fine when electricity generation was 100% regulated and demand curves were steady. Now things are changing faster and we have market pricing for generation. Market forces price peak demand higher than base demand, and smaller, more nimble and cheaper generators are eating the lunch of big 1960s nuclear plants.

The risks and technology issues are also serious and very expensive, but we have pushed out the costs into the future and taxpayers who aren't born yet will be paying those debts. Nuclear advocates always ignore the costs associated with securely storing waste products for a period of time approaching recorded human history. That said, those costs aren't priced into nuclear energy.

If they were so passionate then they should've known more. It was likely just something done as part of a class assignment.

Sure, I don't understand nuclear perfectly - it's their job to convince me of their platform and that involves educating.

Yes, I'd say solar was still more dangerous if it resulted in more deaths per kWh.

The context was the US so I was talking about statistics in the realm of US safety standards. Even worldwide nuclear gets a better score than worldwide everything else: https://en.wikipedia.org/wiki/Energy_accidents

I said in my post that the economics of the operation was a valid point. It was also the only point the students at the table didn't give. My overall point was that it was disappointing how they wouldn't just use the real argument and instead relied on faulty reasoning like "it's old."

Being nice how? If you're putting up a table to convince people of some position, swaying people on the fence is what you signed up for.

> Yes, I'd say solar was still more dangerous if it resulted in more deaths per kWh.

That's a statistic that inherently favors nuclear because:

- It focuses death vs. injury.

- It doesn't normalize for states with tough regulation of falls. Is it reasonable to attribute shoddy labor practices with an energy source? Injuries will be lower in states like New York because employers have 100% liability for falls, and insurance carriers require safety measures.

- It compares installation of solar panels to operational run state for nuclear.

- There are no reliable statistics re: deaths and injuries for construction or operational workers building nuclear plants 40-50 years ago.

An apples to apples comparison that was meaningful would be "Build / Installation Injuries per kWh of Rated Capacity". For operational comparison something like "Injuries per kWh of Generated Capacity".

It's actually a hard comparison that requires some judgement, as operationally a large professionally managed system like a nuclear plant is a low risk, high impact profile for injury and death. Distributed, unmanaged solar generation is likely going to generate more incidents (accidents, transformer fires, etc) but have a lower (and scoped) impact.

Why "trolled"? Those are all reasonable questions to ask. This advocacy has real world consequences.

It's "trolled" because the OP clearly did not actually understand what he was trying to argue for. He was just going for "gotcha" responses.

If someone is trying to spread awareness of global warming, and someone asks this person "but the climate has changed before, why is it different now", and all the climate change awareness advocate can say is "well, uh, climate change is bad, ok" then it still reflects poorly on the advocate for not being able to defend what they're advocating, not on the person who may have been misinformed but was giving the advocate a chance to show them the error of their ways.

And this isn't even accurate, everything I said is a common pro nuclear response. The only part I wasn't informed on at that moment was the budgeting issue.

I've been the "dumb guy at the table" before.

Not for advocacy, but as a representative of an organization whose policy choices were very controversial for a group of stakeholders. (I cannot talk about specifics, which aren't relevant anyway)

It's not smart or productive to get into a public debate with people in that type of forum. You're there to deliver a message, and engaging in ad hoc public debate is unwise for your cause and disrespectful to your hosts. The only way to win is to not play the game; it's better to just end the conversation in the most tactful way possible (which may include playing dumb or remaining silent) or move it to a private forum.

So what do you do if you're called out for obvious BS?

To be fair, the "technical" person at a conference does not mean "engineer". It's like a quarter tier past pure-salesperson. Obviously not an impressive showing overall, but I don't think it should be used as a judge of their technical talent.

I understand that the people sent to a conference may not be highly knowledgeable about some of the companies products. However, if the company isn't going to at least train them for easy questions like why anyone should buy the product featured prominently in the booth's advertisements, then what was the point in renting vendor space?

[1] At a minimum, give them a handbook with the official answers for common questions.

Yea, watson's distinguishing features can't be "magic beans".

I didn't expect a discourse on topology, but Compare your X to Y, and knowing more than 1 use case is basic, basic, sales stuff

Seems like IBM latest sales strategy is I B M and stickers.

A couple of psychologist made an algorithm on punch cards to diagnose stomach cancer in the 1960s more accurately than doctors. Grab a copy of The Undoing Project.

I work in developer advocacy at IBM.

If you would like to follow-up about your experience or to get connected with the right folks to answer you questions, please feel free to reach out to me.

Watson, is that you?

These are parodies, right?

Is it slot filling then to find treatments? I saw the 60minutes episode about how they have it read the literature.

The real problem is that Everybody is afraid to share data with them because they know Watson will become better at the expense of company's business model.

Disclosure: I work in developer advocacy at IBM.

The issue of data privacy and protection is complex. IBM provides explanations and opt-outs in many of the Watson services.

For example, you can set the X-Watson-Learning-Opt-Out header on speech-to-text calls to prevent retention and use of your data for learning. [1]

[1] https://www.ibm.com/watson/developercloud/doc/speech-to-text...

Why not an X-Watson-Retention-Opt-In header?

Are you saying that your model does not improve from learning from customer's data? If the answer is yes, then what prevents you from deploying the model at a customer's competitor?

I agree, IBM would never send anyone who actually knows anything to these events. The developers are locked away for several good (for IBM) reasons...

The marketing has made it very hard to have a real conversation about IBM Watson. There is no such singular thing as "Watson". IBM offers a ML solution for health, for NLP, chatbots, etc. They all have very different capabilities and require different levels of machine learning. The marketing is BS, but most of the tech is real - if you give IBM your data, let them train a model on it, and communicate what you want, you will get an end-to-end custom solution. It's just not the magic IBM sells in its marketing videos.

Disclaimer: SWE at Watson Health

"if you give IBM your data, let them train a model on it, and communicate what you want, you will get an end-to-end custom solution"

If you give a summer intern your data and communicate what you want, they can also give you a custom solution at a fraction of the price. IBM's model might be more advanced, but the real challenge is applying the conclusions, rather than just building a model.

Communicating what you want is the hardest part. Most managers who are excited about AI (including my boss) have no idea what they want to use it for. "Making things more efficient" or "analysing what's going on in the factory" is way too vague.

Getting data is not so hard, but getting good metadata is a challenge. For example, downloading some music in Chinese and tagging the artist/play count/lyrics/etc. When I have all that metadata, I might not need the machine learning part - I can just find the top 1000 songs and process them manually in an afternoon.

Before spending millions on Watson, I think companies would have a higher return on investment by hiring a summer intern to come up with ideas for further research, and gather the data. Then they can apply AI if they really need it.

How do they market it to a tech audience who presumably see through this sort of thing due to a technical understanding of the subject matter?

For me, Watson Health is actually working on a lot of interesting work in the healthcare space that will eventually lead to improvement in care and outcomes. And hospitals are exactly the kind of clients that IBM handles well - complicated data requirements, government regulations, etc. It's a very difficult, and often frustrating, space to get into.

From other responses in this thread, it sounds like they don't, but instead sell it to the executives who make purchasing decisions.

Good to hear from someone working on W.Health. Just yesterday I saw the latest ad. on Watson which is now promising to help Basketball recruiters spot future stars .. which is quite a let down compared to the original promises of Watson.

Anyways, any way for someone outside IBM to read about the health division effort and collaborations ?

https://www.technologyreview.com/s/607965/a-reality-check-fo... was a good recent review.

The only IBM source I know of are the press releases announcing new collaborations: https://www-03.ibm.com/press/us/en/presskit/27297.wss

Thanks, that's a nice and long list. I still wish I could get a more technical view of what you're doing in medicine, are there conferences or symposium ?

I also found https://www.ibm.com/blogs/watson-health/category/case-study/ and https://www.ibm.com/blogs/watson-health/category/white-paper... that show case studies and white papers.

There are also some publications linked at https://www.research.ibm.com/healthcare-and-life-sciences/ and http://researcher.watson.ibm.com/researcher/view_group_subpa...


I'm on the SWE side, while the research side does the conferences. I know that there were health conferences that IBM researchers have presented at but I'm not aware of which ones they were. Hope this helps!

Very much, thanks a ton. Good luck

I don't think this isn't limited to IBM, my partner's PE firm recently hired a small consulting group touting a "revolutionary, AI-driven" real-estate analysis product that has zero AI whatsoever. It's basically a custom spreadsheet tool that they're claiming to be building AI on top of as they consume company data, but for a few hundred grand per year, they have a basic CRUD app on Azure with a reporting tool using D3 visualizations. But they think it's AI.

It's almost like 2016-17 were gold-mine years for marketing buzzwords and some companies are closing deals with no real execution plan for what they're selling.

Later this year I predict they'll relaunch the product as being built on the blockchain.

Problem solved.

"Deep Learning Blockchain"

Also, containerization.

Oh wait, that was 2016, too.

> I don't think this isn't limited to IBM,

I work in quant finance, and indeed there is currently a boom of people claiming to be building AI/ML funds. I mean there's some smart people who actually do have AI/ML degrees, but it's sold just like Watson: magic beans.

Now it's not that it can't be done, it's just not as easy as "hire an AI guy and show him financial data". Same with Watson, it's not physically impossible to build a cancer scanner, it's just very hard and not quite fulfilling lofty expectations yet.

could you share the name of the company offering this real-estate analysis product?

The people with the problem here are the people buying it though. Even if you lie to yourself or are just stupid. If you can sell people that you have AI without actually having it, I'd argue that it's actually a strength.

Watson is one of the biggest empty marketing slogans ever. The marketing makes it almost seems like General AI able to easily solve your pressing problems if you pay IBM money.

As a non-expert, it seems like the top end researchers are working for Google(Hinton, Bengio, etc), Facebook(LeCun), Baidu, Uber (ex CMU faculty). I don't really see a lot of machine learning research coming out of IBM comparable to the others.

IBM seems to running on the fumes of it's previous greatness while burning the ship to generate stock market returns.

> IBM seems to running on the fumes of it's previous greatness while burning the ship to generate stock market returns.

They have great research, but as always with lumbering behemoths they have to earn enough from the tech to keep the behemoth going, and with their bread and butter (hardware and consulting) drying up they're trying to maximise returns and having to oversell their remaining products

I feel like IBM markets Watson towards your non-technical business people through actual business applicable problems where as companies like google market their products towards solving hard technological problems.

As a software developer, I follow the technical solutions more closely so I love google, but actual business that results in revenue/profit points to other companies like IBM (based on other comments in this thread) making the actual money off of these technologies.

Whose strategy will pay off more in the long term is anyone's guess though.

Surprising and a bit disappointing from a company that is famous for reinventing itself.

I quit IBM Watson 5 weeks ago. Here is why IBM is suffering.


PM me for more ;]

Not my circus and not my monkeys, but how long did you work for IBM?

Your blog post is short on actual dates (a time frame)

Best of luck

IBM offered a day of Watson training in San Francisco about a year and a half ago. As engineers working with classifications, we were interested to compare the results of Watson to our algorithms, but also look at the API, the communication, the community etc.

It was a classroom nightmare. WIFI not working, Bluemix required for all workshops not working at that time, teachers very new on the topic themselves (one confessed he only knew Watson for a couple of weeks before the training), no announcement, no nice moment to socialize or build up a community, no coupon given to try on our own after, ...

And... the algorithms didn't work at all. The sentiment analysis was classifying as really positive the sentence: "I wasn't happy at all by the service" due to 'happy' and 'all' present in the sentence.

I've had two encounters with IBM Watson that left me unimpressed. The first was using the IBM Watson Speech Transcription service (give an audio file and get text); the results were pretty bad vs. Google's, for example. The second was in their recent integration into Star Trek Bridge Command (which is an amazing game BTW!); the speech recognition results were pretty bad.

What does IBM even do anymore? Is it some bizarre set of buildings where they just play with computers and print money? What product do they sell? Whom do they sell it to?

I am genuinely curious...

I have a comp sci degree and worked in different industries relating to software and have never even seen or touched any IBM tech except for those old cash registers.

I was working for a UN organisation last year and some of my work overlapped with another project that was being done by IBM. They were hired to make our web applications more secure by adding 2FA and user access control middleware, using their Websphere family of products (IIRC) and building custom integrations.

I didn't get involved much on the project, but from what I understand the developers sent to work with us were useless, the products were bad or at least poorly suited for us, and what they were going to build was a huge waste of money. (However the last point was more to do with deep organisational issues where I was working).

The main product IBM sold was 'mininising risk' which the director of our department loved - as it meant if the project failed he would have someone to take the blame other than himself. The old saying 'nobody got fired for choosing IBM' is still true.

You need to work in crusty old companies that only recently got into doing software. Anything that produces hardware would be a good start.

Within the data science community Watson has long been viewed as snake oil. Glad to see less technical and business folks are finally smelling BS too.

Lotus Notes. Why this garbage of an email system is still perpetrated by IBM explains IBM. It worked back in 1999 but pity you if you're in a company still using it and the CIO still putting upgrade patches to it.

Still using it? We went from Gmail (for business) to Notes because IBM Sales managed to convince the leadership that they could track and file everything better in Notes. Now we have to deal with this shitshow of an app where it takes 10 clicks to open a message because you have to file it under a project first and good luck if you ever need to search for anything.

> and let’s be real, things would look much worse if Google, Microsoft and Facebook were added to this table

Umm, so add them? And Nvidia, Intel, Baidu, Uber, Tesla? Anybody else? That single chart would actually be more interesting than the entirety of this article.

The techcrunch article says that but the Jefferies analyis mentions them in the notes for those charts saying they're compareable to Amazon.

In the last 6 weeks, I have been called by two reporters (Wall Street Journal and Reuters) for background on AI. I talked with the Journal reporter for about an hour, covering 'everything.' However, the Reuters reporter only wanted to talk about IBM Watson - we just had a short talk.

I have seen a lot of negative press on Watson, but really, it can be evaluated like any other API to see if it meets your needs.

the submarine - paul graham

Thanks, interesting read!

While getting my undergraduate IBM said they were going to give an overview of the Watson system they used to solve Jeopardy . I skipped it but there were some professors that went to it. The professors walked out saying that they were using Watson as some kind of marketing term. They gave no technical details either . That's how I found out that Watson was a marketing gimmick.

Disclosure: I work for IBM.

Watson is not a consumer gadget but the AI platform for real business. Watson solutions are being built, used, and deployed in more than 45 countries and across 20 different industries. Take health care alone -- Watson is in clinical use in the US and 5 other countries, and it has been trained on 8 types of cancers, with plans to add 6 more this year. Watson has now been trained and released to help support physicians in their treatment of breast, lung, colorectal, cervical, ovarian, gastric and prostate cancers. By the end of the year, the technology will be available to support at least 12 cancer types, representing 80 percent of the global incidence of cancer. Beyond oncology, Watson is in use by nearly half of the top 25 life sciences companies, major manufacturers for IoT applications, retail and financial services firms, and partners like GM, H&R Block and SalesForce.com.

We have invested billions of dollars in the Watson business unit since its inception in 2014, with more than 15,000 professionals, and more than a third of IBM's research division is devoted to leading-edge AI research. When you consider the vast scope of IBM's work in AI, from Watson Health to Watson Financial Services to the emerging Internet of Things opportunity, it is clear that no other company is doing AI at the scale of IBM.

By the end of this year, Watson will touch one billion people in some way · Watson can “see,” able to describe the contents of an image. For example, Watson can identify melanoma from skin lesion images with 95 percent accuracy, according to research with Memorial Sloan Kettering. · Watson can “hear,” understanding speech including Japanese, Mandarin, Spanish, Portuguese, among others. · Watson can “read” 9 languages. · Watson can “feel” impulses from sensors in elevators, buildings, autos and even ball bearings. · At IBM, there are more than 1,000 researchers focused solely on artificial intelligence

IBM should ask Watson how to fix the company first, then it would have some credibility. They don't treat there employees very well either, but neither does Oracle so why would anybody waste time working for losers.

+1 "dog shit wrapped in cat shit" .. that is awesome.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact