
IBM's "Watson" finally ready for prime-time Jeopardy - shawnee_
http://www.hpcwire.com/features/Must-See-TV-IBM-Watson-Heads-for-Jeopardy-Showdown-115684499.html
======
acangiano
This is going to be, for lack of better words, epic. On a personal level
however, it's a reminder of how truly wondrous the future is.

When Deep Blue beat Kasparov I was a kid in an Italian high school, trying to
explain to disinterested fellow students why that was a big deal.

Not even in my wildest dreams I would have imagined to be working for the
company that made that and this event possible, in a different country than my
own.

Sorry for the slightly off topic comment, but I just wanted to remind everyone
that the future can be a beautiful and surprising time if you stick around
long enough to witness it.

~~~
baddox
Watson's defeat of top Jeopardy! players, while an impressive feat of natural
language processing, is not as impressive as a defeat of a grand master, for
one reason.

The reason is that when multiple contestants know the correct response in
Jeopardy!, it all comes down to reflexive timing. It's no surprise that a
computer could buzz in faster than a human being, and there's no evidence that
Watson knows _more_ correct responses than the human competitors.

~~~
DonnyV
I don't think you fully understand what is going on here. Watson will Process
the slang in the sentence, figure out how the words are being used, understand
the context of the question and then look through its internal databases to
see if it can piece different pieces of knowledge that it has to come up with
an answer.

THATS FUCKING HUGE!!!

~~~
baddox
What led you to believe that I don't understand what's going on? I definitely
understand and appreciate IBM's accomplishment. I'm not minimizing the
coolness of Watson being able to come up with so many correct responses, I'm
minimizing the importance of its actual Jeopardy! gameplay. I'm not sure you
understand the real game mechanics of Jeopardy! With highly proficient
contestants, it comes down to being able to buzz in as soon as the light turns
on indicating that the clue has been delivered by the host. All three
contestants know the correct responses to most clues, and it really does come
down to the buzzer reflexes, something Watson clearly has a huge advantage in.

Again, I fully understand and appreciate what an advance in natural language
processing and sheer computation Watson represents. Even being able to match a
below-average human Jeopardy! contestant would be impressive. However, the
true game-winning technique in Jeopardy! (when all three contestants are
highly skilled) is buzzer reflexes.

~~~
azakai
I think you are exactly right. Other commenters seem to be misunderstanding
you.

It's clear that this is a huge achievement. However, it is different than a
computer beating the human chess champion. All this will prove is that a
computer is about the same as the best humans, not strictly better than them.

(And, that it is strictly better at timing its buzzer response, which is
completely not impressive for a machine to excel at.)

~~~
PakG1
I'd be curious in knowing the mean response time of Watson. That would shed
some light on the topic.

------
waxpraxis
I've been working on the periphery of this project for about a year now (my
company created the on screen "avatar" that's shown at Watson's podium on
stage) and it's been amazing to watch as Watson has progressed.

Some of the early matches I got data from were downright funny. Lots of
nonsense answers and weird correlations that kinda made sense but made it
obvious Watson didn't really understand the problem space.

In case you're interested the avatar gets realtime data from Watson and
visualizes both Watson's internal state and the game state. The core bit is a
collection of "threads" that swarm around the surface of a sphere. The speed,
variability, color, and length of the threads are all tied to the data we get
from Watson. The colors roughly correspond to confidence, and when the threads
bunch up it has to do with what Watson is "doing" (i.e. if he gets an answer
wrong the threads will go slow and gather to the bottom of the sphere, if one
of the other contestants is answering the threads will gather on that side of
of the sphere, etc).

The designer and my team agreed early on that there would be exactly 42
threads. ;-)

Oh, and when Watson speaks the threads push off the surface of the sphere to
the intensity of the audio. It also makes a subtle glow in the center of the
sphere brighten in an homage to Hal.

------
cryptoz
Humanity is closing in on building machines to pass the Turing Test. Watson
beat both champs in the demo round a few weeks ago and I fully expect him -
yes, him - to beat them in the match coming up. I can't wait to watch this
live. What an epic moment in the process of moving slowly from weak AI ->
strong AI.

Watson can do amazing NLP (and presumably Machine Learning), which is
something that the general public perceives as straight up "AI". NLP has been
lagging far behind expectations for decades, but with Google's new Translate
apps and Watson competing on Jeopardy, it seems like NLP is pretty close to
being fully solved.

Very exciting. Very, very exciting.

Edit:

> Eventually the machine will prevail.

This sends chills down my spine.

~~~
JoeAltmaier
Watson is a simulation, a clever puppet designed to mimic human behavior. We
have no more to fear from it, than from a punch and judy show.

Cleverer and cleverer puppets will not become intelligent machines. A
dedicated effort to build a massive neural cluster simulation might become
intelligent, but it will think such incomprehensible thoughts as to be fairly
useless to us (think intelligent rhododendon; what would you have to talk
about?)

We will know when Watson is dangerous when it feels fear, angst, want. Not
just an algorithm to sort facts and simulate speech.

~~~
ugh
A simulation of what?

It can actually answer Jeopardy questions. As good or better than the best
Jeopardy player on the planet. (We will see.) Another task, previously only
conquered by evolution, now conquered by algorithms written by humans. Watson
doesn’t pretend to be good at Jeopardy, it just is.

~~~
losvedir
> It can actually answer Jeopardy questions.

I think you mean, it can actually question Jeopardy answers.

Sigh... I always hated that about the show.

~~~
ugh
That sounds like a fun diversion!

No, I most definitely mean that it can actually answer Jeopardy questions.
Formulating sentences oddly doesn't turn questions into answers and answers
into questions. You could define "question" and "answer" as sentences with a
certain syntax but that would be an extremely stupid definition.

------
tgflynn
One interesting observation, which I don't intend in any way to diminish the
significance of this achievement, which I think will be seen as one of the
most important milestones in the development of computer science, is that
today there is so much information on the web that even Google with it's non-
NLP algorithms can "almost" answer Jeopardy questions.

There is an archive of past Jeopardy questions here :

<http://www.j-archive.com/>

Try choosing a question and typing the category + the clue verbatim into
Google. I've tried this a few times and in most cases the correct answer was
in the top couple of sites (usually in the summary text on the Google search
page).

Of course there's still the problem of actually extracting the answer from the
page and presenting it in the proper form.

------
ajays
A lot of geeks' girlfriends are going to be disappointed on 14th (aka
Valentine's Day, in case you're wondering ;) ), as they huddle around a TV
like schoolgirls watching a Bieber concert.

------
shawnee_
The crazy thing is that Watson is < 5 years in the making.

On PBS last night - <http://video.pbs.org/video/1786674622>

The "Making Stuff" series on Nova that's been airing the last week or so is
very interesting: Making Suff Smarter / Stronger / Cleaner / Smaller. Watson
was first mentioned during the episode of "Making Stuff Smarter" but I guess
it merits its own segment.

------
spot
> Watson is comprised of 90 Power 750 servers, 16 TB of memory and 4 TB of
> disk storage

4TB disk? something is wrong here.

~~~
whatusername
[https://www.ibm.com/developerworks/mydeveloperworks/blogs/In...](https://www.ibm.com/developerworks/mydeveloperworks/blogs/InsideSystemStorage/entry/watson_what_is_the_smartest_machine_on_earth7?lang=en)

"Several readers of my blog have asked for details on the storage aspects of
Watson. Basically, it is a modified version of IBM Scale-Out NAS [SONAS] that
IBM offers commercially, but running Linux on POWER instead of Linux-x86.
System p expansion drawers of SAS 15K RPM 450GB drives, 12 drives each, are
dual-connected to two storage nodes, for a total of 21.6TB of raw disk
capacity. The storage nodes use IBM's General Parallel File System (GPFS) to
provide clustered NFS access to the rest of the system. Each Power 750 has
minimal internal storage mostly to hold the Linux operating system and
programs.

When Watson is booted up, the 15TB of total RAM are loaded up, and thereafter
the DeepQA processing is all done from memory. According to IBM Research, "The
actual size of the data (analyzed and indexed text, knowledge bases, etc.)
used for candidate answer generation and evidence evaluation is under 1TB."
For performance reasons, various subsets of the data are replicated in RAM on
different functional groups of cluster nodes. The entire system is self-
contained, Watson is NOT going to the internet searching for answers. "

------
kefs
_Eventually the machine will prevail_

What a great ending line. The NOVA special last night was great!

[http://www.pbs.org/wgbh/nova/tech/smartest-machine-on-
earth....](http://www.pbs.org/wgbh/nova/tech/smartest-machine-on-earth.html)

------
bengl3rt
Affordable tablets, intelligent computers, commercial space travel... it's so
great to live in the future.

Now, won't somebody please design a viable successor to the Concorde?

~~~
bioh42_2
First we need a viable airline industry any where in the world.

------
johnohara
engadget reported that Watson "destroyed" its human competition while others
reported that humans were "taken down."

In reality, Ken Jennings was beaten on the final answer.

I'm interested in seeing whether Watson has an "aggressiveness" algorithm that
allows it to respond before the answer is fully spoken. Humans have an
advantage in this regard because it goes right to the heart of intelligence.
If the game boils down to reaction time, Watson will probably win.

~~~
senthil_rajasek
"I'm interested in seeing whether Watson has an "aggressiveness" algorithm
that allows it to respond before the answer is fully spoken.

You cannot buzz in until the question is completely read out.

<http://en.wikipedia.org/wiki/Jeopardy!#cite_note-9>

~~~
CWuestefeld
True. But along the same lines...

I wonder if it's able to say to itself "I think I'm in the right ballpark, so
I'll buzz now" and then take the 3-4 seconds it takes the host to recognize
the buzz and ask for an answer, to finish its processing to get to the answer
it's most confident of.

In other words, it doesn't need to actually have the answer to buzz. It just
needs some confidence that the answer is close at hand.

~~~
Splines
I'd wager that humans do the same thing (e.g., when they have an answer and
it's on the tip of their tongue)

~~~
kingkilr
It wouldn't surprise me, when watching (and playing along) I often start
saying the "What is" or "Who was" before I have the answer.

------
prpon
NYTimes had 'Play against IBM Watson' interactive feature few months back.

Here's the link:
[http://www.nytimes.com/interactive/2010/06/16/magazine/watso...](http://www.nytimes.com/interactive/2010/06/16/magazine/watson-
trivia-game.html?ref=magazine)

~~~
wmf
IIRC that does not actually let you play against Watson.

------
gcheong
I've looked around a bit but couldn't find any information on how the game
state and clues are input to Watson. Are they typed in on the fly? Pre-loaded
and just revealed as clues are selected? Is it parsing speech and reading the
board visually?

~~~
kenjackson
I've heard (from Engadget) that it is given text on the fly. It does not do
speech or visual recognition. But it gets the question the same time the
contestants see them.

------
dstein
Watson's voice isn't menacing enough.

~~~
RodgerTheGreat
I'd settle for a decent Sean Connery impression.

------
abrenzel
I am intrigued by Watson. NLP is definitely something that has failed to live
up to its billing so far. I will be very interested to see what kinds of
questions Watson is good at and which he tends to miss, to see if there are
any patterns there.

Going forward, the other real questions will be: is Watson overfitted to the
problem of solving Jeopardy questions, and how practical is the technology?
The former is a real risk to the general applicability of Watson's
technologies, the latter is a question of who can afford it. The article
mentioned on commodity hardware, Watson takes about 2 hours per clue. They
only achieve reasonable response times by using about 3000 cores. That limits
the potential audience.

Either way, I'm very interested to see what happens next week. I watched the
demo videos on YouTube and it was quite cool.

------
mhb
Maybe they should give it the answer "The specifications for a machine to beat
you at Jeopardy."

------
ero5004
Won't Watson have a significant advantage on the final Jeopardy question where
time is not nearly as much of a factor? If it can just keep pace for most of
the game and then bet it all on that last question, it should be no contest.

~~~
teraflop
I wouldn't bet on it. In the "preview" round, Watson seemed to come up with
its answers more or less instantaneously. (Or at least, before the host
finished reading the question.) My guess is the limiting factor is the quality
of the underlying dataset and cleverness of the algorithms, not CPU time.

~~~
andrewce
Human contestants are also able to see the question typed out (much like it's
displayed on the television), so people are able to do this as well.

I'm usually able to read any given question in about a second. Granted, I used
to practice speed-reading trivia questions for an hour or so per day (one of
the leagues we competed in projected questions via powerpoint), but there's at
least one reason most Jeopardy! contestants are able to buzz in almost
immediately on most questions.

------
leelin
Seems unfair for the humans.

Wouldn't a more fair match be a series of individual 1-on-1 matches with
Watson and Jennings / Rutter?

The current configuration means the two humans will both share the questions
that are naturally difficult for computers, but Watson will dominate all the
questions naturally hard for humans.

Alternatively, to make it fair, we would need a 2nd copy of Watson competing,
and if the two Watson's buzz at the same time, randomly pick one to answer.

------
kenjackson
How much of the Watson code is Java?

~~~
kls
I don't know the percentages but Watson is built on UIAM
<http://uima.apache.org/> which has bits in C++ and bits in Java. From there I
am sure different teams used language bindings to use the language that best
suited their particular needs to accomplish their tasks. Not that I know it to
be true, but I would not be at all surprised to find R, Haskel and even some
lisp in their somewhere.

~~~
kenjackson
Thanks for the link. That looks really impressive. For IBM this is presumably
PR for their products/services. I do hope they disclose more of the
architecture as I'd love to build out a system like this, even on a smaller
scale (takes 6 hours to come back with an answer rather than three seconds).

~~~
kls
My understanding that this is the beginning of their DeepQA product line, my
assumption is that the research and portions of the technology will be used
for business insight and analytical to answer what if type questions.

,Just to be clear I don't work for IBM, and I do not know their intentions for
the project, but I do take projects from IBM and have taken projects related
to Watson, but I do not know their plans to monetize Watson, the former is
just pure speculation on my part.

------
derwiki
I think it would be cool to see IBM's creation versus a creation from Google
versus (a person or another machine), although I doubt another company would
want to make such a risky move. If Watson wins, huge PR win; if Watson loses,
still a pretty big PR win.

------
ZoFreX
> When the software was run on a lone 2.6 GHz CPU, it took around 2 hours to
> process a typical Jeopardy clue -- not a very practical implementation. But
> when they parallelized the algorithms across the 2,880-core Watson, they
> were able to cut the processing time from a couple of hours to between 2 and
> 6 seconds.

That is an impressive amount of parallelism! This is very back of the napkin
(and I realise I'm comparing apples and oranges), but a rough estimate for the
time taken if the problem was parallelised with 100% efficiency would be:

(2 hours) / ((2880 * 3.55) / 2.6) = 1.83098592 seconds

~~~
mcrittenden
What's 3.55?

~~~
ZoFreX
Oops! The cores in Watson are 3.55 Ghz, sorry. I realise that ghz is not
necessarily a measure of speed, this estimate could easily be out by a factor
of 2 or 3.

------
bgurupra
<http://www-943.ibm.com/innovation/us/watson/>

~~~
bgurupra
A classic use of watson that IBM is saying is in the field of medical science
- Imagine if we can feed the entire set of medical books into watson (somebody
estimated that would just take about a week for watson to process and make the
connections) and then you have the watson physician's assistant which can
listen into the symptoms and spit out the first five most probably causes -
that would be so damn amazing!

~~~
nradov
There are already experimental clinical decision support applications which,
given a list of symptoms, can produce the differential diagnosis. But they
generally aren't very useful since in most cases the physician can figure out
the same diagnosis just as quickly. Real medicine isn't like House, MD.

IBM is more likely to apply Watson technology toward analytics and data
mining. There are huge amounts of clinical data locked up in unstructured text
reports. If they can analyze that data in a useful way to draw correlations
between symptoms, patient demographics, medications, treatments, and outcomes
then that could add a lot of value for medical researchers.

------
RoboTeddy
Here's a fascinating overview of how Watson (DeepQA) works:

<http://www.youtube.com/watch?v=3G2H3DZ8rNc>

The video goes into some detail, and looks at how Watson analyzes particular
questions.

It feels like AI is starting to become what people once thought it could be.

------
amichail
There's a problem with statistical significance given that the match consists
only of two games:

[http://www.quora.com/Does-IBM-Watson-Jeopardy-match-
contribu...](http://www.quora.com/Does-IBM-Watson-Jeopardy-match-contribute-
to-ignorance-about-statistics)

~~~
cryptoz
I highly doubt that IBM will dismantle the machine when they're done (yes, I
know that's what they did with Deep Blue, but this is different). I'd bet that
far more than two games will be played with Watson, over the next few years.

Also, as pointed out in that thread, this is no different than a regular
Jeopardy game. One game consists of dozens and dozens of questions. If Watson
answers them _all_ correctly and the humans get zero points, that _is_ indeed
statistically significant.

If the score is close, then repetition in games will be necessary. The problem
with the assumption in that thread is that one Jeopardy game is vastly
different than the next. In reality, you could string together a bunch of
games and call it one game. Or take one game and split it up into 50 "games".
The point is, inside one game, there are enough different questions to
definitely count as statistically significant.

~~~
amichail
That depends on what "better" means. If better means actually winning in
Jeopardy, then two games is not enough. Final Jeopardy plays a significant
role in who wins a game, so it's not just like combining multiple games into
one.

Even on the question level, a disclaimer should be given if Watson doesn't
answer enough questions for statistical significance.

Finally, the issue here is with contributing to the statistical ignorance of
people watching the two game match. Sure, Watson has played many games already
before the match but that's not what's being shown on TV.

------
goodgoblin
I wonder whether Watson 'reads' the question or if the questions are fed to it
directly as parameters, or if it has to 'listen' to the spoken words and start
thinking about its response after that.

------
efsavage
We don't need to know math anymore because of calculators, now we don't need
to know facts anymore because of watson.

I guess from their shift to broadcasting opinions that the "news" outlets saw
this coming...

------
EGreg
When deep blue beat Kasparov, IBM quickly moved to shut down the program.
Let's hope it doesn't happen in this case, if Watson wins.

------
aforty
That article is sort of bullshit. They already taped the Watson Jeopardy
episodes last month, they just haven't aired them.

Watson won.

~~~
seancron
That was a practice round: <http://www.wired.com/epicenter/2011/01/ibm-watson-
jeopardy/>

------
xutopia
When the humans playing against Watson get slapped around it'll be a weird
moment for humanity.

------
olalonde
Did they give him HAL 9000's voice or am I crazy?

------
mkramlich
This is the right way to test AI.

"Alan Turing, meet Alex Trebeck."

