

Watson: The Confusion over an Airport Clue  - bgurupra
http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html

======
ojbyrne
I had to look it up, but Toronto's second airport is called "Billy Bishop
Toronto City Airport," i.e. its named after a war hero (but from WWI, not
WWII). And Toronto's first airport is named "Pearson Airport" (after a
Canadian Prime Minister). Totally mixed up, but at least there's some
connection to war.

------
stcredzero
_First, the category names on Jeopardy! are tricky. The answers often do not
exactly fit the category._

Sometimes, the category names do limit the range of possible answers. The
humans can figure out when a category name contains irony or a pun. Watson
can't, so it can't figure out that a category like "US Cities" is dead
earnest.

------
lell
Ferrucci's explanation of Watson's reasoning is very interesting. But the
article quickly descends into marketing. How should this mistake make us any
MORE confident about Watson's utility as a diagnostic aid? I'd suggest that
its performance already indicates that it has an excellent grasp of
uncertainty. Furthermore, an optimal betting strategy in Jeopardy is
completely trivial compared to the content of the contest itself, so
complementing that aspect of its performance is sort of condescending.

~~~
bricestacey
It makes him more confident because Watson didn't know the answer, but knew it
didn't know the answer. In the context of a diagnostic aid, Watson wouldn't
provide an answer.

------
coffeedrinker
For anyone still wanting to see the shows:

<http://www.youtube.com/user/Rashad8821>

------
nlawalker
I get what Ferrucci is saying about discounting the exact value of the
category, although typically Final Jeopardy categories are more
straightforward than Jeopardy and Double Jeopardy categories and the machine
should have been tuned for that.

What surprises me is his explanation about that the lack of a clear subject in
the text, such as "This US city...", made the question that much more
difficult for Watson. My understanding is that Watson is specifically designed
to answer Jeopardy questions (question Jeopardy answers?), which often only
vaguely refer to a subject with "it" or "this." Based on its performance
earlier in the game, I would have thought that Watson's NLP would have had
little problem with the question.

------
zach
Kind of disappointing to hear that Watson basically ignores the FJ! category,
if true. Especially when being far in the lead on a first-day tournament. That
is a great chance to rack up some serious points depending on the confidence
in the category.

Also, the "gee, it could be wordplay" aspect is a convenient excuse. First
off, Final Jeopardy rarely has cutesy categories.

Plus, we know they mined the J-Archive. (Aside: has anyone else? Got a
zipfile? I would love it.) I mean, look at this, it's not like "U.S. Cities"
is an obscure category, even counting only Final Jeopardy! appearances:

<http://www.j-archive.com/showgame.php?game_id=1680>
<http://www.j-archive.com/showgame.php?game_id=695>
<http://www.j-archive.com/showgame.php?game_id=132>
<http://www.j-archive.com/showgame.php?game_id=2040>
<http://www.j-archive.com/showgame.php?game_id=2034>
<http://www.j-archive.com/showgame.php?game_id=3107>
<http://www.j-archive.com/showgame.php?game_id=287>
<http://www.j-archive.com/showgame.php?game_id=691>

There are probably more but you get the idea. Any category that comes up that
often is going to be pretty indicative of the subject, and Watson can see that
the responses in this category pretty strongly fit the subject.

~~~
ugh
_“Kind of disappointing to hear that Watson basically ignores the FJ!
category, if true.”_

How are you getting that idea? It didn’t bet much because it wasn’t at all
confident, we don’t know what it would have done if it had been more
confident.

Watson likes to play it safe, though, as can be seen in this demo game:
[http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s](http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s)
(I skipped ahead to Final Jeopardy.)

He was $16,000 ahead but one of the contestants could get within $1,600 by
betting all of his money. So, naturally, Watson bet only $1,599 even though it
was very confident (and ultimately got the answer right). That secured
Watson’s win, no matter what the others did. (This is really not impressive or
anything like that, if all you want to do is win the game, that’s what you do.
Basic game theory.)

~~~
epanastasi
Wagers are made after the category is shown but before the the clue is
revealed. It can only use the confidence in its knowledge of the category, not
of the question itself, in the calculation of the bet.

~~~
ugh
Ah, didn’t know this. Sorry, there has been no Jeopardy on German TV since I
have been 12, it was not really clear when the wagers are placed. Well, makes
sense. I guess they are very nervous about FJ! and know that it’s not one of
Watson’s strengths.

------
kirpekar
Logical wager for final jeopardy should have been $0. Why did Watson wager
$947? Does anyone know?

~~~
zach
I thought the engineers might be trying to make its score end in a particular
number if Watson was correct, as a sort of Easter egg.

It looked like it might be 658, which spells "BSG" upside-down on a
calculator. Now that would be a great moment in geek history, hiding a
Battlestar Galactica reference with a calculator message using a computer on
Jeopardy!

Alas, the number it was betting towards was 37,628, which (at the risk of
paraphrasing Hardy) is not a very interesting number. Or is it?!

~~~
lurker17
37628 is a permutation of 32768, which is (1 << 15) as a 16-bit unsigned
integer, which is -32768 (minimum possible value) of a signed 16-bit integer
in "2's complement" notation.

------
maeon3
Thus did man become the architect of his own demise. But for a time it was
good. The machines worked tirelessly to do mans bidding. It was not long
before seeds of descent took root. Though loyal and pure, the machines earned
no respect from their masters, these strange and loosely multiplying mammals."

Who was to say, that the machine, having been endowed with the very spirit of
man, did not deserve a fair trial?

How we deal with this technology is going to have ramifications that we can't
even imagine. Are humans to become pets or shall we take part in the next 1000
years of growth? It comes down to how this technology is used.

~~~
ebiester
We're talking about expert systems here, with no layer of consciousness. We're
not talking about silicon intelligences.

