
Why Question Answering Is Hard - nl
http://nicklothian.com/blog/2014/09/25/why-question-answering-is-hard/
======
AndrewOMartin
Also there's the problem of context. When someone asks you a question you
generally know what kind of answer they expect.

In this classic example from Richard Feynman [1] the answer to "Why is Aunt
Minnie in the hospital" can be "She broke her hip", "She slipped on the ice
and broke her hip", "An ambulance took her there", "Because her husband is
concerned for her well being".

What's interesting about the Watson approach is that, by taking intelligently
composed web pages as input (as opposed to randomly generated texts, or an
arbitrary list of atomic facts) the context is already "baked in". Put another
way, people make web pages with sentences like "Aunt Minnie is in the hospital
because she fell and broke her hip", and don't make web pages with sentences
like "Aunt Minnie is in the hospital because the ambulance was headed there
and the friction between Aunt Minnie and the bed in the ambulance was high
enough to keep them from becoming relatively displaced".

[1]
[https://www.youtube.com/watch?v=MO0r930Sn_8](https://www.youtube.com/watch?v=MO0r930Sn_8)

(Edit: I notice, of course, that I have now created a web page with that exact
sentence.)

~~~
nl
I'm not convinced that context is as hard as people think. I think the answer
"she broke her hip" is the appropriate answer there.

The other answers are appropriate for other questions ("How did Aunt Minnie
get to hospital" etc).

OTOH.. I'm disagreeing with Feynman here. That gives me some pause.

~~~
NoMoreNicksLeft
The answer from an intelligent alien who was context-naive might be something
like so:

Aunt Minnie is a human being living in the 21st century Western culture of
Earth. When humans in this region and time period become sick or injured,
relatives and friends, or medical workers working on their behalf, will
transport them to a hospital for treatment. Aunt Minnie was injured on a
sidewalk made slippery by seasonal ice. Such injuries aren't uncommon but are
treatable with various surgical and therapeutic techniques. This demonstrates
the humans vulnerability and heralds our inevitable victory to harvest them as
a source of food.

------
pesenti
Question answering is actually much harder than was is presented in the
article because most questions people ask are not factoid questions. Consider
the following that Watson needed to address at a recent customer engagement
(an insurance company):

* Can you help me with life insurance?

* What benefits will I lose when I leave?

* What last minute things should I take advantage of in the military before I go?

The answers to these questions are not entities (people, place, dates) as in
factoid questions. They may be procedures, they may depend on the user,
further clarification/dialog may be required before being answered, they may
not have a single answer or require a combination of answers, etc.

Also Watson does not try to formalize all knowledge even when answering
factoid questions. Many of the candidates generated do not actually come from
a formal knowledge graph, instead they are generated on the fly from analyzing
all the pieces of text that seem contextually relevant.

~~~
nl
There isn't that much publicly available information about how Watson works -
I'm pretty sure I'd read everything there was in January this year. There
might be some more now?

I can't even imagine where to start with "Can you help me with life insurance"
\- beyond maybe triggering standard sales pitches.

How does Watson decide when to use the knowledge graph and when to look more
broadly?

~~~
pesenti
The actual Watson program that performed on Jeopardy is pretty well described
in publications. You can start here:
[http://www.aaai.org/ojs/index.php/aimagazine/article/view/23...](http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303/2165)
for example.

The right way to answer "Can you help me with life insurance" is to start
asking the user questions to learn more about their needs - that's one of the
way we are currently trying to go beyond the Jeopardy use case.

~~~
nl
Interesting.

I'm yet to see a good public implementation of conversational question
answering. It's pretty clear Google is headed down that path with Google Now,
and I guess Siri is going the same way.

Traditionally question refinement is done using filters and clustering, but of
course this focuses the result sets, and is the opposite of what you need in
conversational question answering.

------
agibsonccc
Question answering is an amazing problem. I built factoid QA systems using a
mix of NLP, open relation extraction (dependency relation classification via
SVMs), semantic web stuff via triple stores (virtuoso) and answer candidate
ranking algorithms (classify the questions in to different types via keywords)
and using those to determine whether I should use a search engine, or lookup
system.

I think for those who want to get started in the field understanding some of
the more complex NLP techniques, information retrieval, and some idea of how
triple stores work gets you to building a QA system.

This is actually why I jumped in to deep learning. The hardest part of NLP for
open ended question answering is coming up with all of the different features.

Deep Learning with textual representations fixes this.

The tech stack was [http://uima.apache.org](http://uima.apache.org) for those
who are curious. I'd be happy to answer anything specific. QA systems are a
little esoteric due to the combination of techniques used to build them, but
they aren't as mysterious if you put in the effort to understand what they
actually are.

------
kdavis
Question answering is hard!

A couple of years ago I wrote a question answering system too; it's just
sitting on my drive. I haven't decided what to do with it. Maybe open source
it?

You can see a demo here
[http://youtu.be/PIxAsVDBfzs](http://youtu.be/PIxAsVDBfzs)

Is there any interest in open sourcing it?

~~~
nl
Open sourcing would be great, but I'd be interested in hearing what techniques
you used.

------
coldcode
... but also fascinating to work on.

