
Long-Form Question Answering - stablemap
https://ai.facebook.com/blog/longform-qa/
======
olooney
I'm not sure "we scraped ELI5"[1] is really such a substantive advancement of
the state-of-the-art that it deserves such a large write up. The Stanford
Question & Answer Dataset is much more carefully curated.[2]

ROGUE[3] and BLEU[4] are pretty meaningful metrics for translations and for
fairly short answers that that really only be phrased one way. For example,
"What is the biggest mammal?" should be answered "The Blue Whale." There is
little room for ambiguity: the words "Blue" and "Whale" _must_ appear, as must
the bigram "Blue Whale" for the answer to be correct. For a large or complex
answer, the situation is different. Metrics based on word overlap like ROGUE
and BLEU must either incentive memorizing the answer given in the training set
(overfitting) or the inappropriately penalize semantically equivalent answers.
For example, for the question "why is the sky blue?" if the algorithm produces
"The sky is blue because Raleigh scattering off of water droplets
preferentially scatters blue light at right angles. This is also why sunsets
are red." and the answer on file is "light with long wavelengths passes
straight through moist air, while light with short wavelengths tends to be
deflected." Both answers are correct - indeed they are basically the same
answer - yet they share so few words, bigrams, and trigrams that they would
have to marked "wrong."

[1]:
[https://www.reddit.com/r/explainlikeimfive/](https://www.reddit.com/r/explainlikeimfive/)

[2]: [https://rajpurkar.github.io/SQuAD-
explorer/](https://rajpurkar.github.io/SQuAD-explorer/)

[3]:
[https://en.wikipedia.org/wiki/ROUGE_(metric)](https://en.wikipedia.org/wiki/ROUGE_\(metric\))

[4]: [https://en.wikipedia.org/wiki/BLEU](https://en.wikipedia.org/wiki/BLEU)

------
aargh_aargh
Am I weird for immediately jumping to conclusion that the first thing this
will be used for is generating tons of rather low-quality content for "SEO"
purposes?

------
6gvONxR4sf7o
It's interesting to see the path towards increasingly sophisticated question
in --> answer out. The way people do this is with dialogue, but that doesn't
fit as easily into standard supervised learning with an easy-to-collect
dataset.

If you asked me "What's a good restaurant nearby?" I wouldn't answer with a
list of restaurants, I'd say "What kind of food do you feel like?" Seems like
we aren't even working our way in that direction. Maybe the sample complexity
of RL and language modeling needs to come down a ton first.

------
dvtrn
I'm having the dickens of a time understanding who Facebook built this for.
Marketers, right?

~~~
52358
did you bother to read the article? It's in the very first line "To help
advance question answering (QA) and create smarter assistants" \-- aka
messenger bots / AI assistants / etc

~~~
dvtrn
Yes. In fact I did, yet I'm a mere passive user of FB and have never come
across QA messenger bots on the platform before, who uses them? I presumed
marketers and advertisers.

Is this presumption incorrect? I'm trying to understand who a
potential/target/ideal user of this technology would be from Facebook's
perspective-a few sentences about mobile users searching for songs but that's
it. That doesn't seem answered in the article, I'd appreciate that in the
place of snarky assumptions about my reading skills, please and thank you.

~~~
the_watcher
Messenger bots are of interest to marketers, but they don't really seem to use
them too frequently. Customer support is a much more common use case from my
observations.

All that said, even if it were marketers, it seems pretty clear _who_ this is
being built for.

