
Facebook's Memory Networks Demo - vierja
https://www.facebook.com/video.php?v=10153098860532200
======
solve
More:

[https://research.facebook.com/researchers/1543934539189348](https://research.facebook.com/researchers/1543934539189348)

[http://arxiv.org/abs/1502.05698](http://arxiv.org/abs/1502.05698)

[http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-1...](http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-1.tar.gz)

So, anyone have open source code that can perform these tasks?

------
Houshalter
Memory is currently (IMHO) the limiting factor of neural networks at general
AI stuff. In order to learn new things, they need to "train" by seeing the new
thing many times and updating their parameters slowly. Analogous to how humans
learn at a low level. You need to practice throwing a ball or riding a bike or
speaking a language, and over time you just learn how to do it naturally.

But you can't ask a neural network a question like "what was the last time you
saw X". Nor are they very good at storing facts. You can tell it "Obama is the
president", but it needs to hear that fact many times before it forms a
connection between "Obama" and "president". Whereas a human would remember
hearing that fact before and recall it, and eventually after recalling it
enough times, would also form that connection at a low level.

This is sort of a step in that direction. They are storing sentences as
memories, and then having the NN iterate through all of them and pick the ones
that are relevant.

~~~
blennon
Indeed, another weakness is that multi-layer perceptrons and their deep
counterparts are susceptible to "catastrophic forgetting" \-- training away
old memories when new training data is presented. This is one reason why it's
necessary to repeatedly run through the training data. This is a result of the
way learning works in these neural networks. Like you say, humans have the
ability to learn associatively in one-shot, a very different learning method.

There are neural networks like Grossberg's Adaptive Resonance Theory (ART)
which do not suffer from this flaw.

~~~
Houshalter
Dropout seems to significantly reduce catastrophic forgetting. So does very
sparse activations, supposedly. It's worth noting most NNs are trained with
SGD which is an entirely online algorithm.

Not that it's a solved problem but its not _unsolvable_. Mostly it's that
researchers don't care much about online learning since its not too difficult
to train offline on stored data.

~~~
blennon
Good point about dropout. Do you know of any quantitative studies that address
this specifically?

~~~
Houshalter
[http://arxiv.org/abs/1312.6211](http://arxiv.org/abs/1312.6211)

------
knicholes
I've only taken one course in AI, but it looks like everyone is assuming this
link's AI is using Artificial Neural Networks. To me it would seem more like
an inference engine with a knowledge base. Can anyone find a link that
describes what is powering this demo?

~~~
nl
It uses a class of RNN they call a MemNN to do something that looks a lot like
inference using phrase embeddings.

It's a pretty revolutionary approach - no one seems to have tried anything
very similar since the 1990's. The related work section[1] is worth reading.

Interestingly, the Jeff Dean RNN slide deck mentions using RNNs for doing QA,
but doesn't expand on what work Google has done in the field. I don't have a
URL to hand for that.

They do experiment with loading the MemNN from a set of statements stored as
triples, which looks a lot more like a traditional Knowledge Base.

[1] 4 in
[http://arxiv.org/pdf/1410.3916v8.pdf](http://arxiv.org/pdf/1410.3916v8.pdf)

------
andycdev
Dunno, might be overlooking lots of complex work, but I'd imagine the more
complex work would be taking a book like Lord of the Rings and creating the
synopsis that they started with! :)

------
jasonlfunk
Not to be too critical... but Frodo isn't in the Shire. He went to the Gray
Havens with Bilbo.

“But I have been too deeply hurt, Sam. I tried to save the Shire, and it has
been saved, but not for me. It must often be so, Sam, when things are in
danger: someone has to give them up, lose them, so that others may keep them.”

------
micjustin33
Now after Google, Facebook want to steal my data with this AI project.. what
is the security infrastructure behind this AI is it safe or not?
[http://www.bestvpnservice.com/blog/facebooks-terrifying-
new-...](http://www.bestvpnservice.com/blog/facebooks-terrifying-new-ai-
project/)

