
Deeply Moving: Deep Learning for Sentiment Analysis - ninjin
http://nlp.stanford.edu/sentiment/index.html
======
dave_sullivan
If you're interested in using deep learning for nlp, I highly suggest putting
a pot of coffee on and watching this a couple times:
[http://www.socher.org/index.php/DeepLearningTutorial/DeepLea...](http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial)

The really neat part is moving away from bag of words representations. Bag of
words was always a bit of a hack. But seriously, watch those videos, it will
give you a good idea on how it all works, the op is using similar methods.

Edit: thanks gallamine, I fixed the link

~~~
RogerL
Thank you so much for this, I'm just getting going in deep learning. I've been
viewing and reading Hinton's stuff, and found it tough to get traction (I am
just starting, that is not intended to be a comment on Hinton, but on me).

------
agibsonccc
I know there's deep hate for java around here, but here's another relevant
link. (You could probably shoehorn in scala or clojure as well) This is a
direct tutorial for doing deep learning.

[http://nlp.stanford.edu/courses/NAACL2013/](http://nlp.stanford.edu/courses/NAACL2013/)

Take a look at a recent homework assignment they had to do with Named Entity
Recognition and Deep Learning:
[http://nlp.stanford.edu/~socherr/pa4_ner.pdf](http://nlp.stanford.edu/~socherr/pa4_ner.pdf)

It's good to see neural networks being leveraged as they are. Conditional
Random Fields have been dominant for sequence based classifiers for a while.
We've been needing to push the envelope a bit further.

------
dkersten
This is pretty cool. I love how in some sentences, the actual words or phrases
are all marked positive, yet it is able to accurately mark the entire sentence
gets correctly marked as negative. For example, from the paypal/mailpile
article from the front page:

 _Are the risks larger because we are successful?_

"larger because we are" is marked as positive, "successful" is marked as very
positive, the rest are marked as neutral. But the entire sentence is marked as
negative (which to me it certainly reads negative).

~~~
hnriot
How is being successful negative? If anything it's neutral, being successful
is clearly good, having exposure to larger risk is negative, a net neutral.

~~~
dkersten
Being successful is positive, but in the context of the article (paypal not
releasing their money) the statement is negative.

------
joeblau
Fascinating project, I worked for a data analytics company and we always had a
basic test that we'd try to throw at any Sentiment analysis engine.

// Negative > This is shit.

// Positive > This is the shit.

Most engines can't sort that out. I'm definitely going to take a more in-depth
look at this.

~~~
byoung2
"This is shit" isn't inherently negative. Consider the question "what is the
brown substance in the toilet?" and you can answer a neutral "this is shit".

~~~
Cyranix
Sentiment analysis implies attitudes, which imply context. A robust sentiment
analysis system must either be trained on a targeted corpus (in which
entities/concepts and attitudes are well aligned) or include some way of
establishing a more targeted context after having been trained on a general-
purpose corpus.

That is to say, people generally aren't describing the physical material
occupying a toilet in a corpus of movie reviews.

~~~
byoung2
I was referring to the previous comment, where the two lines about shit were
used as a test for any sentiment analysis engine. Your point about context is
what I was trying to illustrate.

------
Kronopath
FYI, since it's not exactly obvious: in the live demo
([http://nlp.stanford.edu:8080/sentiment/rntnDemo.html](http://nlp.stanford.edu:8080/sentiment/rntnDemo.html)),
you can double-click on the tree image that it generates to see a zoomed-in
version with more information, and correct improperly labeled nodes to help
the algorithm learn. Try it with some Rotten Tomatoes review snippets - from
the ones I tried, it got almost all of them right.

~~~
ninjin
Also, they have a demo that supplies you with a sentence that you can check
and correct in order to teach the model [1]. It was actually easy and nice
enough to keep me there for quite some time giving me a few nods/laughs and
forcing me to think about the sentiment structure of the sentences. The only
request I would have for a feature is a "skip" button, since at times I am not
entirely sure in regards to the sentiment of a sentence (you can still reload
the page though and it will have the same effect).

[1]:
[http://nlp.stanford.edu:8080/sentiment/labeling.html](http://nlp.stanford.edu:8080/sentiment/labeling.html)

------
thomasahle
It's interesting how the decades of research in computational linguistics
still come down to rating a single sentence + or -. And though they are
getting better, the systems are still pretty open to 'obvious' attacks:

> I have seen many good movies, this one is not one of them.

> All movies, except this one, are good.

> I thought this movie was going to be bad, but I was wrong.

which rate positive, positive and negative.

I'm full of hope that the modern neural net and linear algebra type methods
will eventually crack this though!

~~~
agibsonccc
Another hard problem: sarcasm. it's fun to think about how to approach it.

~~~
spenuke
Yeah, so much fun.

~~~
agibsonccc
Clever. hehe That aside, has anyone saw any recent works on detecting sarcasm?
There's not much advancement in this that I've seen.

~~~
brianshaler
This deep learning method is an example from going from bag-of-words to
document level analysis. To detect sarcasm, you'd have to make another leap
from document analysis to personality modeling. Humans can detect sarcasm
because they^H^H^H^Hwe can build a mental model of what is expected to be
said—by a particular author or about a specific topic. If a statement is
sufficiently contradictory to our mental model and we are sufficiently
confident of the predicted sentiment, we know it's sarcasm.

Without knowing who said something or in what context it was said, even humans
would fail to accurately detect sarcasm. Here are some hasty examples that
could vary, depending on the author or situation:

"Michael Bay really outdid himself on that one." "That Kanye West song is so
well-written." "It'd fun to build that PHP."

~~~
agibsonccc
Very good point. I love the way you framed that. I took some very cursory
attempts at doing it at the sentence level based on ngram representations,
obviously this was limited.

You also bring up a very good point about humans being able to barely detect
it. It's hard to infer from text without tone a good portion of the time.

It'd be neat to see what we could do with tone as an input feature (obviously
we couldn't get that most of the time) but maybe if enough tech becomes
available that voice inputs are common, sarcasm can be a relevant enough
problem to solve.

In general, I know it's not really worth it to try to solve (that I could
think of immediately anyways) aside from "just because".

That being said: ambiguity is ambiguous. What can you do aside from
approximate as best you can?

------
tylerneylon
Structured sentiment analysis has been around for a while - meaning models
that take into account the overall document as more than just a bag of words
(reference below to a 2007 Google paper). There are a surprising number of
challenges for those new to the problem, e.g. sarcasm or idioms. It's kind of
fun to work on.

[http://acl.ldc.upenn.edu/P/P07/P07-1055.pdf](http://acl.ldc.upenn.edu/P/P07/P07-1055.pdf)

------
contingencies
Ethical considerations have always kept me from investing too much time in
areas of computational linguistics that seem to be predominantly used for the
purposes of rapidly commercialised and now widespread mass surveillance system
development. While it is an interesting area, in fact it represents a change
to society that I would personally feel happy not to have accelerated. Not
sure if anyone else has an opinion on the ethics, here.

------
rgonzalez
I would like to get an accuraye overall result of the sentence. For example
this "I found the result of this study amusing since it was far from
accurate." was classified as nothing less than neutral but I was trying to see
if it would pick up the sarcasm. Is sentiment analysis not yet there?

------
someben
New Ruby module on Github to import & parse the treebank corpus:
[https://news.ycombinator.com/item?id=6395420](https://news.ycombinator.com/item?id=6395420)

------
bhelx
Does anyone familiar with NLP know how the sentence trees are parsed? Can
something like python's nltk do that? I always assumed that was very difficult
to do well.

~~~
ninjin
Parsing is one of the most competitive areas of research in NLP and yes, you
are correct, doing it well is very difficult. State-of-the-art performance on
newswire data is around 95% if you score local structure.

miket has already replied with a well-established lexical parser. The Stanford
parser is one of the most robust out there, you could also go for the BLLIP
(or Charniak or Charniak and Johnson) re-ranking parser [1] that although a
bit old and fiddly (the GitHub version is very solid though) tends to perform
well.

However, both the Stanford and BLLIP parsers are constituency parsers (now we
are going to go all linguistic here) and there is an alternate paradigm of
dependency grammars [2]. I have heard a couple of times that Google is running
a variant of the MaltParser [3] in-house and it is a very solid piece of work
(the command-line interface requires some patience though, but there is plenty
of documentation). People for example in Information Extraction (IE) tends to
favour dependency grammars, probably since you get relevant modifiers closer
to your point of interest in the tree.

Now, we could also go into richer grammars like Combinatory Categorial Grammar
(CCG) [4] and Head-driven Phrase Structure Grammar (HPSG) [5] but I think this
reply is long enough by now...

Personally I favour dependency parsing since there is more annotated data for
it across plenty of languages. Also, the paradigm can cope with a lot of
linguistic wonkiness that is very much present in language other than English
(free word order and non-projective structures for example)

[1]: [https://github.com/dmcc/bllip-parser](https://github.com/dmcc/bllip-
parser)

[2]:
[http://en.wikipedia.org/wiki/Dependency_grammar](http://en.wikipedia.org/wiki/Dependency_grammar)

[3]: [http://maltparser.org/](http://maltparser.org/)

[4]:
[http://en.wikipedia.org/wiki/Combinatory_categorial_grammar](http://en.wikipedia.org/wiki/Combinatory_categorial_grammar)

[5]: [http://en.wikipedia.org/wiki/Head-
driven_phrase_structure_gr...](http://en.wikipedia.org/wiki/Head-
driven_phrase_structure_grammar)

~~~
bhelx
thank you ninjin

------
ohashi
It doesn't seem to understand valence shifters. It seems to have been trained
on not good, but not bad is counted as negative too.

------
ranjeetbhatia
Sentiment analysis has never been this good. Sentiment analysis has never been
good.

------
jmcintyre
Wasn't there a company that did a similar sentiment analysis on Twitter and
wanted to trade the stock exchanges on that information? I wonder what
happened to that.

~~~
tcwc
You might be thinking of Derwent Capital -
[http://www.bloomberg.com/news/2013-02-04/dcm-capital-puts-
it...](http://www.bloomberg.com/news/2013-02-04/dcm-capital-puts-itself-up-
for-sale-in-online-auction.html)

------
nfoz
"In constrast, our new deep learning model actually builds up a representation
of whole sentences based on the sentence structure. It computes the sentiment
based on how words compose the meaning of longer phrases. This way, the model
is not as easily fooled as previous models."

Wait, _that_ is all that's necessary to invoke the term "deep learning"?? Wow.
Wikipedia seems to agree. I had thought there was something more to this
buzzword than "slightly less shallow heuristic guesswork than the AI you
already know and love".

~~~
dbecker
To possibly clarify, the way sentences are structured is itself learned,
rather than relying on hard-coded rules.

Maybe you still find deep-learning disappointing, but there have been some
successes from having the computer learn multiple levels of internal
structure/representation from data.

~~~
anaphor
The problem is that many sentences (at least in English) exhibit some
structural ambiguity (more than one valid syntax tree). Also you have to take
into account pragmatics and sociolinguistic factors (like variation), and so
on. You can't just feed a formal grammar to a program and expect it to
correctly parse English. We're still a LONG way from being able to correctly
parse all (or even 99%) of possible English sentences.

~~~
sp332
It does not depend on hard-coded rules. It learns various probabilities for
sentence structures. The result is much more flexible than a formal grammar.

~~~
anaphor
True, but that still means you require some decent priors about the text that
someone reading it might have. People generally have an idea of what the
person who wrote it might be like, what sort of things to disregard, and so
on. My point was that learning the "structure" of the language is not enough.
I don't know exactly how this program works though, so it might be decent at
that, I haven't read up on it enough.

~~~
sp332
The program doesn't need to know priors, if you make sure the priors for the
test set are close to the same as the ones for the training set :)

