
Text Summarization with TensorFlow - runesoerensen
https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html
======
nl
Allegedly (according to HP labs anyway) my 15 year old software[0] was state-
of-the-art at this on the CNN dataset[1].

By my thing was very heuristic based, and couldn't generate new sentences like
this can. I'm pretty impressed - I'd say some of the machine generated
summaries are better than the human ones.

[0]
[http://classifier4j.sourceforge.net/](http://classifier4j.sourceforge.net/)
(yes, Sourceforge! Shows how old it is!!)

[1]
[http://dl.acm.org/citation.cfm?id=2797081](http://dl.acm.org/citation.cfm?id=2797081)

------
fiatjaf
This is way too difficult to get right, even for humans.

I dream of a not-so-smart news summarization engine that will not try to
rewrite the news, but only pickup all the numbers and quotations, then present
them in a table of who-said-what and how-many-what, along with the title.

This would put an end to filler-based journalism.

~~~
hyperbovine
I wish you wouldn't be so dismissive of journalism and journalists. What they
provide is not filler. Controversial though this opinion may be around here,
there is serious value in having an actual carbon-based life form -- one who
has spent years or decades covering whatever beat -- provide context and
insight for the quotes and data. That they have become a dying breed spells
real trouble for our civic life.

~~~
hueving
The journalists you describe are so few and far between that there needs to be
a different term for them. The vast majority of the 'news' out there isn't
anything close to what you described.

~~~
collyw
Agreed in an age where twitter storms are worthy of front page news.

------
trengrj
$6,000 for the Gigaword dataset they used to train...

[https://catalog.ldc.upenn.edu/LDC2012T21](https://catalog.ldc.upenn.edu/LDC2012T21)

~~~
emmab
There's not even a way to buy it as an individual who's not part of an
organization! They don't even state this as a possibility.

------
8ig8
There's SUMMRY ([http://smmry.com](http://smmry.com)). I don't recall it being
very smart -- that is, it doesn't rewrite sentences. It extracts the most
important/relevant ones to basically shorten an article. It's definitely
useful.

~~~
mikebotazzo
Wow, I like this.

As an example, here is the Google article resumed by SUMMRY.

===

Research Blog: Text summarization with TensorFlow Being able to develop
Machine Learning models that can automatically deliver accurate summaries of
longer text can be useful for digesting such large amounts of information in a
compressed form, and is a long-term goal of the Google Brain team.

One approach to summarization is to extract parts of the document that are
deemed interesting by some metric and join them to form a summary.

Above we extract the words bolded in the original text and concatenate them to
form a summary.

It turns out for shorter texts, summarization can be learned end-to-end with a
deep learning technique called sequence-to-sequence learning, similar to what
makes Smart Reply for Inbox possible.

In this case, the model reads the article text and writes a suitable headline.

In those tasks training from scratch with this model architecture does not do
as well as some other techniques we're researching, but it serves as a
baseline.

We hope this release can also serve as a baseline for others in their
summarization research.

~~~
iLoch
FYI, their about section also spells out their algorithm:
[http://smmry.com/about](http://smmry.com/about)

------
vtempest
Here's my approach when I built my text-summary app with TensorFlow's
SyntaxNet. SyntaxNet (Parsey) gives part-of-speech for each word and a parse
tree showing which less-important words/phrases describe higher-level words.
"She sells sea shells, down by the sea shore" => (down by the sea shore) is
tagged by SyntaxNet as lower describing "sells" so it can be removed from the
sentence. Removing adjectives and prepositional phrases gives us simpler
sentences easily. Next, we find key words (central to sentences) for news
article based on n-grams, and then score key sentences in which they appear.
Use MIT ConceptNet for common-sense linking of nouns and most likely relations
between them and similar words based on vectors. Generate article summary from
the grammatically simple sentences.

My question is how well the trained models interpret human meaning in joined
sentences. I discovered that by simplifying sentences you lose the original
meaning when that grammatically-low-importance word is central to the meaning.
"Clinton may be the historically first nominee, who is a woman, from the Dem
or GOP party to win presidency" is way different meaning than that if you
remove the "who is a woman". I am also interested in how it makes sence to
join-up nouns/entities across sentences. This will cause the wrong meaning
unless you are building the human meaning structures like in ConceptNet by
learning from the article itself, as opposed to pretrained models based on
grammar or word vector in Gigaword.

My work for the future, is using tf–idf style approach for deciding the key
words in a sentence, which I would recommend over relative grammar/vectors. In
the example in your blog post ("australian wine exports hit record high in
september") you left out that it's 52.1 million liters; but if the article
went on to mention or relate importance to that number, by comparing it to
past records or giving it the price and so on, you can see this "52.1 million
liters" phrase in this one sentence has a higher score relative to the
collection of all sentences. As opposed to probabilistic word cherry picking
based on prior data, this approach will enable you to extract named entities
and phrases and build sentences from phrases in any sentence that
grammatically refer to it.

~~~
nl
_a parse tree showing which less-important words /phrases describe higher-
level words_

Things lower on a parse tree aren't less important than things higher. It just
represents a dependency relationship.

~~~
vtempest
You're pointing out what's already obvious. You still need some way to find
what's "less important", which is what the topic is all about, like by using
grammar dependency or keywords infrequency.

~~~
nl
I'm not trying to be hostile, I just don't understand what you mean.

 _You still need some way to find what 's "less important", which is what the
topic is all about, like by using grammar dependency or keywords infrequency._

I have some experience in this area[1]. I found keyword frequency worked quite
well.

[1]
[https://news.ycombinator.com/item?id=12356133](https://news.ycombinator.com/item?id=12356133)

------
VierScar
Does this include the state of the network in its already-trained state? It
looks like we need to train it with the $6000 dataset if we want to get good
results like those mentioned. Is it possible for its state to be saved to disk
and restored (so we don't all need a copy of the dataset)?

------
mastazi
The Hainan example in the article is especially impressive, the generated
summary uses completely different expressions compared to the source text, yet
it is spot-on. Of course those are probably cherry-picked results, but still.
As a side note, it would be interesting to see how the algorithm performs with
longer sources.

------
wrp
_We 've observed that due to the nature of news headlines, the model can
generate good headlines from reading just a few sentences from the beginning
of the article._

This illustrates the importance of taking the trouble to understand a domain
before trying to model it. I was taught in journalism class that the first
paragraph of a newspaper article should summarize the story, the next 3-5
paragraphs summarize again, then the rest of the article fill in the details.
Not only do the authors spend time discovering what should have been known
from the outset, they reverse cause and effect. The model can generate good
headlines due to the nature of newpaper writing, not due to the nature of
headlines.

------
ProxCoques
That reminds me. Whatever happened to [http://summly.com](http://summly.com)?

~~~
kornish
Acquired by Yahoo a few years ago.

[https://www.crunchbase.com/organization/summly#/entity](https://www.crunchbase.com/organization/summly#/entity)

~~~
andreasklinger
also: wasn't it just a frontend to a external summarize service (that's what i
recall)

~~~
kornish
I don't believe so, no. They did most of the original NLP through a contracted
team, according to Wikipedia:
[https://en.wikipedia.org/wiki/Nick_D%27Aloisio#Summly](https://en.wikipedia.org/wiki/Nick_D%27Aloisio#Summly).

~~~
andreasklinger
got it! remembered it wrong - thx!

------
greenspot
> we started looking at more difficult datasets where reading the entire
> document is necessary to produce good summaries

Was hoping to get rather some more insights on this.

Because when looking at the examples given, I wonder if we really need machine
learning to summarize single sentences? Just by cutting all adjectives,
replacing words by abbreviations and multiple words by potential category
terms, we should face similar results. Maybe it's just a start or did I miss
anything?

------
paulsutter
Next one could turn this into a comment generator, generating comments in the
style and personality of any HN commenters with an adequate corpus.

~~~
polm23
That would only be accurate if you assume most people read the article.

------
hyperbovine
The table is nice, but I'd like to see examples where it performs poorly as
well.

~~~
peterjliu
Author of post here. I'd say most of the examples generated from the best
model were good. However we chose examples that were not too gruesome, as news
can be :)

We encourage you to try the code and see for yourself.

~~~
probably_wrong
How does the model deal with dangling anaphora[1]? I wrote a summarizer for
Spanish following a recent paper as a side project, and it looks as if I'll
need a month of work to solve the issue.

[1] That is, the problem of selecting a sentence such as "He approved the
motion" and then realising that "he" is now undefined.

~~~
wutbrodo
Wouldn't it suffice to do a coreference pass before extracting sentences?
Obviously you'll compound coref errors with the errors in your main logic, but
that seems somewhat unavoidable.

~~~
mark_l_watson
I am working on this in my kbsportal.com NLP demo. With accurate coreference
substitutions (eg., substituting a previous NP like 'San Francisco' for
'there' in a later sentence, substituting full previously mentioned names for
pronouns, etc.) extractive summarization should provide better results, and my
intuition is that this preprocessing should help abstractive summarization
also.

------
yalogin
May be the wrong context here, but if I am starting out learning Machine
Learning using the Stanford course or some other one, is Tensorflow a good
candidate to look into? Or does it only contain the advanced algorithms?

~~~
nl
TF is great once you start doing neural networks. It does have support for
other things too, but there are better documented frameworks.

Start with Scikit-Learn or R.

Once you start doing neural network stuff, start with Keras on top of TF.

------
gravypod
I remember there was a hackernews comment I read once where the user created
an emacs plugin to do just this. It would generate a single sentence summery
about whatever text was input into it.

Edit: Hey, found it!
[https://github.com/mck-/oneliner](https://github.com/mck-/oneliner)

Probably not as sophisticated but does the business. Nice bit of work done on
this.

------
kharms
The following appears in the README.md

    
    
        # Run the eval. Try to avoid running on the same matchine as training.
    

Why should one avoid that?

[https://github.com/tensorflow/models/tree/master/textsum](https://github.com/tensorflow/models/tree/master/textsum)

------
deepGem
I would love to use this kind of tool and look for gotchas or key hidden
elements in legal documents. It's not exactly summarisation, but very useful.
Something like bubbling up the important elements of the fine print.

------
josephcooney
I seem to recall someone used Microsoft's AutoSummarize feature to reduce and
reduce classical works of literature to a few lines. The results were pretty
hilarious, but I can't find it now.

~~~
GordonS
I think you mean this one: [http://jason-
huff.com/projects/autosummarize/](http://jason-
huff.com/projects/autosummarize/)

~~~
josephcooney
Yep. That's the one. Thanks.

------
Cozumel
I wonder what it'd be like on a novel, say something like 'pride and
prejudice', would it be able to essentially summarise the plot or would it end
up like 'movie plots explained badly'

Either way, this is great research with a ton of real world applications!

~~~
peterjlee
It'll be more remarkable if a machine can read a play script and summarize the
plot.

~~~
duaneb
I doubt most humans can even do this well.

