
Announcing SyntaxNet: The World’s Most Accurate Natural Language Parser - cjdulberger
http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html?m=1
======
xigency
Evidence that this is the most accurate parser is here; the previous approach
mentioned is a March 2016 paper, "Globally Normalized Transition-Based Neural
Networks," [http://arxiv.org/abs/1603.06042](http://arxiv.org/abs/1603.06042)

"On a standard benchmark consisting of randomly drawn English newswire
sentences (the 20 year old Penn Treebank), Parsey McParseface recovers
individual dependencies between words with over 94% accuracy, beating our own
previous state-of-the-art results, which were already better than any previous
approach."

From the original paper, "Our model achieves state-of-the-art accuracy on all
of these tasks, matching or outperforming LSTMs while being significantly
faster. In particular for dependency parsing on the Wall Street Journal we
achieve the best-ever published unlabeled attachment score of 94.41%."

This seems like a narrower standard than described, specifically being better
at parsing the Penn Treebank than the best natural language parser for English
on the Wall Street Journal.

The statistics listed on the project GitHub actually contradict these claims
by showing the original March 2016 implementation has higher accuracy than
Parsey McParseface.

~~~
weinzierl
spaCy is another active open source (MIT) POS-tagger. In a previous discussion
on HN[1] it was well received.

There is a simplified educational 200 lines python version [2] of it. It
claims 96.8% for the WSJ corpus.

What am I missing here?

[1]
[https://news.ycombinator.com/item?id=8942783](https://news.ycombinator.com/item?id=8942783)

[2] [https://spacy.io/blog/part-of-speech-pos-tagger-in-
python](https://spacy.io/blog/part-of-speech-pos-tagger-in-python)

~~~
kamkha
spaCy's 96.8% accuracy is for the task of POS tagging while Google's reported
94% accuracy is for dependency parsing, a significantly harder problem.

~~~
weinzierl
Thanks (also to charlieegan3) this explains the difference nicely.

Actually I found an educational 500 lines python parser from the same author
as well [1]. It claims an accuracy of 92.7 for the WSJ corpus.

[1] [https://spacy.io/blog/parsing-english-in-
python](https://spacy.io/blog/parsing-english-in-python)

------
teraflop
This is really cool, and props to Google for making it publicly available.

The blog post says this can be used as a building block for natural language
understanding applications. Does anyone have examples of how that might work?
Parse trees are cool to look at, but what can I do with them?

For instance, let's say I'm interested in doing text classification. I can
imagine that the parse tree would convey more semantic information than just a
bag of words. Should I be turning the edges and vertices of the tree into a
feature vectors somehow? I can think of a few half-baked ideas off the top of
my head, but I'm sure other people have already spent a lot of time thinking
about this, and I'm wondering if there are any "best practices".

~~~
jlos
This would be very interesting when applied to Biblical Studies. Any serious
academic discussion of biblical texts will involve syntactical breakdown of
the text being discussed. Most of the times the ambiguities are clear, but its
still quite common for a phrase to have several possible syntactical
arrangements that are not immediately clear. These ambiguities are also
challenging becuase the languages are dead (at least as used in the biblical
texts). So the type of ambiguity of "Alice drove down the street in her car"
can lead to some significant scholarly disagreement.

I could see Parsey McParseface helping identifying patterns in literature
contemporaneous to the biblical texts. Certain idiomatic uses of syntax, which
would have been obvious to the original readers, could be identified much more
quickly.

~~~
giancarlostoro
I was going to say... my main interest in this project is precisely for
Biblical studies... I could talk about analyzing the Bible for hours, but
let's just say there's way more depth than many even realize. The Aleph Tav in
relation to the Book of Revelation is one such example, many translations omit
it, but the Aleph Tav Study Bible explores it in depth. There could be many
discoveries made with these kind of projects that are missed by just about
anyone only reading a translation.

There are a ton of Jewish Idioms in the Bible that many don't understand at
all, including "No man knows the day or the hour" which is a traditional
Jewish Wedding Idiom. Lots and lots of things could be explored with enough
data and resources.

~~~
arthurdenture
I'd think that the advantage of machine translation is on corpora that are not
known up front (i.e. user-supplied text) or corpora that are exceptionally
large.

If you have a small (ish), well-known text, I don't think you will get much
insight from machine translation. Certainly there are plenty of uses for
computer text analysis/mining in biblical studies, but I doubt translation is
one of them. And for obscure idioms or hapax legomena, machine translation
definitely can't help you because by definition there are no other sources to
rely on.

~~~
giancarlostoro
Well I wouldn't look for idioms, but it would be interesting to throw in
information such as "Strong's Concordance" into the mix, I've yet to really
think of an application for this library fully, but it would be fun to play
around with it nonetheless. I would be analyzing the Hebrew / Greek / Syriac
scripts, seeking verses omitted, or missing, etc. It would make for
interesting studying if anything.

~~~
skissane
You might be interested in Andrew Bannister's research on computer analysis of
the Quran. He wrote a book on it [1], and there's also this paper which gives
a high-level overview [2].

[1] [http://www.amazon.com/Oral-Formulaic-Study-Quran-Andrew-
Bann...](http://www.amazon.com/Oral-Formulaic-Study-Quran-Andrew-
Bannister/dp/0739183575)

[2]
[http://www.academia.edu/9490706/Retelling_the_Tale_A_Compute...](http://www.academia.edu/9490706/Retelling_the_Tale_A_Computerised_Oral-
Formulaic_Analysis_of_the_Qur_an)

------
fpgaminer
One of the projects I'd love to develop is an automated peer editor for
student essays. My wife is an english teacher and a large percentage of her
time is taken up by grading papers. A large percentage of that time is then
spent marking up grammar and spelling. What I envision is a website that
handles that grammar/spelling bit. More importantly, I'd like it as a tool
that the students use freely prior to submitting their essays to the teacher.
I want them to have immediate feedback on how to improve the grammar in their
essays, so they can iterate and learn. By the time the essays reach the
teacher, the teacher should only have to grade for content, composition,
style, plagiarism, citations, etc. Hopefully this also helps to reduce the
amount of grammar that needs to be taught in-class, freeing time for more
meaningful discussions.

The problem is that while I have knowledge and experience in the computer
vision side of machine learning, I lack experience in NLP. And to the best of
my knowledge NLP as a field has not come as far as vision, to the extent that
such an automated editor would have too many mistakes. To be student facing it
would need to be really accurate. On top of that it wouldn't be dealing with
well formed input. The input by definition is adversarial. So unlike SyntaxNet
which is built to deal with comprehensible sentences, this tool would need to
deal with incomprehensible sentences. According to the link, SyntaxNet only
gets 90% accuracy on random sentences from the web.

That said, I might give SyntaxNet a try. The idea would be to use SyntaxNet to
extract meaning from a broken sentence, and then work backwards from the
meaning to identify how the sentence can be modified to better match that
meaning.

Thank you Google for contributing this tool to the community at large.

~~~
schoen
I think this is still risky if used in a context where the student might think
that the computer is somehow always right. Great English writers often
deliberately use sentence fragments or puns, or use a word with a nonstandard
part-of-speech interpretation (especially using a noun as a verb). They may
also sometimes use a sentence that's difficult for readers to parse and then
explain the ambiguity after the fact.

If a teacher gave students a grammar-checking tool to check their writing,
they might assume that the tool knew better than they did, which is only
sometimes true.

~~~
wmeredith
Those great writers that break all the rules still know them.

"Learn the rules like a pro, so you can break them like an artist."

-Picaso

~~~
erikpukinskis
This is often repeated, but there's no evidence it's true. Many great writers
had no formal training.

"And that quote is almost certainly made up."

\- Willem Shakespere

~~~
DougMerritt
Knowing the rules is not the same thing as having formal training.

With then-innovated art like Cubism, which in some sense broke earlier rules,
the point is that those artists, like Picasso, were _able_ to do
representative art in correct proportions -- they were just going beyond that.

This is a nontrivial issue, because there are always students who think they
can skip learning boring mundane old fashioned art and go straight to
something like cubism, but in 99% of cases that doesn't work well at all
compared with learning "the rules" first.

With writing, many great writers have broken "the rules" with punctuation,
spelling, grammar, etc. But the important thing is that they do so _on_
_purpose_.

Whereas if one doesn't know the rules in the first place, one doesn't have the
choice of whether to follow them or break them.

Such a person will always break those rules they don't know (subconscious
knowledge counts btw) -- but not for aesthetic reasons, only out of ignorance.

~~~
vidarh
My perhaps favourite example of knowing when and how to break the rules is
Franz Schuberts "Erlkönig" [1] because it is so stark.

If you listen to it without paying attention to the text (based on a poem by
Goethe by the same name; both the German text and an English translation is
found at [1]), parts of it sounds like horrible jammering and poor harmonies
and it's easy to write it off as not sounding very nice.

Here's [2] a much clearer rendition (two singers, with much stronger
delineation of the three different characters) than the one linked from
Britannica.

If you _do_ pay attention to the text, it is very clear that the unpleasant
parts are very deliberate:

The singer(s) switches between the role of a father, his sick dying child, and
the Erl-king that occurs in the hallucinations of the child while the father
is riding to bring the child to a doctor.

The big difference between the unpleasant-sounding parts of this song and a
bad composer is the clear intent and delineation - Schubert made things sound
bad intentionally explicitly at the points he wanted to illustrate pain and
fear, rather than because he didn't know how to make things sound pleasant
when he wanted to.

The song clearly proves this by setting the childs jammering and the fathers
fearful attempts to soothe him up against much more pleasant segments where
the Erl-king speaks and tries to seduce the child to come with him.

You only get that clear separation if you know how to evoke each effect
precisely. Arguably a bad particularly composer wouldn't even know how to make
things sound bad the "right way" \- there's a big difference between random
bad sounds and making things evoke a child in pain.

[1]
[http://www.britannica.com/topic/Erlkonig](http://www.britannica.com/topic/Erlkonig)

[2]
[https://soundcloud.com/sean_contretenor_lee/erlkonig](https://soundcloud.com/sean_contretenor_lee/erlkonig)

------
jrgoj
Now for the buffalo test[1]

`echo 'Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo' |
syntaxnet/demo.sh'

    
    
       buffalo NN ROOT
    
       +-- buffalo NN nn
    
       |   +-- Buffalo NNP nn
    
       |   |   +-- Buffalo NNP nn
    
       |   |   +-- buffalo NNP nn
    
       |   +-- buffalo NN nn
    
       +-- Buffalo NNP nn
    
            +-- buffalo NNP nn
    

[1]:
[https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...](https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo)

~~~
nihonde
How does it handle the "translation" from that wiki page?

Bison from Buffalo, which bison from Buffalo bully, themselves bully bison
from Buffalo.

~~~
jrgoj
> Input: Bison from Buffalo , which bison from Buffalo bully , themselves
> bully bison from Buffalo . Parse:
    
    
       Bison NNP ROOT
        +-- from IN prep
        |   +-- Buffalo NNP pobj
        +-- , , punct
        +-- bison NN ccomp
        |   +-- themselves PRP nsubj
        |   +-- bully RB advmod
        |   +-- from IN prep
        |       +-- Buffalo NNP pobj
        +-- . . punct

~~~
nihonde
Interesting.

"bully RB advmod"

I assume this is wrong, and probably triggered by the "ly" ending? Also, odd
that it dropped the second clause and didn't see any trouble with the parsed
results lacking a single verb. I guess it has to be able to handle fragments,
etc.

------
deanclatworthy
It's really nice to have access to these kinds of tools. I am sure some folks
from Google are checking this, so thank you.

Analysis of the structure of a piece of text is the first step to
understanding its meaning. IBM are doing some good work in this area.
[http://www.alchemyapi.com/products/demo/alchemylanguage](http://www.alchemyapi.com/products/demo/alchemylanguage)

Anything in the pipeline for this project to help with classifying sentiment,
emotion etc. from text?

~~~
zappo2938
Yes, we derive syntactic meaning from grammatical structure. It's one thing
getting a machine to understand grammar and another to get a human to
understand. If anyone is interested, Doing Grammar by Max Morenberg is an
excellent source of knowledge about grammar.[0]He approaches grammar very
systematically which is helpful if people want to train machines.

[0] [http://www.amazon.com/Doing-Grammar-Max-
Morenberg/dp/0199947...](http://www.amazon.com/Doing-Grammar-Max-
Morenberg/dp/0199947333)

------
feral
I'd love to hear Chomsky's reaction to this stuff (or someone in his camp on
the Chomsky vs. Norvig debate [0]).

My understanding is that Chomsky was against statistical approaches to AI, as
being scientifically un-useful - eventual dead ends, which would reach a
certain accuracy, and plateau - as opposed to the purer logic/grammar
approaches, which reductionistically/generatively decompose things into
constituent parts, in some _interpretable_ way, which is hence more
scientifically valuable, and composable - easier to build on.

But now we're seeing these very successful blended approaches, where you've
got a grammatical search, which is reductionist, and produces an interpretable
factoring of the sentence - but its guided by a massive (comparatively
uninterpretable) neural net.

It's like AlphaGo - which is still doing search, in a very structured, rule
based, reductionist way - but leveraging the more black-box statistical neural
network to make the search actually efficient, and qualitatively more useful.
Is this an emerging paradigm?

I used to have a lot of sympathy for the Compsky argument, and thought Norvig
et al. [the machine learning community] could be accused of talking up a more
prosaic 'applied ML' agenda into being more scientifically worthwhile than it
actually was.

But I think systems like this are evidence that gradual, incremental,
improvement of working statistical systems, can eventually yield more powerful
reductionist/logical systems overall. I'd love to hear an opposing perspective
from someone in the Chomsky camp, in the context of systems like this. (Which
I am hopefully not strawmanning here.)

[0]Norvig's article:
[http://norvig.com/chomsky.html](http://norvig.com/chomsky.html)

~~~
fighting
Not in either camp, quit the nlp field due to disillusionment that it would
lead to anything useful or meaningful.

Both rule-based and statistical approaches are fundamentally flawed by not
incorporating any real world information. Humans do language going from real
world info, mapping to grammar or rules. Computers are trying to go the other
way and are not going to succeed other than as mere toys.

Even tech progression-wise, both rule and model/nn approaches are really bad
since there is no meaningful sense of iterative progress or getting better
step by step, unlike cpu chips or memory speeds. They are more of a random
search in a vast space, hit or miss, getting lucky or not, which is very bad
no good, as a technology or as a career.

~~~
igravious
How about looking at it this way. We now have multiple implementations that
get 94%+ success _without_ knowing anything about the world. Isn't that
remarkable?

Now to get to 99.4%+ how about we combine techniques such as spaCy or Parsey
McParseFace (love the name Google) with very simple real-world cognitive
models. So for the example given "Alice drove down the street in her car." a
simple cognitive model would _know_ that streets cannot be in cars and so be
able to disambiguate. A cognitive model wouldn't know all the facts about the
world, it would know certain things about streets, certain things about cars
and be able to infer on the fly whether the relationship between streets and
cars matches either the first parse possibility or the second. To me this
seems like the obvious next step. If it's obvious to me it must have been
obvious to someone else so presumably somebody is working on it.

~~~
fighting
The 94% success rate is in made up, limited tests. Real world, they fail
constantly in weird horrific or laughable ways. See any Microsoft AI public
demo ever. It's like the self driving car claim of millions of miles without
accidents, except that humans took over anytime there was a chance of one.

------
mdip
This looks fantastic. I've been fascinated with parsers ever since I got into
programming in my teens (almost always centered around programming language
parsing).

Curious - The parsing work I've done with programming languages was never done
via machine learning, just the usual strict classification rules (which are
used to parse ... code written to a strict specification). I'm guessing source
code could be fed as data to an engine like this as a training model but I'm
not sure what the value would be. Does anyone more experienced/smarter than me
have any insights on something like that?

As a side-point:

Parsy McParseface - Well done. They managed to lob a gag over at NERC (Boaty
McBoatface) and let them know that the world won't end because a product has a
goofy name. Every time Google does things like this they send an unconscious
remind us that they're a company that's 'still just a bunch of people like our
users'. They've always been good at marketing in a way that keeps that
"touchy-feely" sense about them and they've taken a free opportunity to get
attention for this product beyond just the small circle of programmers.

As NERC found out, a lot of people paid attention when the winning name was
Boaty McBoatface (among other, more obnoxous/less tasteful choices). A story
about a new ship isn't going to hit the front page of any general news site
normally and I always felt that NERC missed a prime opportunity to continue
with that publicity and attention. It became a topic talked about by friends
of mine who would otherwise have _never_ paid attention to _anything_ science
related. It would have been comical, should the Boaty's mission turn up a
major discovery, to hear 'serious newscasters' say the name of the ship in
reference to the breakthrough. And it would have been refreshing to see that
organization stick to the original name with a "Well, we tried, you spoke, it
was a mistake to trust the pranksters on the web but we're not going to invoke
the 'we get the final say' clause because that wasn't the spirit of the
campaign. Our bad."

~~~
danieldk
_Curious - The parsing work I 've done with programming languages was never
done via machine learning,_

Artificial languages (such a programming languages) are usually designed to be
unambiguous. In other words, there is a 1:1 mapping from a sentence or
fragment to its abstract representation.

Natural language is ambiguous, so there is usually 1:N mapping from a sentence
to abstract representations. So, at some point you need to decide which of the
N readings is the most likely one.

Older rule-based approaches typically constructed all readings of a sentence
and used a model to estimate which reading is the most plausible. In newer
deterministic, linear-time (transition-based) parsers, such ambiguities (if
any) are resolved immediately during each parsing step.

In the end it's a trade-off between having access to global information during
disambiguation and having a higher complexity. So, naturally, the rule-based
systems have been applying tricks to aggressively prune the search space,
while transition-based parsers are gaining more and more tricks to incorporate
more global information.

------
Someone
For those wondering: the license appears to be Apache 2.0
([https://github.com/tensorflow/models](https://github.com/tensorflow/models))

------
syncro
Dockerized version so you try without installing:

[https://hub.docker.com/r/brianlow/syntaxnet-
docker/](https://hub.docker.com/r/brianlow/syntaxnet-docker/)

------
TeMPOraL
> _Humans do a remarkable job of dealing with ambiguity, almost to the point
> where the problem is unnoticeable; the challenge is for computers to do the
> same. Multiple ambiguities such as these in longer sentences conspire to
> give a combinatorial explosion in the number of possible structures for a
> sentence._

Isn't the core observation about natural language that humans _don 't parse it
at all_? Grammar is a secondary, derived construct that we use to give
language some stability; I doubt anyone reading "Alice drove down the street
in her car" actually parsed the grammatical structure of that sentence, either
explicitly or implicitly.

Anyway, some impressive results here.

~~~
glup
Various syntactic theories (HPSG, GPSG, minimalism, construction grammars)
from linguistics are certainly derived constructs, but most researchers would
agree that they all reflect real abstractions that humans make. I think the
NLP community has good a job of harvesting the substantive aspects (which tend
to be fairly conventionalized upon across theories) without overfitting on
specific cases. "Alice drove down the street in her car" is easy for people to
process, "The horse raced past the barn fell" is not, because it requires a
pretty drastic reinterpretation of the structure when you get to the last
word.

That said, there is some interesting work on "good-enough" language
processing, which suggests that people maintain some fuzziness and don't fully
resolve the structure when they don't need to. [1]

[1]
[http://csjarchive.cogsci.rpi.edu/proceedings/2009/papers/75/...](http://csjarchive.cogsci.rpi.edu/proceedings/2009/papers/75/paper75.pdf)

~~~
danieldk
_but most researchers would agree that they all reflect real abstractions that
humans make_

They reflect a particular language in its well-written form. However, humans
are extremely robust against syntax errors. I am not a linguist, but I think
this speaks in favor of lexicalist approaches: we can be very free in word
order, as long as our brain can match up e.g. verbs with their expected
arguments.

------
ohitsdom
I'm sure it's only a matter of time before someone puts this online in a
format easily played with. Looking forward to that

~~~
xigency
It's already available here -
[https://github.com/tensorflow/models/tree/master/syntaxnet](https://github.com/tensorflow/models/tree/master/syntaxnet)

    
    
        echo 'Bob brought the pizza to Alice.' | syntaxnet/demo.sh
    
        Input: Bob brought the pizza to Alice .
        Parse:
        brought VBD ROOT
         +-- Bob NNP nsubj
         +-- pizza NN dobj
         |   +-- the DT det
         +-- to IN prep
         |   +-- Alice NNP pobj
         +-- . . punct

~~~
ohitsdom
I mean fully online, where I don't have to download and setup tensorflow.

~~~
xigency
Yes, probably a few days, unless you go through the effort to setup a web
server.

For other non-Parsey McParseface dependency parsers and POS taggers that are
web accessible, see [http://corenlp.run/](http://corenlp.run/) and
[http://nlp.stanford.edu:8080/parser/](http://nlp.stanford.edu:8080/parser/).

~~~
astrange
Does Google's have a better sense of humor than the 3 in this thread? They all
fail on:

Time flies like an arrow. Fruit flies like a banana.

~~~
xigency
Really, the mechanism of all these parsers, including SyntaxNet, is the same
in that they use statistical training data to set up a neural network. Here's
a paper on the Stanford CoreNLP parser, which you can compare with Google's
paper:
[http://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf](http://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf)

So, really all of the above parsers are weak in that they only output a single
best parsing, when in reality sentences can have more than one valid
structure, the principal example being the second sentence you've provided. I
don't think Google's model has a better sense of humor than the others, no. I
anticipate that they all have used relatively similar training data.

However, there is probably a trivial way to get the second sentence to parse
as

    
    
          Subject --- Verb --- Object
         Noun       Verb   Article  Noun
          |   \       |     |        |
        Fruit flies  like   a      banana .
    

and that is to provide training data with more occurrences of ...

    
    
      > N{Fruit flies} V{like} honey. 
      > N{Fruit flies} V{like} sugar water.
    

than occurrences of

    
    
      > A plane V{flies} PREP{like} a bird.
    

The more sentences using simile that the parser finds, the less likely the
neural net is to consider 'like' as a verb. It's also impacted by all of the
uses of [flies like].

That's the nature of statistical language tools.

The stock parser debuted here gives the same answer as CoreNLP, by the way.

    
    
        flies VBZ ROOT
         +-- Fruit NNP nsubj
         +-- like IN prep
         |   +-- banana NN pobj
         |       +-- a DT det
         +-- . . punct
    

So much for Parsey McParseface's sense of humor.

------
rspeer
I'm glad they point out that we need to move on from Penn Treebank when
measuring the performance of NLP tools. Most communication doesn't sound like
the Penn Treebank, and the decisions that annotators made when labeling Penn
Treebank shouldn't constrain us forever.

Too many people mistake "we can't make taggers that are better at tagging Penn
Treebank" for "we can't make taggers better", when there are so many ways that
taggers could be improved in the real world. I look forward to experimenting
with Parsey McParseface.

------
weinzierl
Say, I wanted to use this for English text with a large amount of jargon. Do
have to train my own model from scratch or is it possible to retrain Parsey
McParseface?

How expensive is it to train a model like Parsey McParseface?

~~~
wodenokoto
Training the model is basically free. Obtaining annotated text for your domain
can be quite expensive if you hire a trained linguist.

~~~
escap
>Training the model is basically free if you have a few GPUs and don't pay
electricity

~~~
YeGoblynQueenne
More like a thousand or so CPUs and a few hundred GPS.

------
scarface74
I started working on a parser as a side project that could parse simple
sentences, create a knowledge graph, and then you could ask questions based on
the graph. I used [http://m.newsinlevels.com](http://m.newsinlevels.com) at
level 1 to feed it news articles and then you could ask questions.

It worked pretty well but I lost interest once I realized I would have to feed
it tons of words. So could I use this to do something similar?

What programming language would I need to use?

~~~
zodiac
Is your work available somewhere? I would love to play with something like it.

~~~
scarface74
It was a hack in C# with no unit tests it never got past the conceptual stage.

------
jventura
As someone who has published work in the NLP area, I always take claimed
results with a grain of salt. With that said, I still will have to read the
paper to know the implementation details, although my problem with generic
linguistic approaches such as this one seems to be is that it is usually hard
to "port" to other languages.

For instance, the way they parse sequences of words may or may not be too
specific to the English language. It is somewhat similar to what we call
"overfitting" in the data-mining area, and it may invalidate this technique
for other languages.

When I worked on this area (up to 2014), I worked mainly in language-
independent statistical approaches. As with everything, it has its cons as you
can extract information from more languages, but, in general, with less
certainties.

But in general, it is good to see that the NLP area is still alive somewhere,
as I can't seem to find any NLP jobs where I live! :)

Edit: I've read it in the diagonal, and it is based on a Neural Network, so in
theory, if it was trained in other languages, it could return good enough
results as well. It is normal for English/American authors to include only
english datasets, but I would like to see an application to another language..
This is a very specialized domain of knowledge, so I'm quite limited on my
analysis..

~~~
wodenokoto
It's not particularly hard to port nlp to other languages when you use these
methods. You are mostly limited by tagged corpora.

Nlp is very much alive and well.

------
the_decider
According to their paper
([http://arxiv.org/pdf/1603.06042v1.pdf](http://arxiv.org/pdf/1603.06042v1.pdf)),
the technique can also be applied to sentence compression. It would be cool if
Google publishes that example code/training-data as well.

------
neves
Shouldn't the title be renamed for "The World's Most Accurate Natural Language
Parser _For English_ "?

It's impressive how Google's natural language features, since the simpler
spell check, degrades when it work with languages different from English.

------
zodiac
> It is not uncommon for moderate length sentences - say 20 or 30 words in
> length - to have hundreds, thousands, or even tens of thousands of possible
> syntactic structures.

Does "possible" mean "syntactically valid" here? If so I'd be interested in a
citation for it.

Also, I wonder what kind of errors it makes wrt to the classification in
[http://nlp.cs.berkeley.edu/pubs/Kummerfeld-Hall-Curran-
Klein...](http://nlp.cs.berkeley.edu/pubs/Kummerfeld-Hall-Curran-
Klein_2012_Analysis_paper.pdf)

------
joosters
I don't see how a linguistic parser can cope with all the ambiguities in human
speech or writing. It's more than a problem of semantics, you also have to
know things about the world in which we live in order to make sense of which
syntactic structure is correct.

e.g. take a sentence like "The cat sat on the rug. It meowed." Did the cat
meow, or did the rug meow? You can't determine that by semantics, you have to
know that cats meow and rugs don't. So to parse language well, you need to
know an awful lot about the real world. Simply training your parser on lots of
text and throwing neural nets at the code isn't going to fix this problem.

~~~
jonknee
That's exactly why it's using a neural net and yes, a lot of text will fix
this problem. The only reason why we know cats meow and rugs don't is by
learning about cats and rugs. Throw enough training data at it and the parser
will figure out what is meowing.

An interesting example of this you can easily try for yourself is playing with
Google's voice to text features--if you say silly things like "the rug meowed"
you will have terrible results because no matter how clearly it can hear you
its training data tells it that makes no sense.

~~~
atdt
This is actually a rather serious limitation of statistical approaches to
language: they work best with utterances that have already been said, or with
concepts that are already strongly associated in common speech. Such
utterances may make up the bulk of what we say and write, but the remainder
isn't gobbledygook. It contains most of the intimacy, poetry, and humor of
interpersonal communication, all of which trade on surprise and novelty.

~~~
andreasvc
That's basically Chomsky's argument with the "colorless green ideas" sentence.
If you put words together to form a sentence never seen before, supposedly a
statistical model cannot help you. The thing is, a paper later showed that a
simple Markov model is actually perfectly able to discriminate this
grammatical sentence from an ungrammatical one. Novel and surprising sentences
are never completely alien. They use familiar structures of the language, and
combinations of words and other building blocks that we have seen before, and
this is exploited when we analyze such sentences. Surprise and novelty are
actually strongly related to statistics (cf. information theory).

------
aaron-santos
I'd love to see the failure modes especially relating to garden path
sentences. [1]

[1] -
[https://en.wikipedia.org/wiki/Garden_path_sentence](https://en.wikipedia.org/wiki/Garden_path_sentence)

~~~
xigency
Statistical models for parsing are almost guaranteed to fail at parsing garden
path sentences.

------
mindcrash
Anyone planning (or already busy) training Parsey with one of the alternative
Treebanks available from Universal Dependencies [1]? Would love to know your
results when you have any :)

I am personally looking for a somewhat reliable NLP parser which can handle
Dutch at the moment. Preferably one which can handle POS tagging without
hacking it in myself.

[1] [http://universaldependencies.org/](http://universaldependencies.org/)

------
hartator
> At Google, we spend a lot of time thinking about how computer systems can
> read and understand human language in order to process it in intelligent
> ways.

There is 6 links in this sentence in the original text. I get it can help to
get more context around it, but I think it's actually making the text harder
to "human" parse. It also feels they have hired a cheap SEO consultant to do
some backlink integrations.

~~~
RichieAHB
Is this a joke? SEO consultant at Google? Backlinks? It's just a simple way to
give it context, and I barely noticed the links.

Can't work out if this is a joke.

------
jdp23
Parsey McParseface is a great name.

~~~
chubot
Yeah I noticed that :) Is this a Simpsons reference? Can't quite place it.

~~~
UK-AL
Havent heard about Boaty McBoatface?

~~~
worldsayshi
I kind of feel that it dates further back?

~~~
QuercusMax
Definitely; I think it's originally from Friends (possibly in a slightly
different form; Xey McXerson). Haven't tracked down the exact reference, but
it's well over a decade old.

[http://www.duetsblog.com/2016/03/articles/trademarks/namey-m...](http://www.duetsblog.com/2016/03/articles/trademarks/namey-
mcnamerson-what-happens-when-you-crowdsource-names/)

------
sourcd
What would it take to build something like "wit.ai" using SyntaxNet ? i.e. to
extract "intent" & related attributes from a sentence e.g.

Input : "How's the weather today"

Output : {"intent":"weather", "day":"Use wit-ai/duckling", "location":"..."}

------
amelius
How would you feed a sentence to a neural net? As I understand, the inputs are
usually just floating point numbers in a small range, so how is the mapping
performed? And what if the sentence is longer than the number of input
neurons? Can that even happen, and pose a problem?

~~~
vhold
What wodenokoto said, and also look up "Word Embeddings", word2vec is a
popular method.

[https://www.tensorflow.org/versions/r0.7/tutorials/word2vec/...](https://www.tensorflow.org/versions/r0.7/tutorials/word2vec/index.html)

There's a bunch of blogs, tutorials, etc, around word2vec and other methods of
generating vectors from a training set of words.

Also, in the tensorflow models codebase where this syntaxnet code lives, there
is an another tensorflow-using-method of generating word embeddings with
demonstration code called Swivel

[https://github.com/tensorflow/models/tree/master/swivel](https://github.com/tensorflow/models/tree/master/swivel)

------
w_t_payne
Cool - I reckon I'm going to try to use it to build a "linter" for natural
language requirements specifications. (I'm a bit sad like that).

------
WWKong
Anyone know a tool that does Natural Language to SQL?

~~~
escap
[http://quepy.machinalis.com/about.html](http://quepy.machinalis.com/about.html)
[http://kueri.me/product/](http://kueri.me/product/)

------
Animats
This could lead to a fun WordPress plug-in. All postings must be parsable by
this parser.

Surprisingly, this thing is written in C++.

------
zem
one interesting use i can think of is new improved readability scores that can
take into account words that are common or uncommon depending on part of
speech. (e.g. a text that used "effect" as a noun would be lower-level than
one that used "effect" as a verb)

------
vicaya

        1. WordNet
        2. ImageNet
        3. SyntaxNet
    
        ...
    
        n. SkyNet

------
instakill
What are some use cases for this for hobbyists?

------
degenerate
I'd love to let this loose on the comments section of worldstarhiphop or
liveleak and see what it comes up with...

------
bertan
Parsey McParseface <3

------
jweir
Parsey McParseface? Nice touch Google.

[https://github.com/tensorflow/models/tree/master/syntaxnet/s...](https://github.com/tensorflow/models/tree/master/syntaxnet/syntaxnet/models/parsey_mcparseface)

~~~
daveloyall
Bill, help me name this parser.

What kind of parser is it?

An _English_ parser.

... :)

You know, it's a brilliant name. It's a person's name, like Siri or Cortana.
But if they'd named it Jane or John, the mass media would have a fit. So they
side-stepped that by making it sound computer-y, while remaining a proper
name. Brilliant.

~~~
Bombthecat
Why is everything English only? :( there are other languages too.

#otherlanguagesmatters

~~~
TeMPOraL
Because we didn't win the II World War... :).

(INB4 yes, that's the reference to an old joke about Lufthansa pilot in
Frankfurt complaining he's forced to speak in English to the control tower)

Honestly though, English is a quite good starting language due to its
simplicity, but I do hope more attention will be expended on making the
technology like this trainable on other languages as well.

~~~
tuukkah
Simplicity!?

------
scriptle
Did I just read it as Skynet ?

------
PaulHoule
Meh.

This kind of parser isn't all that useful anyway. Parts of speech are one of
those things people use to talk about language with, but you don't actually
use them to understand language.

~~~
corin_
You do subconciously. When you read "Dave punched John" you don't need to
think "hmmm, is 'John' the object or the subject?" but if your brain hasn't
figured out which is the object and which the subject you won't know who is
hitting who.

You don't need to be able to name or define parts of speech but you need to be
able to parse them, or you won't understand anything.

~~~
TeMPOraL
Yeah, except are you sure that the way you parse a sentence maps 1:1 to what
you learned as "parts of speech"? I think an equivalent understanding, one
that seems to be more intuitive to the way my own brain works (if I can
believe introspection), is that "punched" is to be read by default as
"\--punched-->", and "was/is punched by" pattern-matches to <\--punched--".
Arrow denotes who's punching whom.

~~~
corin_
That's your brain taking a shortcut because you know that "punched" is a verb,
and that a verb in that form is likely to be followed by a subject noun, etc.
Even if you never study grammar and couldn't even answer the question "which
is the verb in this sentence?" your brain has still learned to recognise the
different types of word which is why you understand them.

Disclaimer: I'm not a linguist, but by layman's standards I'm pretty
confident.

