
SyntaxNet in Context: Understanding Google's New TensorFlow NLP Model - syllogism
https://spacy.io/blog/syntaxnet-in-context
======
projectramo
Wouldn't syntactic parsing also improve by more knowledge of the real world?

From the example in the text:

"They ate the pizza with anchovies"

Isn't it important to know that humans don't use anchovies to eat pizza? You
need some semantics as well. Otherwise, from pure syntax, how can you tell it
apart from:

"They ate the pizza with forks"

~~~
syllogism
I guess I should've connected the dots a bit better on this.

The neural network model offers a pretty compelling answer to this. The idea
is that you each word is represented as a real-valued vector, and these
vectors encode information about the word's meaning. So, "pizza" encodes in it
that it's a type of food, and so does "anchovies". But "fork" is closer to
"knife" and "spoon".

Linear models struggle with this. In a linear model, mostly, each word is an
island. "fork" and "spoon" each have unique IDs, and so you'd better hope you
see enough examples of each of them to understand what they mean. It helps to
add semantic cluster IDs to the words as well, but this isn't quite as good as
the word vectors.

What doesn't work particularly well is encoding external knowledge explicitly.
Having a semantic resource that tells you explicitly that "fork" and "spoon"
are members of class "cutlery". This has been tried by many people, and it
produces small or no benefit.

~~~
projectramo
How does the NN encode the fact that when you eat "with" a fork, the word
"with" isn't the same sort of "with" as eating something "with" anchovies.

Everyone has their favorite way to think about NNs. I think of them as
hyperplanes slicing up n-dim space. I just don't know how they "reason" about
semantic information.

~~~
danieldk
_How does the NN encode the fact that when you eat "with" a fork, the word
"with" isn't the same sort of "with" as eating something "with" anchovies._

By looking at the noun in the NP embedded in the PP rather than the
preposition. Whether this actually happens depends on various factors:

\- In some dependency annotation schemes it's actually the head of the NP that
attaches to the head with the prepositional phrase relation. Though this is
uncommon, since typically the preposition is the direct dependent of the head.

\- In some transition systems, attachment of a dependent is done when all its
dependents are attached. So, at the moment the preposition is attached to a
head, you already know what the preposition governs.

\- In some transition systems, previously-made attachments can be changed
(e.g. when you encounter a noun in a PP which suggests a different
attachment).

\- You might already look ahead in the buffer, either by having a small number
of buffer positions as the input of the NN or by forming a representation of
the current buffer using an RNN.

tl;dr NN can definitely learn such things, but you may have to help a bit by
picking the right transition system or classifier inputs.

------
nxzero
>> "On the time-honoured benchmark for this task"

Anyone know what exactly this benchmark was, it's relevance, and if it's been
independently reproduced?

~~~
LanguageGamer
According to the paper linked by the original announcement [1], the parser
scores 94.41% for unlabeled attachment on the Wall Street Journal corpus [2],
a parsed and labeled data set of 30 million words.

This corpus is a standard for NLP research on english syntax, but I think its
worth remembering there is a great deal of disagreement among linguists about
what the syntax of english is and what the lexical categories are.

[1] [http://googleresearch.blogspot.com/2016/05/announcing-
syntax...](http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-
worlds-most.html?m=1) [2]
[https://catalog.ldc.upenn.edu/LDC2000T43](https://catalog.ldc.upenn.edu/LDC2000T43)

------
fizzbatter
Somewhat off topic, but for anyone more familiar with spaCy:

Any idea if spaCy has a sane way to be used in other languages? Specifically
Rust, in this case? I'm in need of a decent NLP library to train and or map
sentences to intents. Somewhat similar to what wit.ai offers, but it needs to
be offline-able. spaCy sounds great, but requiring a runtime _(if spaCy indeed
does)_ on the installed system is a no-go for me.

Not knowing much about Python, It looks like it is actually compatible with
CPython, so could it therefor be compiled to C and imported into Rust?

~~~
syllogism
spaCy's implemented in Cython, which compiles into C++. So, the end-state at
the moment is a .so object that expects to be loaded into Python.

Now, you can instead tell Cython to compile into standalone objects, so that
they can be imported from C/C++ code. I'm not well versed on this, but if you
can call into C/C++ libraries easily from Rust, you should be able to call
into spaCy.

You'll miss some of the niceties from the Python layer, but the C-level API is
pretty well behaved. So, you should be able to get by okay.

~~~
danieldk
Curious: since spaCy is a commercial project (I know that it is open source),
did you consider writing it in C or C++, it would make binding to other
languages a lot easier. I can imagine that importing the Python runtime can be
problematic in some projects. Also, I assume that you will see some Python
types in the API when you want to call the parser from C?

(My own NN dependency parser is written in Go, since it's mostly for research,
but if I wanted to make it embeddable, I'd have implemented it in C++ or Rust
with a C API.)

~~~
syllogism
Well, I like writing in Cython, and I figured I was better off pleasing some
people (Python) a lot before trying to please a lot of people a little. It
won't be hard to port to C++ when the time comes.

------
visarga
Haha. It's like AST (abstract syntax tree), but for human language. We just
need the code generator to make human language run on CPUs. </joking>

