
OpenNMT: Open-Source Neural Machine Translation - groar
https://github.com/OpenNMT/OpenNMT
======
blennon
The amazing thing about this new wave of innovation in AI is that the
technology is democratized -- anyone can have near state-of-the-art technology
to power their applications without paying for it.

One of the lasting principles in the "business of AI" is that the data is the
true source of value. If you don't have a monopoly on the data, you're going
to have a tough time sustaining a competitive business because everyone else
will more or less be able to clone your technology. My PhD advisor taught me
this based on his experience founding one of the first really successful
neural networks companies, HNC, which was later acquired by Fair Isaac (FICO).
HNC put together a consortium of banks in order to train their credit card
fraud detection system and maintained a monopoly on the data. FICO still runs
this business today and I've heard from recent employees it still runs on the
same old single-hidden-layer multi-layer perceptron architecture as it did in
the 90s.

------
pmontra
I tested it on the live demo with a few sentences from Google News, English to
Italian and viceversa. It looks to be at the same level of Google Translate:
some sentences are better translated to Italian by OpenNMT, others by Google,
even in the same excerpt from the news.

Both tools seems to be better at translating to English, but maybe because I
don't realize some of the mistakes they do.

Overall it's a great tool. I suggest to read the FAQ at
[http://opennmt.net/FAQ/](http://opennmt.net/FAQ/)

~~~
srush
The code implements a pretty close model to what Google Translate has
published
([https://arxiv.org/abs/1609.08144](https://arxiv.org/abs/1609.08144)).
However the two systems are likely trained on very different datasets.

------
srush
Hi, I'm Alexander Rush (@harvardnlp), one of the project leads on OpenNMT and
an assistant prof at Harvard. Feel free to ask me anything about the project.

~~~
arthur2e5
As a terrible human being, I tested Chinese-to-English translations on f-word-
based profanity on the demo page ([https://demo-
pnmt.systran.net/production](https://demo-pnmt.systran.net/production)). The
results were arguably not as good as Google Translate, which can recognize the
colloquial form 操[^1] but not the more orthodox 肏. PNMT's results were great
in another way as they form more fluent English.

    
    
      [^1]: Everyone hijacked the character.
    

On the other hand, OpenNMT appears able to translate the "f __* you! "
sentence to "你妈！" in Chinese, a form of "操你妈" with the F word not spoken but
implied. The single-f-word sentence generates amusing variations when used
with different punctuations, although many ("...", "?", "!", "") fail by
reducing the sentence to a normal call for one's own mother.

These experiments make me curious about what kind of corpora the system was
trained on.

* * *

The test sentences used were:

    
    
        操你十八辈子祖宗！[^2]
        操你十八辈祖宗！
        操你妈！
      [^2]: This one is technically wrong.
    

For the orthodox version, replace all instances of 操 with 肏.

As an additional experiment, I tested some variations of "mother" profanities
without "操":

    
    
        1. 你妈！
        2. 他妈的！
        3. 他妈的。
    

Between #2 and #3, PNMT appears more sensitive to punctuations than human do.
#1 is only included as a round-trip validation.

Edit: (#1) ... and is not supposed to be translated back into profanity due to
high false positive rate.

~~~
srush
While this type of translation is heinously understudied, the opposite problem
of controlling the politeness forms of translation is actually an important
area of research. For instance Controlling Politeness in Neural Machine
Translation via Side Constraints
([http://homepages.inf.ed.ac.uk/abmayne/publications/sennrich2...](http://homepages.inf.ed.ac.uk/abmayne/publications/sennrich2016NAACL.pdf))
is something that OpenNMT can support.

------
imurray
Neural translation has been really amazing and impressive. I stumbled into a
way to be mean to Google translate this week:
[https://twitter.com/driainmurray/status/818934530207862784](https://twitter.com/driainmurray/status/818934530207862784)

OpenNMT can also run amok given totally unreasonable input too (example
below). I'm being unfair, but the more serious point is that translation
systems might go crazy in less obvious ways. While fluency is greatly
improved, they certainly don't always get the meaning correct. I'm excited to
see what people will do to better diagnose when we can trust the translation.

Input: _" Toll ach toll toll ach toll toll toll toll toll toll toll toll toll
toll toll toll toll toll toll toll toll toll toll toll Deutsch."_

Output: _" Tell me great great, great great, great great great great great
great great great great love great ? ? ? great love ? ? ? ? ? ? ? great ? ? ?
? great love ? ? ? great great ? ? ? great great ? ? ? great great ? ? ? great
great ? ? great great ? ? ? great great ? ? great great ? ? great love ? ?
Good ? ? ? great love ? ? great love ? ? Good love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great
love ? ? great love ? ? Good ?"_

[Each ? was a musical note glyph, but HN ate the unicode character.]

~~~
srush
Unfortunately we do run into these types of "robustness issues" in our
experiments. They're rare enough that they don't really change many of the
performance metrics of the system, but they are quite embarrassing. It is
currently a bit of a mystery as to why they go so wrong, better diagnostic
tools are sorely needed. Informally, these mistakes similar to the more well-
known mis-classification of noise issues in cnn's for vision.

------
skykooler
What resources does this require to run? Obviously you need some fairly heavy
lifting for the initial training, but once trained, could the models be used
on, say, a mobile phone?

~~~
srush
Yes, you can actually run them on a phone! We had an earlier demo of very
strong system running on android [https://github.com/harvardnlp/nmt-
android](https://github.com/harvardnlp/nmt-android)

Without tricks the large models takes up about 700 megs, opennmt.net/Models/.
They can run on a standard CPU relatively quickly. We also have a pure C++
decoder if you want to do translation without the NN framework.

------
vicaya
These NMTs seem tragically bad at translating simple straightforward Chinese
e.g.: 我要下班了。下班了再说。

    
    
      Google: I have to get off work. Off to say.
      OpenNMT: I am going to work. After work, again.
    

If this is the state of art for NLP. We have a long way to go.

~~~
srush
It's not there yet, but the improvement has been quite significant in
aggregate. Key table from the GoogleNMT paper, empirically showing a 60%
relative improvement on this task:

    
    
                      PBMT  GNMT  Human Relative Improvement
    

English → Spanish 4.885 5.428 5.504 87%

English → French 4.932 5.295 5.496 64%

English → Chinese 4.035 4.594 4.987 58%

Spanish → English 4.872 5.187 5.372 63%

French → English 5.046 5.343 5.404 83%

Chinese → English 3.694 4.263 4.636 60%

~~~
vicaya
This is obviously data dependent. I suspect that the advantage of human is
much higher in colloquial content vs written (esp. news) content. "Universal
Adversarial Perturbations"[1] last year showed that you can easily generate
reasonable (to human) perturbations to completely fool the state of the art
DNNs for images. I suspect that the same is true for the current batch of NMTs
as well. As a simple demo, I change the example Chinese a little
(就要下班了。下班了再说吧。Only aux characters changes with the same meaning) and all NMTs
failed spectacularly in different ways.

    
    
      Google: It is necessary to get off work. To say it again.
      OpenNMT: it's going to work. Go back to work again.
      Baidu: It's going to work. After work.
    

[1] [https://arxiv.org/abs/1610.08401](https://arxiv.org/abs/1610.08401)

~~~
srush
Yeah this is a nice connection. Note however that there has been much less
success in using perturbation in language. The fact that inputs are discrete
makes it harder to apply some of the tricks in the adverserial image work.

------
chillydawg
Train top20 language pairs.

Make mobile app that takes picture, runs OCR, detects language and feeds into
model.

Translate

???

Profit

All offline, after the models have loaded once.

~~~
srush
Better: Train the same model for all twenty language pairs
([http://opennmt.net/Models/#multi-way---
fresptitrofresptitro](http://opennmt.net/Models/#multi-way---
fresptitrofresptitro)).

Even better: Use OpenNMT to do the OCR too
([https://github.com/opennmt/im2text](https://github.com/opennmt/im2text)).

