
An Adversarial Review of “Adversarial Generation of Natural Language” - sebg
https://medium.com/@yoav.goldberg/an-adversarial-review-of-adversarial-generation-of-natural-language-409ac3378bd7
======
slashcom
It's worth noting this post caused a very intense twitter debate among the NLP
and DL communities, especially after Yann Lecun replied to Yoav's comments.
[https://www.facebook.com/yann.lecun/posts/10154498539442143](https://www.facebook.com/yann.lecun/posts/10154498539442143)

~~~
sillysaurus3
What are some twitter links? I'd be interested to see the kind of nuanced
conversations that twitter can spawn. Also looking to follow some NLP people.

~~~
Mathnerd314
Twitter has a search box:
[https://twitter.com/search?f=tweets&vertical=default&q=An%20...](https://twitter.com/search?f=tweets&vertical=default&q=An%20Adversarial%20Review%20of%20%E2%80%9CAdversarial%20Generation%20of%20Natural%20Language%E2%80%9D)

This newsletter looks interesting (mentioned on Twitter):
[http://digest.deeplearningweekly.com/issues/deep-learning-
we...](http://digest.deeplearningweekly.com/issues/deep-learning-weekly-
issue-44-ai-understanding-google-mobilenet-machine-intelligence-summits-
sobolev-trainings-for-nns-flag-planting-and-overselling-results-61446)

This seems like the only real discussion from the search:
[https://twitter.com/kchonyc/status/873305485428088833](https://twitter.com/kchonyc/status/873305485428088833)

I guess search is broken, the author's twitter feed has a bunch more
discussion
([https://twitter.com/yoavgo/with_replies](https://twitter.com/yoavgo/with_replies)):

[https://twitter.com/yoavgo/status/872831207163265024](https://twitter.com/yoavgo/status/872831207163265024)
[https://twitter.com/yoavgo/status/872968874521702400](https://twitter.com/yoavgo/status/872968874521702400)
[https://twitter.com/Smerity/status/872971766959718400](https://twitter.com/Smerity/status/872971766959718400)
[https://twitter.com/yoavgo/status/873489277157507072](https://twitter.com/yoavgo/status/873489277157507072)
[https://twitter.com/jacobandreas/status/873109327644573696](https://twitter.com/jacobandreas/status/873109327644573696)
[https://twitter.com/yoavgo/status/873175136056336386](https://twitter.com/yoavgo/status/873175136056336386)
[https://twitter.com/hugo_larochelle/status/87336968867543040...](https://twitter.com/hugo_larochelle/status/873369688675430400)
[https://twitter.com/kchonyc/status/873306255204507648](https://twitter.com/kchonyc/status/873306255204507648)
[https://twitter.com/yoavgo/status/873786844315607040](https://twitter.com/yoavgo/status/873786844315607040)
[https://twitter.com/haldaume3/status/873565061754781697](https://twitter.com/haldaume3/status/873565061754781697)

~~~
sp332
None of those tweets have the phrase you searched for, so why do you conclude
that search is broken?

~~~
Mathnerd314
Several of the tweets link to the post, and the title shows up in Twitter's
preview box (although not in the tweet proper). If Twitter search "worked" it
should have found those, via some magic string munging or whatever.

My definition of "works" is
[https://en.wikipedia.org/wiki/DWIM](https://en.wikipedia.org/wiki/DWIM),
others are free to disagree.

------
fizixer
This is an instance of the general issue of conflict across, what I like to
call, the salesperson-slacker spectrum.

Most researchers/academics lie somewhere on this spectrum. (Well I guess most
human beings involved in any activity probably).

On the one end are salespeople who love to make a mountain out of a molehill
they just discovered. On the other, slackers are like the perfectionists who
never get anything done because they never resolved their analysis-paralysis.

There are very few who are exactly in the middle of the spectrum. The middle
is a point of unstable equilibrium. You have to work very hard to stay there
and can easily fall off to one side or the other.

~~~
obstinate
I wouldn't say perfectionism and sales-orientedness are on the same dimension.
However, I have noticed that being given the hard-sell is more often than not
associated with the product being sub par. A product or project that is doing
very good work often doesn't have to sell quite as hard, because the work
speaks for itself. (My experience on this issue mostly deals with internal
teams. I don't know how much this applies to external teams.)

This is especially true when two products have teams of about equal size,
experience, and expertise. Time spent selling is time not spent making a
better product. All other things being equal, every hour spent selling is one
less hour spent working on making your product better.

~~~
kthejoker2
Thought experiment at the extremes: if you wait for your product to be
literally perfect you will never sell anything.

And of course you can sell a product that doesn't even exist.

The dimension is confidence or maybe approval.

~~~
aisofteng
That's not a thought experiment, it's just a statement.

------
wodenokoto
If you want to read more discussion on this topic (and this article), see the
article "A computational linguistic farce in 3 acts" and its HN discussion:
[https://news.ycombinator.com/item?id=14532306](https://news.ycombinator.com/item?id=14532306)

------
paganel
I'm not going to comment about that paper being published on arxiv and I don't
generally care about the NLP vs DL debate, just wanted to say that those
generated examples did look indeed all rubbish to me.

As do most of the Google Translate pieces, even though I get the feeling that
automatic translation of texts is now seen almost as a solved problem (it's
not): all that Google Translate does is change some text from some original
language to a second one, which is not a real language, it's just a language
that's sometimes very close (grammatically and lexical) to one which the
agent/user knows.

The idea is that we should try to look harder and have fairer judgements about
the actual results and not get stuck on the methodologies.

------
a3_nm
> This post is also an ideological action w.r.t arxiv publishing

Anyone else thought that this was very weird? The author appears to be
complaining about the fact that reputable people/labs can post a PDF on arXiv
and be taken seriously. How is this avoidable? Without arXiv, they could just
post the PDF on their website or anywhere else.

The "risk" associated to publishing crap on arXiv is the same as always: have
people notice it's crap and get a bad reputation. I'm not sure what ideology
has to do with it.

~~~
Al-Khwarizmi
Indeed it's possible in theory to do flag planting without arxiv, for example
by posting a timestamped technical report to an institutional repository, but
it has never been a common thing. Probably because you don't get much of an
audience, so people won't cite you unless you bug them in reviews. And if
someone publishes the idea some months later in a paper, they will probably
get credit as there's the reasonable assumption they just hadn't read your
report.

On the other hand, arxiv has a huge audience, lately maybe even more than
DL/NLP conferences, making the flag planting really effective, especially if
you are from a prestigious group. So there is a real problem now with large,
prestigious groups posting half-assed preliminary results in arxiv, and
deterring more modest groups from working on the problem or bogging them down
because they now need to compare themselves to the well-known arxiv approach,
which often has serious reproducibility issues because it has not been peer-
reviewed.

I'm not an arxiv hater, in fact I check arxiv every day and for the last year
or so I have posted most of my papers in it. But the problem is real and
something must be done. Not about arxiv which is just the messenger (and a
good one), but about the flag-planting culture using it that has emerged in
the field.

~~~
a3_nm
Thanks a lot for bothering to explain the problem. I have never seen this in
my field (theoretical computer science), so I wasn't really aware of the
problem. (More accurately: reviewers may ask why you do not compare yourself
to arXiv preprints, often with indulgence if they are recent, but I have not
seen this culture emerge of posting half-baked results to arXiv to claim
priority.)

It appears that we agree that the problem is not with arXiv. Part of the
problem is unsolvable: if prestigious groups can announce what they are
working on and discourage other groups from working on the same problem, this
may just be reasonable self-interest from the smaller groups and can hardly be
avoided. As for reviewers asking for comparison to these works, I guess the
problem lies with the reviewers: if an arXiv preprint has not been refereed
and/or is hard to reproduce, it should be OK to say so in another paper and
not be blamed for the lack of comparison.

In any case, this is an interesting problem, thanks again for making me aware
of it.

------
thearn4
For another point of comparison, has there been a "us vs. them" dynamic in the
computer vision community when it comes to deep learning? After all, it seems
like deep convolutional networks did sort of railroad there way through a ton
of topics in that field. I used to do image analysis research, but moved on
before much of this came into play so I lack any sort of inside scoop.

------
denzil_correa
> Communities will naturally recognize contributions and give credit when
> credit is due. It's always happen that way.

"Let the market decide!"

~~~
mostafab
The market needs signals, and the point of Goldberg was that those signals are
missing with arxiv, and that just looking at the lab reputation was a poor
signal. I am working on it, contact me for details.

