
Psycholinguists Are Foolish to Dismiss GPT-2 and Bert - ZhuanXia
https://www.lesswrong.com/posts/ZFtesgbY9XwtqqyZ5/human-psycholinguists-a-critical-appraisal
======
PaulHoule
It can "write" like Nabokov but it can't read.

Symbolic A.I. Did not fail at text generation but it did not have a real
market. Today the web is full of "mad-lib" text generated by SEO spammers.
Practically many software users don't seem to care if forms in an application
use the right articles (e.g. 'a', 'an'). For that matter you can get
superhuman performance at grammar by squashing it's to its 100% compared to
ordinary people who do not punctuate correctly.

The language embedding business has traded percentages of accuracy on
benchmarks for percentages of possible commercial accuracy because of
information that is provably lost in the front end.

The dialog between linguists and the embedding industry has been slow to
develop because when you look for linguistic or semantic concepts in the
embedding (say by training a classifier) they are hard to find.

For instance you might try a set of 10 color/non-color words, predict the
result for another 10 words based on the WordNet embedding and superficially
think it is promising. With every word you add to the training and test sets
it will perform worse. It passes superficially because it knows many biases
like 'anyone named Tyrone is a thug.'

That kind of machine may be able to write fiction, but you need a different
level of understanding to act justly, do hard things correctly, etc.

