

Noam Chomsky on Where Artificial Intelligence Went Wrong - edmaroferreira
http://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/

======
antics
Chomsky's argument here is basically the same argument he used to reinvent
linguistics in the 50's and 60's, so understanding what's _really_ being said
here depends on a bit of context. Since neither previous discussions nor this
article has quite nailed it, the discussion is indeed worth having (sorry,
ColinWright).

The crux of the debate is this. When Chomsky was a grad student at UPenn most
linguists thought that language was learned by a complicated mimicry -- that
we learn language by imitating behavior, similar to how birds learn to call.
The "hard problems" of linguistics were completely solved and linguistics had
become a primarily classificatory science, with linguists simply cataloging
words into parts of speech. This line of thought was interchangeably known as
behavioralism or empiricism.

One of Chomsky's transformative insights was that most sentences that are
reasonably long are completely unique in human history, and will also never
again be uttered by another person, ever. (For example, try Googling that
exact sentence.) One consequence of this is that people realized that the
mimicry argument could not really account for the robust well-formed structure
of sentences _and_ give us an infinite set of them. What we need to generate
an infinite set of well-structured sentences is a _grammar_. Thus the
universal grammar was born, and while Chomsky did not convince all the
prominent linguists of the time, he did convince all their grad students, and
the field of linguistics has seldom looked back.

Where this begins to intersect with AI is where Chomsky is usually criticized
for not having been quite revolutionary enough. His outline of linguistics
basically split the field into semantics (which studies the _meaning_ of
language) and syntax (which studies the _structure_ of language). He argued
that everything about language that must be interpreted (like meaning) must go
on the semantics side of the line, and everything else should go on the other
side of the line. He does not believe syntax to be interpretive at all, and
tends to react violently when anyone tries to push empiricism into the syntax
dialogue. Even a lot of his students don't buy that syntax is completely not-
interpretive, and so someone like Lakoff would claim that if he was
revolutionary, he was not quite revolutionary enough.

Here's what this means for AI. Chomsky sees the statistical approach to
learning as a type of empiricism. You take a corpus, learn some stuff
statistically, and then perform well on a task. To someone like Chomsky this
probably looks like Skinner's old model, but instead of words like "mimicry",
we use words like "statistical inference." Remember that empiricism and syntax
should be strictly separate, and it becomes easy to see why something like
this would make him cranky.

Of course, computer scientists like Norvig and linguists like Lakoff disagree.
Their collective argument is that some aspects of syntax are indeed
interpretive, and that (in the case of Norvig) they can be learned
statistically (using, e.g., PCFGs). For example consider the sentence "John
called Mary a Republican and then SHE insulted HIM". This really only makes
sense if you presume that the participants think that "Republican" is an
insult, but how do you know that? The answer seems to be through some sort of
past experience, which Norvig would say can be and should be modeled
statistically. And that in short is the debate and why it exists.

The other complaints with what Chomsky said here are that it's scientifically
incorrect. Chomsky claimed, for example, that statistical models are basically
not real science, and not used in the history of science, which is obviously
wrong. Norvig pointed out, for example, that in physics sometimes our only
choice is to infer something statistically, as in the case of the
gravitational constant or the Higgs boson. But given Chomsky's history, it's
fair to assume he meant this in the context of behavioral science, in which
case his point is mostly true (modulo the "old" model of linguistics, which he
hates). But of course it's important to remember that his model of linguistics
was also without precedent in the history of science, so that alone is not
really justification for his position.

~~~
wololo
Non sequiturs:

1\. > _One of Chomsky's transformative insights was that most sentences that
are reasonably long are completely unique in human history_

... as with a sufficient amount of information in every error-tolerant format.

Every photo ever taken of the Mona Lisa is unique because the pixels are
different, but has a very similar high-level meaning, echoing the paraphrase
detection task results discussed in
[http://www.socher.org/index.php/DeepLearningTutorial/DeepLea...](http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial).

2\. > _One consequence of this is that people realized that the mimicry
argument could not really account for the robust well-formed structure of
sentences and give us an infinite set of them._

What about gradient well-formedness? Does anyone even in generative
linguistics believe in programming language-hard grammaticality? What about
acceptability?

3\. > _What we need to generate an infinite set of well-structured sentences
is a grammar. Thus the universal grammar was born ... the field of linguistics
has seldom looked back._

...which grammarless statistical models have no problem doing (or if you want
to reject them for making mistakes, do you reject people?)

Which linguistic univerals, if any, have survived the test of time? What are
some testable predictions it has made?

I hope I'm not missing the point, and apologies if this has already been said
(but not in these exact words, ha ha...)

------
david927
Chomsky on the lack of progress in AI:

 _If you take a look at the progress of science, the sciences are kind of a
continuum, but they're broken up into fields. The greatest progress is in the
sciences that study the simplest systems. So take, say physics -- greatest
progress there. But one of the reasons is that the physicists have an
advantage that no other branch of sciences has. If something gets too
complicated, they hand it to someone else.

If a molecule is too big, you give it to the chemists. The chemists, for them,
if the molecule is too big or the system gets too big, you give it to the
biologists. And if it gets too big for them, they give it to the
psychologists, and finally it ends up in the hands of the literary critic, and
so on._

~~~
bornhuetter
Great quote. Chomsky has an amazing talent for really getting to the heart of
issues and framing them in useful and insightful ways.

~~~
agumonkey
I watched 'The manufacture of consent' recently, I had a hard time stopping
listening to him. His wording dug right through my skull, very concise yet
simple.

~~~
bones6
He's the most concise when it comes to explaining complicated ideas. Einstein
said something about simple explanations but not too simple. The biggest point
Chomsky made to me was that no real explanation or discussion can be had when
you are limited to 2 minute soundbites on TV. I won't give his example but he
said "You can't say <extremely flammatory counterculture yet objectively true
statement> and not take the time to explain why."

------
ColinWright
Previous discussion (but now closed) here:

* <http://news.ycombinator.com/item?id=4729068>

That one even gave the link to the single page version:

* [http://www.theatlantic.com/technology/archive/2012/11/noam-c...](http://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/?single_page=true)

Many other submissions which I won't enumerate

------
numlocked
As someone who was familiar with the differences between Chomsky and Norvig (a
simplification - maybe better to say Chomsky's view that developing empirical
models is not scientifically interesting), I found this really enriched my
understanding of Chomsky's view and gave me a lot of subtlety. He certainly
doesn't seem antagonistic towards statistic modeling the way I imagined; just
sort of bored by it. Be sure to persist to the end - there are some fun word
games on the final page.

------
deltasquared
Personally I am with Norvig on this issue. Statistical methods work and are
progress.

------
saosebastiao
Norvig still wins, IMO. Statistics may not explain everything, but it does an
amazing job of separating the explainable from the unexplainable. I still see
this as the requisite Minimum Viable Product for Artificial Intelligence. If
we sat around waiting to perfectly understand neurocognitive linguistic
processes before we tried to replicate it, we wouldn't have a functioning
Google Translate for another 200 years.

------
dwiel
Does anyone have any recommended reading about computational efficiency of
language he keeps mentioning?

I've found chomsky, n (2005). Three factors in language design, but was
wondering if anyone knew of anything else, or anything better.

