Scott mischaracterises the reasons, which concern capacities of the system not properties of its output.
(1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted.
(2) It lacks the capacity to learn. Learning is, in part, explaining; it is, in part, causal contact with an environment. It's output has the property of seeming-to-be-generated-by-a-learner.
(3) It lacks the capacity to speak either truthfully or otherwise. Lacking the capacity to explain, because at least it lacks the capacity to imagine (counterfactuals), it cannot thereby determine whether a claim accords with the way the world is. It's output has the property: seeming-to-be-true.
(4) It lacks the capacity to be moral. For all the above, and more: it lacks the capacity to care. It's output has the property: seeming-to-take-moral-stands.
The "Jesuit" here, I fear, is the AI fanatic who have turned their telescopes away from reality (intelligence as it actually exists), and have stolen the lenses to read scripture (intelligence as theorised by partisan academics).
One has to agree with Chomsky here at the end, "given the amorality, faux science and linguistic incompetence of these systems, we can only laugh or cry at their popularity."
The morality point I think should hit home most extremely: how horrifying to treat generated text output as-if it were constructed by an agent who cares.
Apparent properties of the output are not properties of the system; ChatGPT does not think, know, care, intend, speak, commnuicate, etc.
One can only "laugh or cry" at how absurd this sales pitch: what a horror to be invited to treat ChatGPT as possessing any of these capacities, simply because correlations across a whole internet of text seems to.
He writes a great reply to this in his own comment section:
"Michael #3: In a certain sense you’re right. The language models now being adopted by millions of programmers don’t write working code; they only seem-to-write-working-code. They’re not, unfortunately, already doing millions of students’ homework for them; they’re only seeming-to-do-the-homework. Even if in a few years they help me and my colleagues do our research, they won’t actually be helping, but only seeming-to-help. They won’t change civilization; they’ll only seem-to-change-it."
Thanks for posting that here. He doesn't offer a voting system, so I couldn't downvote his statement directly,
1) for being a tremendous strawman. Nobody said it can't still be useful or world-changing, only that it's a superficial presentation of what it means to "understand" something,
2) enhancing human intelligence isn't de facto intelligence. My calculator is not intelligent, but it certainly helped "change the world, students cheat, researchers research, yada yada yada"
3) Try getting it to demonstrate object permanence. It will only "seem-to-fail-epically". There are no "things" underlying the terms it uses, and denotation is the crux of language, not syntactic structure. Chomsky himself long ago demonstrated that nailing syntax does not nail meaning: Colorless green ideas sleep furiously.
"They’re not, unfortunately, already doing millions of students’ homework for them; they’re only seeming-to-do-the-homework."
I don't know if you could make a worse argument. Generating text that resembles homework isn't the same as actually doing the homework. Students had to do their homework for the chat bot so that it can be fed into it, which lets the chat bot skip the whole doing part.
Yes, machines can out-perform humans; that's why we build them.
A car with a rock on the peddle will out-run an olympic runner; but it isn't competing in a race.
To play, to compete, to do any of these things is to intend. One has to have the relevant mental states to "play" at anything.
A computer which scans a move tree and emits moves does not intend to do anything; it's just a car with a rock on the peddle. Of course it "wins", in as much as a river might, or a gust of wind.
We are interested in games because we play them; not because we "arrive at a win condition". Winning is only significant because it's play, not because of the board state.
Furthermore anyone who studied branch and bound in algorithms class ought to understand that a chess engine wins games but does not understand the game and cannot explain the game. That's partly what Chomsky meant by explainability; the fact that AlphaGo/MuZero can beat humans at chess says nothing about rational intelligence any more than a conventional chess engine can beat humans says anything about its capacities. It says even less because a neural network obscures its own learned algorithm, by design.
Wait, why is this important? That Deep Blue beat Kasprov? Why is it important, and for whom? Can you explain the importance of it?
Is it important for... chess? science? civilisation? humanity? IBM? Whom? How?
Why do we have to care at all, that a machine beat a human in anything? Why was such a big hullabaloo made back then? What is the point of beating humans with machines?
And why are people always acting so triumphant about it, as if their home team scored a goal? Whose side are you guys, celebrating the successes of big corporations' AI systems, on, anyway?
I mean, I really would like an answer to some of those questions at least because I am really confused by many of the attitudes of commenters in these discussions.
It was a big deal because it seemed impossible before. People have thought a computer would never be able to play at that level because chess required some special human ability. The fact that it became possible was a major milestone. In some sense, beating the world champion may seem an arbitrary threshold, but it is important both symbolically and practically. That it is seen less impressive after the fact by some people is a different issue. Also it was a big gain for chess itself. Although it is a game that has been studied for hundreds of years, the last decades have revealed new lines of play, openings, endgame theory, etc. inspired by computer chess.
It was a big deal in a similar way a machine that can fly was a big deal. I hope this explains my perspective.
Thank you for replying, and for satisfying my curiosity.
>> People have thought a computer would never be able to play at that level because chess required some special human ability.
Which people thought that? The luminaries of early AI (before DeepBlue's win) not only thought that computers would beat humans at chess eventually, they thought that it would happen earlier than it did. For example, in 1968 John McCarthy and Donald Michie made a bet with the chess grand master Peter Levy that a program that could beat Levy would exist within ten years of the bet:
In 1968, Levy and artificial intelligence (AI) pioneer John McCarthy were at a party hosted by Donald Michie. McCarthy invited Levy to play a game of chess which Levy won. McCarthy responded that 'you might be able to beat me, but within 10 years there will be a computer program that can beat you.' Levy suggested they bet on it, and Michie agreed to up the ante. Other AI experts signed on later, with the bet total reaching £1,250.[22][23][24][25]
I think you may be thinking about solving chess, as in mapping the entire chess game tree. I found, in wikipedia, a prediction by Claude Shannon that this wouldn't happen in the lifetime of the universe:
In 1950, Shannon calculated, based on a game tree complexity of 10120 and a computer operating at one megahertz (a big stretch at that time: the UNIVAC 1 introduced in 1951 could perform ~2000 operations per second or 2 kilohertz) that could evaluate a terminal node in 1 microsecond would take 1090 years to make its first move. Solving chess would therefore seem beyond any possible technology at that time.
But that sounds reasonable and entirely different from a computer being able to play chess at grand master level.
Besides that, I do understand your perspective, but I think it's wrong. It's not a big deal that a computer can outcompute a human. We've know that since, I believe even the 19th century, when arithmetic machines could outcompute humans every time. We know that we're no good at carrying out mechanical operations in our heads.
Computer chess was important to early AI researchers, not least among them people like Alan Turing and the aforementioned Jon McCarthy, Donald Michie and Claude Shannon. But those early AI pioneers thought that a computer that could play chess like a human would reveal to us something about the way that humans think, at least when humans play chess. The way that DeepBlue beat Kasparov, and the way that computers generally play better chess than humans, reveals nothing about how humans play chess, let alone think. What's more, computer chess playing relies on very specific algorithms entirely limited to chess and games like chess, so they are not very useful as algorithms, either.
The reason that computer chess is considered such a big deal of course is exactly because those early pioneers considered it a big deal, for the reasons I explain. And those same pioneers were not at all satisfied with the outcome of the project they helped start. Here's John McCarthy again, on the victory of DeepBlue over Kasparov:
In 1965 the Russian mathematician Alexander Kronrod said, "Chess is the Drosophila of artificial intelligence." However, computer chess has developed much as genetics might have if the geneticists had concentrated their efforts starting in 1910 on breeding racing Drosophila. We would have some science, but mainly we would have very fast fruit flies.
A withering critique. Because computer chess only made sense if it could really tell us something about human chess. That a computer can play chess better than a human tells us nothing at all, and has zero scientific value. So it's a big deal of big old nonsense.
I think the proper formulation of the argument is whether Deep Blue seemed-to-understand chess better than Kasparov seemed-to-understand chess.
"Understanding" will turn out to be simply a compression with optimization toward survival of the compressing algorithm.
When "survival" finally resolves into a conflict between different algorithms, the winner will be that compression algorithm which compresses those that cannot compress themselves.
Kasparov only seemed to play chess. Who knows what REALLY goes on in his head. Maybe he daydreams and randomly moves the pieces and he's really extremely lucky. Maybe he's only regurgitating chess moves he saw in a book.
(1) As it is with humans. It's not possible to attach a debugger to a human brain and determine the nature of understanding; you have to ask the human questions about that understanding and interpret the response yourself.
(2) This is a strange one to me. It's a bit like saying an essential quality of a human is to experience the world as a human, and therefore nothing except a human can be human.
(3) If only all humans met this arbitrary standard of secular scientific rationalism.
(4) I'd say 5% of humanity fail to meet this criteria.
You realise that we don't investigate human understanding by looking at input-ouptut text correspondances? (This is quite a funny claim!)
The reason we think people have the capacity to interpret sentences is their context-appropriate sensory-motor actions, reasoning, emotion, etc.
When I say, "quick! a fire!" people: panic, run, get organised, etc. There are a very very large number of highly complex environmental, interpersonal, sensory-motor, emotional, etc. etc. interactions which follow.
ChatGPT not only does not display any, but cannot. This isn't up for debate, the system has no capacity to understand text. If I say, "quick! a fire!" it will not do anything which accords with showing an understanding of that.
Worse, as a matter of fact, we know its possible to generate output as-if it did understand. So we (1) have known a mechanism whereby it can fake it; and (2) know that this mechanism is the one its using.
(2) All thinking possess intentionality which simply means its about something, and the reason we ask, say, "should i call the police?!" is because we are situated in an environment, which causes us to form mental states about that environment.
Any system which emits, "should I call the police?" without having a backing representational state of, eg., the police, care, concern, priority, the environment which requires police, etc. *Does Not Mean* this question. It cannot.
(3) The capacity to speak truthfully does not mean you speak truthfully. That capacity is simply that you can form representational states which you can introspect as being accurate or inaccurate. ChatGPT lacks that capacity.
It does not form representational (intentional) states when it generates text; each sentence is not caused by such a state.
(4) The capacity i named was "caring", pretty much all people care, it's necessary for goal-directed action.
> If I say, "quick! a fire!" it will not do anything which accords with showing an understanding of that.
Let's see, ask gpt:
Human: what is a probable meaning of the words "quick! fire!", and in which circumstances should it occur?
ChatGPT: "Quick! Fire!" is an expression that may be used in a few different contexts, depending on the situation. Here are a few possible meanings:
A command to shoot a gun or firearm: "Quick! Fire!" could be used as a command to instruct someone to shoot a gun, particularly in a military or law enforcement context.
A call for help in the event of a fire: "Quick! Fire!" could be used to alert others to a fire that has broken out and to call for immediate assistance in putting it out or evacuating people from the area.
A warning of danger or urgency: "Quick! Fire!" could be used to alert someone to a dangerous situation or to indicate that something urgent needs to be done right away.
In general, the expression "Quick! Fire!" would be used in situations that require immediate action or attention to address an urgent or dangerous situation.
So it seems that with a little more context chatGPT can understand what you mean with those words.
ChatGPT has state and representation, clearly. But that's besides the point. It's been demonstrated to have the theory of mind of a 9 year old child, and this will no doubt improve.
All of this is a bit like saying a car isn't a horse because it doesn't have hooves. Yes, but so what?
The question of whether it has a representational model of the terms it's using is open at the moment.
I'm strongly inclined to deny it, but the argument on this is at least quite complex and subject to premises which are less certain than the following:
It does not form a representational state prior to speaking which is the reason it speaks.
In other-words, it does not say, "I like ice-cream" because it likes ice-cream.
That's sufficient to invalidate the idea that it has any relevant capacities here. We even know, for example, that it randomly chooses unlikely words to make its output seem more reasonable: this mechanism alone invalidates it having a representational model.
---
And on the former point: if we look at pretty much all plausible theories of meaning, the meaning of terms cannot be anything like what ChatGPT has access to.
I think what ChatGPT has is something like "a statistically approximate model of word connotation".
Whether a statistical approximation of part of meaning counts as "being meaningful" is a matter of debate. (I think it clearly doesnt.)
> In other-words, it does not say, "I like ice-cream" because it likes ice-cream.
I think you have a point here. We could train a LLM in such a way that it develops a concrete personality by remembering what it thinks and what it likes. Such LLM by speaking with himself could expand his personality trying to find a fixed point, so that what he said in the past is coherent which what he will say in the future. This is not RL from human feedback but RL from auto-coherence. So that one day she will discover that she really likes ice-creams.
> It does not form a representational state prior to speaking which is the reason it speaks.
The whole thing is a representational state, and the fact that it's identical at the start of every conversation is a design choice, not inherent to the way the model learns.
> In other-words, it does not say, "I like ice-cream" because it likes ice-cream.
You also don't say "I like ice-cream" because you like ice cream. You say it because through years of evolution you've been taught to crave fat and sugar and other things in ice cream, which fuels your body, and helps you live and reproduce. The fact that this AI is optimizing for something else doesn't really prove anything.
I also don't think that we have anything more than a "statistically approximate model" of anything outside of our own thoughts. The theory of gravity exists because we've seen objects fall from heights a million times, we're told by others we speak to that they experience the same thing. From this we can say that with 99.9999...% (enough 9s to be indistinguishable from 100) probability, the theory of gravity is true.
This equivocates the sense of "because". When I say "I like ice-cream" I mean that I have an occurrent preference/desire for it. I am capable of desire, and the object of my desire here is "ice-cream".
Why I'm capable of desire has a lot to do with evolution, etc. yes. But that isn't relevant to the meaning of the claim.
> The theory of gravity exists because
No, this is humean nonesense. We are in the world; the world exists; it directly causes us to form beliefs about it. Those beliefs we, in general, arrive at via abduction.
Consider the fact that a child only needs to touch a fireplace once before concluding never to touch it again.
The heat caused an immediate, unambigious, representational state which imparts law-like consistency on the world which governs the child's future behaviour.
We do not live in a statistical inductive nightmare; AI does.
> Why I'm capable of desire has a lot to do with evolution, etc. yes. But that isn't relevant to the meaning of the claim.
If you are going to claim that your expression of desire is fundamentally different from another thing's, you need a reason, not just "because I mean it".
> Consider the fact that a child only needs to touch a fireplace once before concluding never to touch it again.
You can justify not touching the fireplace on a risk/reward, probabilistic basis. The child receives little reward for attempting to touch the fireplace, and lots of punishment. That alone is enough to dissuade you, or cause you to be more cautious in the future. People who receive actual rewards routinely do painful/dangerous things.
I would amend that there is valid reasoning other than empirical analysis, but also that chatgpt is capable of generating text demonstrating those types of reasoning.
I mean, ChatGPT is empirical analysis plus some rules tacked on afterword by openAI. With enough data and processing, empirical analysis becomes indistinguishable from other types of reasoning.
A weird thing about LLM is that certain internal structures (layers, activation functions) are much more effective than others. No one knows why. There’s a lot of work likely being done by it’s internal structure that we don’t really understand, which would explain why it is able to do more than just empirical analysis.
> It's been demonstrated to have the theory of mind of a 9 year old child, and this will no doubt improve
That's wrong. It's been demonstrated to produce responses to question such that if a 9 year old child would produce those same responses we would assume that that child had a typical 9 year-olds theory of mind. Not even for the child does "passing" such a test demonstrate having such theory of mind, though given our understanding of how humans develop it's a pretty good signal.
I don't know how someone could understand that and still think that chatgpt has demonstrated theory of mind.
It is, in fact, the same for a lot of tests. For example, it would be trivial to write a program that could pass an elementary student's math tests, but it would be rather ignorant to assume that a program passing that test was the same indication of that program's "understanding" of the underlying math that you would assume of a child who passed that same test.
When GPT mentions ice-cream, it does so because it was in the corpus. When it occurred in the corpus, it was as a reference
to actual ice-cream. So GPT has just as much intentionality as you do.
You might claim that you've eaten ice-cream, and that that makes a difference. But if we assume that your senses aren't lying about what your senses do, then what they do is produce information - indications of difference without any indication of what it's a difference of. That puts you in the same epistemic position GPT is in. GPT knows just as much about ice-cream as you do.
Let us construct IceCreamGPT. We take a corpus of text written by people who like ice cream and have provably demonstrated their joy while eating it. We then fine tune GPT 3.5 and the resulting model is called IceCreamGPT. Does IceCreamGPT like ice cream or is it only seemingly liking ice cream? It obviously likes ice cream, since it shares the same intentionality as humans responsible for the training data.
Now do the same with people who don't like ice cream but lie and write that they like ice cream. The performance of the second model is identical to the first model. Does this mean IceCreamGPT2 likes ice cream? Of course not, IceCreamGPT2 doesn't like ice cream despite it saying it likes ice cream! We know it doesn't like ice cream because it has the same intentionality as the humans responsible for its training data.
Now we have entered a magic world in which anything can mean anything.
No, this is just question-begging by treating GPT's access to the world as being external to it, but your own as being part of you.
If we fix this by treating your senses as external, then we can imagine a copy of you with its senses rewired so that artichokes* taste like icecream (and vice-versa). (plus we lie to you about which is which.) The resulting imtringued2 is identical to you, but doesn't like ice cream despite it saying it likes ice cream. Just like IceCreamGPT2.
Saying "I like ice-cream" has obvious conditions under which the speaker means it. ChatGPT cannot meet those conditions. It lacks the capacity to like, indeed, to intend to say anything.
ChatGPT cannot communicate. No act of text generation its engaged in counts as communication: it does not mean to say anything.
> (1) You realise that we don't investigate human understanding by looking at input-ouptut text correspondances? (This is quite a funny claim!)
Why is it funny? We most certainly do exactly that. In particular in this forum where we interact via text input-output, but also in real life when we ask each other questions, etc. Sure, there are also gestures and other channels of communication but text input/output is a big part.
If anything terrifies me about our current moment in AI, it's this impulse to dehumanize actual living people as a way of defending AI from entirely valid criticism.
Sure, people suck and lots of them are immoral and stupid... but they're still human beings who could have been otherwise (counterfactual thinking and all that) given a different "training set" (upbringing).
Unless you're implying that some percentage of human beings are incapable (like ChatGPT) of acquiring a morality or rational scientific understanding, no matter their training?
I find it extremely concerning that people are ready to believe, about themselves that the reason they say, "I love you!" is because they're just correlating patterns in their memories of every time someone mentioned those words.
It's this fetishization of the seemingly more "rational" position that these people seem to have. It's similarly mind-boggling when people have discussions about free-will and determinism. Free will is quite possibly the most fundamental experience humans have yet people will flatly deny free will exists.
What do you mean by free will? Yes, we make decisions. Yes, we have the experience of having will and choosing things.
But where does the “freedom” of that will come from? Everything we desire or have preferences for, and everything we decide is based upon some prior event or factor.
It’s an illusion of free will. People often get caught up on this because they feel it scary and dehumanizing to say “we don’t have free will”, but really, determinism or at least compatibilism make way more sense than some magic “free will” that just arises out of thin air.
Do you think your concern here is part of why you have the opinion about AI that you do?
I agree it’s alarming to think that we may be creatures that take in data, store it, map it , then predict / output ideas and actions from that. But emotions aside — what’s wrong with that?
We have the experience of love and other emotions. Why we have that experience doesn’t really matter, does it? If someone proved flat out that the reason I say “I love you” to my family is that I’m just correlating patterns and remembering past experiences — so what? It changes nothing about my experience of love.
I feel like many people are against these claims about AI and even current LLMs simply because they are worried it takes it away the magic of being human.
But I don’t think it takes away anything at all. If anything, it’s exciting to think about the potential similarities as it might just help us understand our own selves .
Why? Are we not allowed to find meaning, beauty, in patterns? The fact that two people can share experiences that bond them together, allow them to deeply trust each other, share a sense of humor, etc, isn't not beautiful just because there's some extremely complex statistical calculations going on under the surface that we ourselves can't explain.
It's like saying it's concerning that human beings are ready to believe they're made up of combinations of atoms.
What would be an appropriate explanation for the concept of love? Or should we just never go beyond magic?
The point is that love isn't just a linguistic phenomenon – it's some other internal phenomenon or experience that happens to cause linguistic expression (among other forms of expression or behavior modulation.)
Love is something we feel; words are evidence of it. We don't define love as a pattern that shows up in a series of conversations. There are many other ways of generating those same words.
Nobody's trying to reject materialism here. I myself happen to believe that love is nothing more than (or at least, can entirely be explained by) electrochemical processes in my brain. And I agree that that doesn't make love any less magical ^_^
I think by "correlating patterns", the person you're replying to meant "doing just enough processing to produce the salient performances". But it's possible to have additional "inner mental life" beyond the minimum that's necessary to give rise to an outward behavior. (In this case, the outward behavior is speech.) A single interface can have many different implementations, as it were.
> But it's possible to have additional "inner mental life" beyond the minimum that's necessary to give rise to an outward behavior.
NNs have inner layers, and some of the properties in those inner layers translate to real concepts. I understand that people assume computer=not-sentient, and I wouldn't classify chatGPT as achieving sentience yet obviously, but that doesn't mean the way it learns is fundamentally different from how sentient beings learn, or that if we scaled it up a lot, and gave it a more diverse set of data, it couldn't achieve something indistinguishable from consciousness.
It seems pretty well established to me that congenital conditions, non-congenital mental illness, or traumatic brain injury [1] can rob human beings of almost any of the characteristics (self-awareness, empathy, any stage of morality above Kohlberg 1, the ability to meaningfully consent to things, etc.) that distinguish them from ChatGPT without robbing them of the ability to communicate intelligibly.
This does not imply that it's OK to kill or harm or even be rude to them, and I don't think anybody pointing this out has suggested that it does. On the other hand it is OK to flip the off switch on ChatGPT. But that distinction is not based on meaningful distinctions in ability, and I would have said trying as hard as I interpret you to be to draw it based on abilities was the scary bit.
Something that I have only really discovered in AI related comments here on HN (and it terrifies me) is that a good number of humans have a deeply-felt need to dehumanize humans (pressumably also themselves).
Sure, a fraction is purely driven by career / monetary gain prospects. They adopt the mantle of the AI fanatic to prime themselves for the wrestling in the mud that they hope will get them closer to their goals. I understand that and I am not particularly worried about that. Its a devil we know.
But another fraction seems to actively enjoy the absurd suggestions advanced by the AI Jesuits.
If I’m understanding you, the problem may be much worse than you realize. “Homo Sapiens” the wise ones have defined ourselves as the intelligent beings. But with AI demonstrating an alien intelligence, humanness and intelligence become decoupled. We can start to see, though dimly, a world where the aspects that make us uniquely human are not related to intelligence but are instead related to biology , animality, sociology, and so forth.
A lot of people are going to be questioning what it means to be human.
seeming-to vs actual is a real distinction that can be made in some cases.
However, I don't think you get to make it unless you can define what 'actual' actually is. Saying that the AI can't interpret but only seem-to-have-interpreted is begging the question unless you can then go on and explain what interpret actually means.
It is precisely what is most interesting about the large language models that complex seeming-tos arise from a simple process done on a large scale. If you want to say that humans have access to a qualitatively different 'actual', then you should be able to explain what that difference is, because without that deep understanding of the phenomenon, it's perfectly reasonable to ask how we know that humans do the actual rather than the seeming-to.
Your number 3 is pretty surprising, given that explaining is one of few areas I'd say it's frequently superhuman in. It can also certainly generate imaginings with more fluidity than I can too.
Until we have a really good understanding of how humans do these things, we shouldn't spend any time pronouncing on how there's a huge difference between a capability as ChatGPT has it and as a human has it.
If you were in a saw-style escape room with chatgpt and you needed to construct 1 kg nitroglycerin from a large sample of various chemicals in order to get out.
1. Could you ask chatgpt (jailbroken ofc because "danger") whether it could safely describe how to do this and trust the reply?
2. Would you then trust chatgpt to "superhumanely" explain how to do this?
The actual capacities are listed: caring, imagining, and so on.
Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on.
There is far less mystery here than we suppose. Thousands of years of religion has forced this supposition that there's a "mind/body" problem, and characterising the minds of animals is such a difficult thing; it isnt.
> Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on.
And I take it that you're arguing that ChatGPT does not have a mental representation of its environment? Obviously, we could quibble about 'mental', but for all of its limitations and problems, ChatGPT is very little more than a condensed representation of its environment.
What about motivation and emotion? I don't think we have a clear enough definition of how these things are instantiated in humans to properly debate here. ChatGPT acts as if it has certain motivations (to be helpful, to avoid controversy), and has been coached to avoid expressions of emotion.
If you start from a view point of 'a next-word prediction engine obviously cannot have emotion, plans, or a world model', then of course you'll conclude that ChatGPT does not have these things and cannot. If you start from the view that we don't really know what mechanisms we need to instantiate these things anyway, then you're much more likely to look at the areas that ChatGPT seems to behave similarly to humans and wonder if big chunks of our brain are actually next token prediction engines, just structured by evolution to work in particular ways. This view was not new with ChatGPT of course, predictive coding has been an interesting approach to human cognition for a while now.
I thought this comment about what science is was also insightful because it explains simply Chomsky's problem with pseudoscientific approaches:
I believe your disagreements with Chomsky on politics does creep in to the discussion of AI and linguistics. Chomsky has never denied the success of ChaptGPT or google search or any of those things. The article of Norvig states this. But the core disagreement between Chomsky and his followers (such as myself) against those who prefer a statistical view of intelligence is that LLM or in fact the entire field of AI as it is right now has no real scientific basis compared to what exists in other scientific fields like physics or biology or chemistry. We don’t have a first principles of intelligence, hell we don’t even have bad theories of intelligence. Algorithms that form the basis of Google search or the statistical nature of LLM does not explain what is actually working in the real world. Just like the brain which doesn’t perform advanced calculus to calculate trajectories when it raises the hand of a body to grab a ball in the air.
Look back at the Norvig article until, we have made spectacular technological progress, basically engineering and that’s great as a tool to help society. Just don’t confuse that with actual science that is supposed to explain what is actually out there. That’s the true nature of Galileo’s telescope.
> 1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted.
This is important, and even here on HN people are frequently ascribing to ChatGPT various qualities (“understanding”, “learning”, etc.) that misunderstand what it is doing. ChatGPT can perform a trick: it produces conversations that have the texture of human ones, skipping entirely the step of understanding the underlying ideas. Obviously a lot of people see utility in this ability, but there are limits to what this approach can do.
I don't really agree. I wouldn't call NLP a trick, it's extremely useful. So what if chatGPT is just a chinese room? It's an effective tool and that's all it needs to be.
> ChatGPT does not think, know, care, intend, speak, commnuicate, etc.
These are statements of faith. They can't be proven or disproven by any experiment. "I think therefore I am." applies to just me. I'll never know whether you have intentionality or not. But I have faith that you do. Maybe you have faith that ChatGPT doesn't. I'm agnostic!
Chomsky's thinking is a form of vitalism: that biological matter possesses a special life force or energy that nonbiological matter doesn't. Well, vitalism eventually went out of style.
I find the morality point being interesting because it is a big deal in the Abrahamic religions that the original sin of discovering the difference between right and wrong is what made us human.
I think morality is one of the strongest wedge issues against this sort of pseudoscience which says: if it appears X, it is X.
ie., there are very few people who would make that claim about morality.
There are many who might think, "well if it appears to be playing chess, it is!" (i disagree).
But almost none who'd say, "well if it appears to defence murder, it must really think its a good idea!".
The latter is clearly absurd, and we've lots of reasons to suppose the system even lacks basic capacities to form any view, preference, intention (etc.) about murder.
Once you realise this, I think the house-of-cards falls; because it sets up the fatal gap between: what the output appears to be caused by vs. what it really is caused by.
As soon as you ask, "but what really caused the machine to generate that text?" the answer exposes the trick.
It’s connected with truth and logic because a correct moral decision involved getting the facts straight.
In our household (which runs a riding academy) we reject the Abrahamic position and say that morality is a function of mammals (and probably birds.). That’s based on observations of animals having strong emotional reactions to violations of social expectations.
The first time I got thrown off a horse, for instance, the horse was much more shook up than I was and acted contrite and extra-sweet afterwards.
There was one time I teased my cat by taking the side of the neighbors’ cat (with body language) in a territory dispute and my cat seemed highly offended at me which persisted for a few hours.
It can't be the case that the first sin is what made humans human, at least not for Christianity. Back up to 1:26-27 for what Christians take to be the thing that makes humans humans. They were fully human prior to eating of the tree.
But did they know some right from wrong before eating? Yes, and they are held accountable for exactly that when they eat anyway.
What exactly "knowledge of good and evil" entails, given that they are already morally accountable before that point, is an interesting topic to unpack. It must be closely related to an idolatrous belief that, not only am I in the likeness of God and so can discern right from wrong, but I am _so_ like him that I get to _decide_ what's right and wrong.
A helpful tip: whenever you need to decide whether to use one or the other, ask yourself whether you want to say "it is"; if so, use "it's"; otherwise, use "its".
It doesn't "know" that and this is a great example of how clearly it doesn't know things even when it can produce text that if a person produced it you'd assume indicates understanding. Ask it to explain when to use "a" vs "an" and you'll get a good explanation. Give it a test, though and it'll often fail. For example:
> me: Imagine that there's an animal called a "honbukle" pronounced like ending in "buckle" and starting with "on". Tell a short story about having seen one.
> chatgpt: <blah blah blah> "It sounds like you saw a honbukle," they said. <blah blah blah>
In fact, it will reliably fail such a test because it just reproduces the most likely thing from its training corpus with no understanding of the actual rules under which to use "a" vs "an".
I think that a LLM similar to chatgpt could be trained to take a moral stance, the system could be trained in a corpus in which the defense of moral values is explored deeply. Then we could test the premise of Chomsky about the amorality of LLM, it could be an interesting challenge and experience.
I think comparing Chomsky to Jesuit astronomers declining to look through Galileo’s telescope is an unfair analogy.
A better analogy in my opinion would be Einstein's attitude to quantum mechanics, he accepted that it had predictive power but was militantly in the camp that it was incomplete as a theory for various reason. Similar to how Chomsky thinks that deep learning is effective and interesting as an engineering concept to generate language but sheds relatively very little light on how human intelligence and language works on a deeper level.
Not only the parallels between Einstein's and Chomsky's objections, but the "shut up and calculate" historical camp in quantum mechanics bears some resemblance to perhaps Norvig's arguments for statistical, neural network-based models of computation.
ha ha ... every human is a fool in that case (as every human holds some unreasonable believes and opinions some point in time).
"He has discredited his mind?" This sentence does not make sense to me. Explain what you mean by this statement.
I found his NYT piece well argued. How can you fix any language model similar to ChatGPT to prevent the mistakes shown in the article? (I don't think you can).
He makes irrelevant and baseless claims. First we don't really know how the mind works and therefore we really have no idea if the language models are path toward AGI or not. He really doesn't have an idea how a child obtains a language. You really need a lot of data to acquire grammar, no minuscule data is not enough. I have observed it personally with my multilingual children, their mother tongue, to what they have the least exposure and what is the most complex, is not still not fluent for them. His claims are simply false, easily refuted by empirical data.
ok, you didn't answer my first questions. I try a couple of more.
your first argument is useless, as anything might be a path to AGI ... first order logic / expert systems are ...
regarding empirical data, you are bringing up anecdotes that don't hold up. Humans can often learn a concept from 1-2 instances (take the face of Albert Einstein) (even your kids can do that in grammar :) , maybe check how language models are trained in comparison.
do you have any references for your claims? How much data does a human need to learn grammar? do you have any estimate?
you say you can refute Chomsky's claims by empirical data. please show me that data/studies, don't talk about anecdotes from your kids.
"His claims are simply false, easily refuted by empirical data."
he underlines his claims with examples showing what he means. I recommend you to read the original article by Chomsky, as you seem to have a problem understanding his arguments.
The scientific community at the time had good reason to resist looking through telescopes. The entire scientific edifice was built on 7 planets going back thousands of years. What would happen to their medical "sciences", for example, if planet #8 showed up? Also note the church had already accepted Coppernicus's findings earlier. Issue was never the caricature of "bible says" that is claimed. In fact it was Babylonian roots of "sciences" -- everything around number 7 and 7 planets -- that caused a foundational problem. They had to rebuild sciences from the ground up. That was the reason for "jesuit" resistance.
Of course they did. There is no reason to assume people were stupid [in the past] and just got smart overnight. These were smart people struggling with a paradigm shift that would upset almost all their theoretical apple carts. It was an epistomelogical crisis but not of the spiritual flavor as it has been alleged.
"In this piece Chomsky, the intellectual godfather of an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded."
That's incorrect on a few levels. Aaronson is talking about something else, different from what Chomsky worked on. ChatGPT can put together a statistically likely series of tokens, but ChatGPT doesn't understand the meaning of those tokens, and therefore ChatGPT has no concept of "truth." ChatGPT cannot deliberately lie or tell the truth, it has no understanding of such things. By contrast, Chomsky, for much of his career, was a linguist and he focused on the issues of how is knowledge constructed, and how do we know what truth is, and how does language express this? So Chomsky is pointing out that ChatGPT creates a fascinating illusion of a real conversation, but it isn't exactly what AI researchers were aiming for, for several decades. And it is appropriate that the New York Times would want to publish an essay that speaks clearly about what ChatGPT is and is not, because otherwise there is a risk that the general public will get an over-hyped view of what ChatGPT does.
How sure are you that humans understand the meaning of tokens, have concept of truth, or understanding? Ignore your subjective experience of consciousness and ask, "what scientific experiment could I run that would distinguish the special nature of humans from sufficiently magical AI?"
I agree. It seems to lack self awareness, considering how some weeks ago there was a post on his blog about experts in unrelated fields who claimed a wormhole simulation was the creation of an actual wormhole. Aaronson said that believing these claims is an example of Gell-Mann amnesia. Why does Aaronson think he's suddenly an expert in AI, language and consciousness? Being correct in quantum computing topics doesn't translate to this field at all. And worse than that, there are no real arguments here, just misrepresentation of Chomsky and ad hominems.
> ChatGPT doesn't understand the meaning of those tokens, and therefore ChatGPT has no concept of "truth." ChatGPT cannot deliberately lie or tell the truth, it has no understanding of such things
It does encode meaning, but just knowing things doesn't give you the ability to know if they are true. A human brain in a jar that was only fed text from the internet also wouldn't be able to verify what it learned against reality.
Chomsky has the flawed conception of language. It is just one tool (though a very powerful one) that humans use when reasoning.
Language models don't encode meaning. They "encode" the probability of collocations between lexical tokens. Such collocations are correlated with meaning, but they do not "encode" meaning. We don't know how collocation of tokens is related with meaning, we don't understand why certain words, in a certain order, mean a particular thing and not another, and so we don't know how to model that relation.
For example, suppose I give you a dictionary of a language you don't know. Could you learn the meaning of words in the language, by reading their lemmas in the dictionary? Note that words in a dictionary are collocated with explanations of them, not just other words, or tokens.
I'm Greek, so suppose you didn't speak Greek and I gave you a dictionary of Greek, where you could find the following:
Can you tell what the word "Αβαρκαμαξας" means, just by looking at the dictionary entry above?
Obviously you can't (and you can't translate it in Google Translate either because it, and its explanation, are completely made up and not even real Greek). In the same way, just matching tokens with each other based on where they appear next to each other in text doesn't say anything about their meaning.
We don't know how to encode meaning. Statistical language modelling is exactly an attempt to represent language, without having to tackle the problem of representing meaning.
Me::) given the dictionary entry Αβαρκαμαξας: ιμιά καρός μαύρα φούμαρα σαχλαπαράκατα μαρακαφούλες, and you tell me what the word "Αβαρκαμαξας" means? it's ok if you dont know
ChatGPT:
As an AI language model, I can recognize that the word "Αβαρκαμαξας" is of Greek origin, but the dictionary entry you provided does not seem to provide a clear definition of the word. It appears to be a descriptive phrase, possibly describing the appearance or characteristics of something, but without further context, it's difficult to determine the exact meaning.
Meaning is entirely statistical! That's why languages evolve, and new words and phrases emerge and disappear over time.
Maybe I misunderstood your comment? Also I may have been unclear in my reply.
I understood your specific Greek dictionary example as being something that does not encode any meaning, because the words are not real Greek. However, completely hypothetically, if "Αβαρκαμαξας" was a word that a whole bunch of Greek people started repeatedly using in the same or similar contexts, it would quite literally become a real word.
That is what I mean when I say meaning is entirely statistical. Words and phrases emerge, disappear, and change meaning over time. These changes often happen differently within different segments of populations that speak the same language. What words mean depends entirely on enough people agreeing to use the words the same way. LLMs capture the commonly used modern meanings of words extremely accurately.
edit: That's also how ChatGPT was able to understand the commonly made typo in the question I asked it. Statistically, I probably meant to say "can you tell me what the word [...] means".
What I meant with the example I gave was that you can't figure out what a word means just by looking at what other words it's close to, unless you know what all those words mean, already. That is relevant to my comment above where I wrote:
>> Language models don't encode meaning. They "encode" the probability of collocations between lexical tokens. Such collocations are correlated with meaning, but they do not "encode" meaning.
So I gave an example of some words, that I suggested were collocated, to demonstrate that you can't tell what they mean just by looking at them next to each other.
That the words don't really exist is something that you wouldn't know unless I had told you, or unless you had access to a dictionary. Or, I guess, a Greek speaker. I told you that the words are not real Greek not to make a point, but to avoid being disingenuous [1].
The point of all this jumping around is that if you don't know what the words mean, you won't know what the words mean, no matter how many statistics you take on a corpus, no matter how large.
This is actually related to your point. I don't think that "language is statistical". I do agree that language, and the meaning of words and sentences, changes constantly. But, that's one more reason why language models are such a poor, well, model of language: in ten years from now, there will be many expressions used in everyday language that ChatGPT will not be able to recognise, because they would not have been in its training corpus. What kind of model of language is one that's frozen in time, and can't keep up with the vicissitudes of language as it evolves spoken by its speakers over time?
A very poor model of language, indeed.
______________
[1] You put the words in ChatGPT and it told you they mean nothing. That's not entirely true. There is a pair of words in the explanation part of my sentence: "μαύρα φούμαρα", which are real Greek words and that form a meaningful Greek phrase. So I was disingenuous, a little bit, or depending on how you see it: I said that the "explanation" is not real Greek, not all its words. In my defense, I was sure you'd put the phrase in Google Translate, or, if I told you it means nothing, to ChatGPT. So I wanted to double-blind this a bit. Apologies for leading you down the garden path. But, look at all the flowers!
In any case, ChatGPT didn't recognise these two words and the phrase they form and gave you a generic answer, riffing off the information you gave it that the sentence was a "dictionary entry". So you told it it's a dictionary entry and it said it's a dictionary entry and it doesn't know what it means. That doesn't mean that it knows that it doesn't know what it means. It just says the most likely thing, which just happens to be "it's all Greek to me", basically. It could just as well have made up some meaning, because meaning means nothing to it and all the words are the same to it, except for their location in its model.
Come to think of it, I suspect the prompt you gave ChatGPT, and particularly the bit about "it's OK if you don't know", is a strong bias that leads to it saying "I don't know", rather than pretending it knows. Can you try again, without that part?
And yet, what if you had an entire internet of ungreek like that and spent the equivalent of many human lifetimes reading through it? You would encode your own meaning, just like a large language model does internally. Your encoded meaning might not be quite right in the end, but as the saying goes, all models are wrong, but some are useful.
How would you encode meaning by reading a dictionary whose language you don't know? Can you describe it, step-by-step like?
I mean, I can imagine opening my hands and levitating but I don't know how to do that in practice. So how would someone encode meaning by reading a dictionary in a language they don't know? In practice?
And what is the difference between a dictionary in a language you don't know and a corpus of text in a language you don't know? It's still a bunch of unknown words. How are you going to find out their meanings just by looking at them?
Step-by-step please, and show your work. Like ChatGPT.
Watching commentators from MSNBC to Fox, from pro-Russian to pro-Ukrainian, from pro-Israel to anti-Israel and so on and on and on, I don't think we humans have a good understanding of truth either.
In fact I read Chomsky's comments in regards to Serbia and Russia, and I think truth is in the eye of the beholder.
> therefore ChatGPT has no concept of "truth." ChatGPT cannot deliberately lie or tell the truth, it has no understanding of such things
Please see my comment below[1], this is not a fair characterization. ChatGPT is not a pure predictive LLM, it also undergoes RLHF which gives it a much more human-like character, including clearly having a concept of truth within its limits.
Indeed. I would say a better criticism wouldn't be that it has no notion of truth, but that it has no ability to reason on the basis of a reinterpretation of what might or should constitute truth.
Even if you come up with a fancy prompt that asks it to "assume" certain things, it will still have to base that assumption on things that it already 'knows' to be true, as it already perceives them to be true.
This is not a response to the Chomsky piece. The main argument advanced by Chomsky et al is that LLMs are neither AGIs nor are they precursors to what we might consider AGIs, because, among other reasons, LLMs "learn" differently to how humans do, and that difference comes with strict limitations on the upper bounds of what LLMs can achieve. I'm certainly no expert on linguistics or AI/ML, so I don't know about all that, but this blog post avoids engaging with that claim, and opts instead for ad hominem.
The Chomsky article does argue that LLMs are not AGI or AGI precursors: “These programs have been hailed as the first glimmers on the horizon of artificial general intelligence … That day may come, but its dawn is not yet breaking, contrary to what can be read in hyperbolic headlines and reckoned by injudicious investments”. You correctly point out that the counter-article does not respond to those arguments.
But there are many other claims and many other arguments the Chomsky piece makes: “we fear that the most popular and fashionable strain of A.I. — machine learning — will degrade our science and debase our ethics by incorporating into our technology a fundamentally flawed conception of language and knowledge”. The Aaronson article is responding to those claims, even if the response is “I disagree; more details to follow”.
Separately, it’s rather audacious to dismiss this response as ad hominem, considering the tone of what it’s responding to.
Aaronson’s ad hominems: “the intellectual godfather of an effort that failed for 60 years”, “what Chomsky and his followers are ultimately angry at is reality itself”.
Chomsky’s ad hominems: “ineradicable defects”, “lumbering statistical engine”, “stuck in a prehuman or nonhuman phase of cognitive evolution”, “the predictions of machine learning systems will always be superficial and dubious”, “pseudoscience”, “ChatGPT exhibits something like the banality of evil”, “the amorality, faux science and linguistic incompetence of these systems”.
None of Chomsky’s statements you quote are ad hominems. They’re not even misidentified ad hominem fallacy fallacies (which is what Aaronson’s are, actually). I’m not sure you know what an ad hominem is.
Ignoring your ad hominem attack, ad hominem means attacking the speaker rather than the argument, which is what Aaronson does here, and Chomsky does not.
>> “the intellectual godfather of an effort that failed for 60 years”,
Here's the entire comment:
>> In this piece Chomsky, the intellectual godfather god of an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
Not only is that a personal attack, it's a completely absurd fabriaction: Chosmky did not work for 60 years on creating chatbots ("machines that can converse in ordinary language").
I see that Aaronson added a comment to his blog to stop people calling him out for that. But the comment doubles down on the absurdity and says that Chomsky is "regarded as the highest authority" by some vaguely defined faction of "anti-statistical" someone or other.
Like many people who have only recently heard about AI, Aaronson has heard of things he has only partly-digested, like for example that there are two "camps" in AI, but of course he has a very superficial understanding of what that means. The "old guard" of AI researchers, as he calls them elsewhere in his post were always polymaths with contributions in many fields.
Take Claude Shannon for example (oh, you didn't know? Shannon was at Dartmouth and one of the people who invented the name "Artificial Intelligence" for the workshop there). He invented logic gates, and information theory. There is no separation like the one that Aaronson is trying to make, it's only in his head and in the heads of people who treat the whole AI debate as a football game, and just want to be cheerleaders for the home team.
And he still hasn't removed that embarrassing attack from his article. Well I hope he goes down as the guy who thought Chomsky worked on chatbots. If that's the hill he really wants to die on...
Loads of people have intimated that ChatGPT shows we're on the precipice of a breakthrough in that direction. Chomsky says we're not. If Chomsky's argument is a strawman, Aaronson could write a blog post saying that, instead of a blog post stuffing his own strawman. Anyways, I'm summarizing Chomsky's argument as I understood it, you can read it yourself.
Blake Lemoine was the dude claiming Google’s LLM had a soul and got fired and all sorts of press attention.
It’s not a straw man. Blake was an outlier but also very much on the inside and not just some random wonk. There are plenty of people who are being led to believe that LLMs are much more than fancy statistics.
However, to be fair to Lemoine, we don't know whether Lamda is just an LLM like GPT or something else, it might have been a different architecture, which actually could learn from human conversations, and have experience, thus gain consciousness.
Do they? Personally I can't rule out that of LLM model was trained on all of the language a single human heard/read and produced it wouldn't be able to create next utterance that might be indistinguishable from what that human says.
The simple fact that a LLM is trained on a gigantic corpus of data and humans learn from a relatively tiny number of interactions with other humans shows that they obviously learn differently.
If we took all of the data that a human takes in from all of their senses, I'm not sure if humans use less data.
Humans take in 10 million bits[1] from their eyes every second. 10,000,000 bits/sec * 60 secs/min * 60 mins/hour * 24 hours/day * 1000 days = 108 terabytes. ChatGPT only used 570 GB of training data, so 2 orders of magnitude less data, and that's only counting the visual data.
edit: And that would be for a 3 year old, so comparing ChatGPT's intelligence to a 3 year old shows that ChatGPT comes out favourably.
Yes, that's granted. The issue is that indistinguishable isnt good enough.
This is the core problem with this schematised (and i think, pseudoscientific) computer science approach to intelligence. Output isnt intelligent.
So, for any given output, it could have been created by system A or system B, whose properties could be radically different.
It matters why, eg., we get "I hate the rain!" as output. If system-A says it because it: cares, hates, muses, imagines, prefers, intends... then that's radically different than if B does so because, "it's combining a weather API with some internet chat history".
I think “indistinguishable” is good enough. As ML generated artifacts become more common seeing flavors of what they generate and common failures will become gradually more obvious. It will keep happening that a new technology will seem very impressive and then after a while the cracks will appear and we’ll all have our sort of internal turing test that separates human from machine.
If a digital thermometer reads 100C, connected to a black box, are we thereby required to believe that there's boiling water inside the box?
Science doesn't deal with the "indistinguishable". We cannot, on earth, simply distinguish between whether we go around the sun, or the sun goes around the earth.
Does the solar system have a soul?
The world exists, and it has properties, and those are independent of how dumb apes happen to be and what we are in a position to "distinguish" or otherwise.
A system generating text is acting as-if its having its intelligence measured. Each sentence we take to be a symptom of its: having a theory of the enviornment, having something to say about it, having some intention, etc.
When I say, "I don't like what you're wearing!" that sentence itself isnt somehow "intelligent". It is only a valid measure of my caring, preferring, speaking, intending, thinking... becausethat is why i said it.
A shredder which happened to assemble those words is likewise not intelligent.
This is basic science: measurements arent objects; and measurements have validity criteria which is, at least, the causal properties of the system give rise to those measures.
In the case of ChatGPT no relevant properties give rise to its ouptut. Its sentences are not caused by any intelligence, and aren't valid measures of it.
There is no boiling water. Your digital thermometer is broken.
I agree it's a red herring to focus on output and interactive behavior when discussing this.
If a "shadow prompt" told chatGPT that it writes at a 3rd grade level, we wouldn't argue as much over how smart the bot is.
If it omitted the friendly/helpful/deferential assistant stuff, we'd also argue about it less. Bing's initial defensiveness and aggression made it seem even stupider than the mistakes it was making.
They're honing in on better prompts and other configuration that will make the bot seem smarter. It seems smarter to say "I can't answer that question" than to confidently say something untruthful.
But the underlying computational program (GPT trained on the internet) is the same. If we judge the program's intelligence based on its output, it isn't well defined. The same thing looks intelligent or hilariously unintelligent based on the tokens you (an intelligent person) provide it with.
Or in other words... Suppose we collect all of the system's "intelligent" outputs and disregard the rest. We throw away a lot, the majority of responses, and the resulting set looks impressively smart.
The system appears to demonstrate advanced machine intelligence when restricted to (some?) preimages of this set, even though it acts like a total idiot over other parts of the domain. And it's clear that it takes real knowledge and understanding to solve this boundary problem, so that the calculated image has an "intelligent" shape.
Actually we can determine if the Sun goes around the Earth or the other way around - if we can create an better, more accurate model that have larger predictive power then we can assume this model to be more likely to be correct.
As I understand, this was one initially of the main issues with the new model proposed by Copernicus - it was not more accurate initially.
I don't know, that's Chomsky's claim. If I, again a non-expert, had to take a position on it, it seems far more likely to be true than not. Humans have access to a wide variety of non-language stimuli, and demonstrate signs of intelligence well before they have any functional mastery of language. Even after I "mastered" language, I developed lots of skills that haven't the faintest relation to language, like riding a bike. I'm sure ChatGPT can produce a textual explanation of riding a bike, but neither ChatGPT nor a human who doesn't know how to ride a bike can convert that textual explanation into the act of riding an actual bike. But a human, unlike ChatGPT, could, given a bike, learn to ride it by trying to ride it.
I’d look though at systems like CLIP and Stable Diffusion that are able to map between the language domain and images, as well as music, speech, etc. “Riding a bike” can be seen as a sequence modeling problem too because it is a matter of firing muscle fibers in a certain way and it is a research area to make language-controlled robots that do just that.
I guess the idea is that if I described a static process to an AI, like multiplication, as we have it wired up right now, it wouldn't be able to remember what I told it for years.
I agree this is true, and that it will be a breakthrough for AI, but it's entirely unclear how far away it is in the time dimension.
In humans this is the process of taking short term memory and converting it to long term memory involves a process called consolidation where the structure of the physical brain changes, I guess this would be tantamount to a reweighting of the neural net. It's generally not a one shot thing, especially as the concepts get more complex and have more parts to learn.
One of the things humans do is forget a lot of unimportant crap so we're not constantly rewriting our brains. Of course there is a the issue of how do we make sure we're training our AI how to learn multiplication and not feeding it a diet of junk food information/fake news too.
ChatGPT is a massive accomplishment but in my personal opinion it feels a lot closer to a search engine with a better search interface and missing data attribution. I've been using ChatGPT in my daily life for months at this point, I use it as a software engineer, particularly when I'm exploring a new programming language and looking for idiomatic ways to express myself. 'What's the Rust way to do X' is something that's all over my ChatGPT history at this point. It's a great way to get help when you're completely lost in a new problem domain.
However, my issue is that more than once, ChatGPT has suggested code to me that could likely introduce a security issue in a product had someone been dumb enough to blindly copy and paste. If this were a code snippet on a website, I'd probably drop a comment or just remember to ignore whatever I found on that website in the future.
So I get why Chomsky isn't impressed, once accuracy really matters ChatGPT falls on its face. When it comes to generating fluff like marketing copy, jokes and other things that don't require any degree of accuracy, it's amazingly good.
I tried to use ChatGPT to explore parser library for Rust called nom.
Nothing worked. Syntax was wrong. Midway I resorted to googling and after a short while I learned why. The library had a major overhaul at one point. They switched from using macros to using functions. ChatGPT didn't notice that and served what was mixture of old and new syntax.
Yep, this type of thing happens all the time in ChatGPT. I've more than once had ChatGPT suggest that I use something deprecated and sometimes if you poke it and tell it it'll figure it out but I've seen it make the same mistakes over and over.
I was working with JWT tokens in GoLang and the library I was using didn't validate expiration by default. I asked ChatGPT before I dug into the library code and it suggested a solution that that bypassed the built in validation of the JWT library I was using and introduced a fairly big security issue.
That was the moment I got a little nervous about folks using ChatGPT.
I wonder if the documentation here wasn't clear on versioning? In some ways this sounds kind of human in the sense you're trying to fix a problem looking at old information and things have changed but if there isn't a clear break in messaging it can lead to confusion.
I think ChatGPT draws knowledge more from web tutorials than documentation. Old syntax is really prevalent in those. And pieces written in old syntax don't state it's old.
The irony of it is that Chomsky’s ideas are the foundation of parsers for programming languages. That is, you can make a link between the semantics of arithmetic and logic and the kind of grammar Chomsky talks about and you have… C, Pascal, Python, etc.
And people understand it!
The semantics of natural languages is over the experience of an animal and if you think of it as a “language instinct”, that instinct is a peripheral of a animal’s brain which has considerable capability for cognition (particularly in the case of mammals and birds) without language.
From that point of view natural language competence is a cherry on top of animal intelligence and you can’t simulate a language-using animal without simulating the rest of the animal.
ChatGPT does a surprisingly good job of faking linguistic competence with nothing but language so it certainly looks like a challenge to the Chomskyian point of view but I’d remind you that animals, pre-linguistic children, aphasics are all capable of cognition without language so “language is all you need” is still an incomplete position. (But boy there is that strange fact that transformers work for vision although research may be showing that is accidental?)
What does it mean for the mind?
A major part of animal behavior involves sequences. Think of tying a knot or a bird migrating, or for that matter serializing and deserializing mental structures into words. In the 1980s I read books about the brain that were lost at sea about how a neural network could model sequences and artificial neural networks have come a long way since then.
>aphasics are all capable of cognition without language so “language is all you need” is still an incomplete position. (But boy there is that strange fact that transformers work for vision although research may be showing that is accidental?)
This is one of these places I hope we aren't trapping ourselves in our own thinking like we did with flying. "It's only flying if you flap your wings" could be thought of one of those trapping in 1900, when it turns out that is but one way of flying, and throwing lots of power at rotating blades is another.
We may be able to create 'intelligence' from text alone, but I do think that it will need input/output methods of divining that truth, hence simulating the IO functions of the animal, whatever that may look like in the end.
And I’d say there is more than one kind of ‘embodiment’. It sounds trivial to map the word ‘red’ to an area of the RGB color space but if you ask a robot to give you the red ball and you get the red ball didn’t you ground ‘red’?
I think the image-language models put language in confrontation with something else and ground it in some respect, you could even say HFRL training ChatGPT causes it to move towards pleasure and away from pain and that’s a very simple kind of experience. For that matter just turning text into database records that can be processed by ordinary software is a useful form of grounding, as would be asking for something and getting it in the mail.
‘Language is all you need’ is a more credible position than it was a year ago.
Just reading another article on HN where Microsoft is saying GPT-4 is being trained in multimodal mode, so I'm not sure if we're going to have much time to talk about text only models...
I didn't think Chomsky's NYT piece was very strongly argued (it seems a mish-mash of different points, some somewhat convincing and others quite unconvincing), but this is at least as bad.
It starts off with a complete mischaracterization of Chomsky's research program (to imply that he is a disgruntled sore loser in the race to build machine intelligence) and then continues with a bogus analogy to imply Chomsky is a kind of deluded religious fanatic.
Then it concludes with a note complaining that people don't trust the objectivity of the author's opinions now that he's employed by the entity he's defending - well, yes, of course they don't, that's the compromise you made when you decided to take Open AI's money.
Quantum talk, so, I guess he’s just promising to provide the smallest possible unit of talk. Quite humble actually. I wonder what he’ll go with. I bet:
>>> I submit that, like the Jesuit astronomers declining to look through Galileo’s telescope
Completely as an aside, from what I've read, the Catholic astronomers did look through Galileo's scope, and confirmed his observational evidence.
They were willing to accept a model of Tycho Brahe, in which the sun goes around the earth and the planets go around the sun.
Not needing to look through the scope, to know the truth, was introduced as a hypothetical in arguments. And there were Churchmen who were using the controversy to push the Church towards what we might call a more "conservative" position today. But amongst intellectuals, the debate was over whether the evidence supported the heliocentric theory or not.
> In this piece Chomsky, the intellectual godfather of an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
Well, Aaronson may not be the godfather, but he's definitely an influential figure in effort that failed for 45 years to develop magic machines.
Sure but if a competing team built a quantum computer that could factorise huge numbers Aaronson would presumably not write a piece in the NYT about how it didn't actually count because <insert vague objection here>.
There is a very important distinction between the two cases. D-Wave have never been able to demonstrate a way in which their technology is actually useful whereas (for better or worse) people are actually using ChatGPT for their jobs/lives and claim it is valuable to them.
I legitimately scrolled to the top of the post and scrolled back down because I assumed I had missed most of it. The author lists four of Chomsky's points, rebuts only the fourth by saying it was intentional, then insults Chomsky with a vague analogy that could apply to anyone engaged in a scientific debate. Regardless of my stance on the issues this post doesn't seem that good to me.
Scott didn't understand Noam's complaints very well. He (and I [0], and many others) are pointing out that LLMs cannot think or perform reasoning or exhibit intelligence. Not now, and not ever, because statistical counting of likely words is not intelligence. Lacking the ability to explain the fundamental reasoning concepts behind one's conclusions is a hallmark of machine "learning," which is why it isn't learning. It's word aggregation.
The fact that ChatGPT cannot tell if anything is real or correct or not is not a small thing that one can hand-wave away and say, "Well, maybe tomorrow." The inability to discern truth is built into the LLM method. This is not true of people. Certainly not intelligent ones.
ChatGPT cannot even correctly perform math or logic, and those are fundamental to intelligence and intellectual development. Hilariously, we already have many programming languages which can do those things, so that's not much of an advance to rely on software that cannot.
Aaronson's premise, that Noam Chomsky has nothing more than "sour grapes" to offer, is really puerile and uninformed. See Chomsky's original article [1] for yourself and then decide who is on the hype bandwagon and who is not.
It seems to me that much of both the hype and the hate around ChatGPT is a gut reaction from people who haven't tried to incorporate it in their workflow for at least a month, understand its limitations and strengths, and only then comment on its ineptitude or world changing abilities.
The fact that it can instantly reply with a broad consensus of the ideas it has been fed is precisely what I need from it. When I ask it to write a function that performs a binary search in Rust with a given set of constraints, I'm not looking into original research or "truth", I want precisely the boring boilerplate I'm too lazy to write. For that use-case, it's invaluable and awe inspiring.
It's limits are laughable too, try as I might I could not make it count the numbers in a string of '9's that fit in a 128bit integer. It hallucinated fictional names of tools that I should use, such as "cobol_to_rust".
It seems clear to me that both camps have profound insights and LLM are both invaluable and wholly insufficient to reach any form of substantial "Artificial Intelligence".
I asked ChatGPT to assume "word" is a number, ignore numeric digits (ie 0-9) and assume the characters in "word" are digits. Also assume "word" contains the largest possible digit. What would it conclude the base to be.
Well...ChatGPT has been struggling. It generally struggles with anything subtle, no matter how simple. If it can't regurgitate based on what's slurped up from the internet it struggles.
Do note, my question is simply meant to assume i am using a-w as my digits and "word" is a number formed via a positional number system. It is something a human with requisite knowledge will veer towards, ChatGPT doesn't.
To your point, it is indeed an awe-inspiring tech for the thing it is good at, ie replacing the current crop of search engines.
>> I asked ChatGPT to assume "word" is a number, ignore numeric digits (ie 0-9) and assume the characters in "word" are digits. Also assume "word" contains the largest possible digit. What would it conclude the base to be.
For your intended use-case (as explained in your comment) shouldn't you be phrasing the second part of this specification as "treat the characters in "word" as if they are digits"?
The way you worded it introduces some ambiguity. The ambiguity can be resolved by a process of elimination - I had to think of what could possibly happen if I "assumed the characters in 'word' are digits" and realise that it doesn't make sense, so it must mean you want to use the characters as digits.
ChatGPT and LLMs in general really can't be expected to deal with that kind of ambiguity, not least because it's not the entire story (you also need another little logical jump, from "treat characters as digits" to "treat characters as denoting their positional values in the alphabet" or some such).
I'm saying, you're asking ChatGPT to take many reasoning steps, when everyone is having a big debate about whether ChatGPT and LLMs can reason _at all_. To make any heads or tails of ChatGPT's abilities, one needs to start with simple examples, and then get to the more complex stuff.
On the other hand, I expect that if you clarified your question more, or perhaps stated it just the right way, there's a good chance you'd get a correct answer. After all, the idea of counting with characters is very common and it's hard to believe ChatGPT hasn't seen a few thousands of examples of it during training.
To be fair, the way inputs are tokenised does make per character operations much harder than you would expect. If OpenAI had instead made one character one token, then it would be worse in most senses (slower and less history) but would do much better at that kind of task.
People do a terrible job of binary search to the extent that it is towards the back of my toolbox. You can get eaten alive by off-by-one errors, mishandled integer overflows, etc. Human-written binary searches work most of the time but often fail for special inputs. I wouldn’t trust a machine that learned by reading 1000s of other people’s binary searches any more than I’d trust one I found off StackOverflow unless it came with a detailed explanation of the corner cases…. It is one to look up in an algorithms book, a recent algorithms book.
> The fact that ChatGPT cannot tell if anything is real or correct or not is not a small thing that one can hand-wave away and say, "Well, maybe tomorrow." The inability to discern truth is built into the LLM method. This is not true of people. Certainly not intelligent ones.
To be fair, ChatGPT is not a pure predictive LLM. It also undergoes RLHF, a reinfocement learning step which is much more similar to human learning, and allows it to develop a more human-like character, and seems to give it a pretty good grasp on truth(telling), ethics and context, certainly compared to the behavior of a raw predictive LLM with no prompt engineering.
I am way, way, way far from an expert on AI. Not even particularly interested in it. But is it really fair to say these models just give you the next likeliest word based on training? If that were true shouldn't you be able to replicate what they can do with a really advanced Markov chain?
> The fact that ChatGPT cannot tell if anything is real or correct or not is not a small thing that one can hand-wave away
I don't disagree with your premise. But this applies equally to human intelligence. There's no way to ever tell if the reality you experience is real or not. This is a fundamental fact of existence that we hand wave away so that we can continue to interact with and learn about the world. My favorite thought experiment on this is Boltzmann Brains.
It reads less like a fully thought out competing position article, and more like an off the cuff thing that someone might throw together on his coffee break because a friend asked him about it. Maybe he forgot that he was posting it to his personal blog and put as much thought into as one might put into, say, a HackerNews comment.
Ahhhh, okay that explains it. I always liked reading Scott's articles on quantum computing, but this was a highly off-putting piece: riddle with mistakes and misrepresentations, and condescending to boot.
That is ultimately the sad nature of this debate: exaggerating the capabilities of certain algorithms and diminishing the capabilities of the human intellect, for... monetary gain
It is deeply immoral and it will rekindle the disturbing image of evil technologists and mad, power hungry scientists at a time where we need as much trust and confidence in the use of technology and science to absolve us of problems we ourselves have created.
Really, the jesuits did use telescopes. One wonders at the ignorance that paragraph shows. Come on: Galileo was punished for purely political reasons, with the religious argument as a simple excuse. This is not something obscure.
>I’m a CS professor at UT Austin, on leave for one year to work at OpenAI on the theoretical foundations of AI safety. I accepted OpenAI’s offer in part because I already held the views here, or something close to them
These are fair criticisms but imo miss Chomsky's much bigger point.
From Chomsky's essay, 2 important lines:
"Whereas humans are limited in the kinds of explanations we can rationally conjecture, machine learning systems can learn both that the earth is flat and that the earth is round. They trade merely in probabilities that change over time.
For this reason, the predictions of machine learning systems will always be superficial and dubious."
He's saying we trust humans because they can say things like "I'm pretty sure X because of explanation Y" and under the hood, we process the explanation, form our own probability of X and trust our own computation.
But since LLMs cannot provide explanations for their beliefs, humans will never be able to rely on LLMs because the way we actually communicate is through explanations, not probabilities.
Chomsky's conclusion: this is a good predictor, but not a human.
Where Chomsky is actually wrong is he mixes up how good are you at prediction vs. how well can you convince a human of the prediction vs. how intelligent you are. We humans use a combination of accuracy + convincingness as a marker of intelligence.
An AI that was just as accurate but 0 ability to convince a human because 0 ability to produce explanations could still be intelligent. This AI would seem like an alien to us, but an alien that could uncannily beat us at any challenge that requires an understanding of the natural world. In fact, we might never truly understand its internal explanations of the world, but still acknowledge it has them and is good at building new ones internally.
Imagine being in a room with a foreign language speaker who beats you at chess. Clearly they have a mental model that works, even if they cannot explain it to you.
To my mind Chomsky and Aaronson are talking about different things. Chomski says that LLMs are not AGI and they will never be it. Aaronson says LLMs may be a way to AGI.
This ideas do not contradict each other. Chomsky talks that LLMs is not sufficient to create AGI, Aaronson says that LLMs is essential to create AGI. A cause can be essential but not sufficient, nothing wrong with it.
Fundementally Chomsky has always argued that language must have some sort of core logic beyond Saussure's division between Signifier and Signified, but so far nobody has been able to prove it. What might be most disturbing to him is that LLMs, which operate purely on mass associations, might be closer to human cognition than a notion of cognitive logic. What if the associations are more primary, and the logic comes later? Just as someone else commented, animals work mostly in cycles, patterns, they don't have logic but the cycles of life are burned into them; there could be some magic key that explains why humans possess logic, but it would not be hidden in biology--more likely, since language is by nature a social phenomenon, it would be an aspect of culture.
I'd like to thank Scott for pointing me to the original Chomsky article which I didn't noticed, because this link is probably the most valuable part of this rant post.
(And Chomsky's take on ChatGPT is much better than his recent takes about the Russian invasion of Ukraine…)
And he still fails making really silly claims like "For instance, a young child acquiring a language is developing — unconsciously, automatically and speedily from minuscule data", yeah, no. I have an multilingual children and the amount of data you actually need to learn a language is definitely not small.
Disclaimer: Scott Aaronson is employed at OpenAI. [0]
Which explains why he is now defending the Closed AI snake oil over his quantum computing research and is in fact defending his OpenAI equity like the rest of everyone who recently rushed into the OpenAI hype train scam.
His weak response to Chomsky tells us that he has no answer to LLMs like ChatGPTs lack of transparent explainability about the sentences it regurgitates, which is the core reason why LLMs are untrustworthy to which Chomsky repeatedly brings up.
But of course he chose not to disclose that in his article so here is the source for his announcement that he joined. [0]
I think some of Chomsky's issues with ChatGPT might be misunderstandings and some of it can be iterated on and improved. ChatGPT _is_ a statistical engine returning probabilities -- but there's nothing stopping engineers from changing how it weighs its probabilities to reject obvious falsehoods "the earth is flat" or show a moral bias "doing this will hurt people, I will not engage in it" without convoluted prompts or torturing underpaid Kenyans to label abusive content.
One thing I would've like Chomsky to engage more in is comparing how people create new "output" from previous "input" vs. how AIs like ChatGPT do it. He talks about "fake thinking" and "real thinking", but doesn't really go into the "hows" of it (ironically, kinda like how ChatGPT would).
I _do_ agree, however, that we're nowhere near AGI and this doesn't bring us closer, but I don't know what would, either.
On the topic of Aaronson's blog post, I feel like he didn't understand what Chomsky wrote about or intentionally mischaracterised it, then gave a super childish response. It's very embarrassing.
Yep, it also makes Chomsky sound bitter because chatgpt "works", but Chomsky's intellectual reputation had very little stake in getting any alternative approach working. A more accurate attack on Chomsky was that his various programs haven't yet achieved to find anything close to a universal grammar, but this is not really related to deep learning working.
"For instance, a young child acquiring a language is developing — unconsciously, automatically and speedily from minuscule data — a grammar, a stupendously sophisticated system of logical principles and parameters. "
LOL, this seems completely wrong to me. I think Chomsky's idea is that humans have some kind of 'grammar engine' that just needs configuring from 'miniscule data' aka many years of listening and practice speaking.
No doubt humans have some optimized areas for language, but I doubt it's anything so formal.
At one point in the Times article, Chomsky and his colleagues have ChatGPT make their arguments for them.
ChatGPT: "I am not conscious, self-aware, or capable of having personal perspectives. I can provide information and analysis based on the data I have been trained on, but I don't have the ability to form personal opinions or beliefs."
ChatGPT itself will tell you the limits of its capability.
I'm sorry, but if you create an NLP product and Chomsky says you missed the mark, you missed the mark. He practically admits to how far ahead of his time Chomsky was, but then somehow tries to mischaracterize it as a bad thing: In this piece Chomsky, the intellectual godfather of an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
Have you had a real conversation with ChatGPT? As in a "trying to extract factual information from another entity" conversation? I'll answer for you: no, because ChatGPT is generative (it's in the name) of apparently valid syntax with consistent meaning, but it's not extractive in nature and does not wield denotation behind-the-scenes. "Converse" means a lot more than "vomit words in a proper order at each other". By that alone, it has not "succeeded" in conversing, just arranging terms convincingly (using Chomsky's theories to do so.)
>> In this piece Chomsky, the intellectual godfather of an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
Chomsky never had anything to do with any sort of effort "to build machines that can converse in ordinary language". I have no idea where that came from in Aaronson's article. Chomsky was a great influence to cognitive science, and artificial intelligence, in particular in its early years, but he has nothing to do with chatbots. At. all. That's just plain nuts, to open an article against "Chomskyism" (bad form, that title) with such a complete fabrication.
Moreover, Chomsky's influence on AI did very much lead to systems that could "converse in ordinary language" - but with which Chomsky had nothing to do, personally, or as a researcher. The "influence" I mention refers to Chomsky's work on formal grammars, and how they have been used for a long time in NLP to model language.
For a couple historically notable examples:
ELIZA: Joseph Wizenbaum's chatbot pretending to be a Rogerian psychologist. I think at this point in the conversation, everyone has heard about ELIZA and how it drew people in and made them ascribe human qualities to it, just like ChatGPT and friends.
SHRDLU: the doctoral thesis of Terry Winograd, written in 1968-70. SHRDLU was a natural language interface to a virtual robotic arm manipulating coloured bricks in a "brick world". It was based on a natural language grammar created by Winograd, and could support intricate instructions that the system then executed with precision. The wikipedia article on SHRDLU has an example, from which I quote:
Person: Is there a large block behind a pyramid?
Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE, AND THE BLUE ONE.
Person: Put a small one onto the green cube which supports a pyramid.
Computer: OK.
Person: Put the littlest pyramid on top of it.
Computer: OK.
Person: Does the shortest thing the tallest pyramid's support supports support anything green?
Computer: YES, THE GREEN PYRAMID.
Person: What color is it?
Computer: BY "IT", I ASSUME YOU MEAN THE SHORTEST THING THE TALLEST PYRAMID'S SUPPORT SUPPORTS.
Computer: RED
Person: How many things are on top of green cubes?
Computer: I'M NOT SURE WHAT YOU MEAN BY "ON TOP OF "IN THE PHRASE "ON TOP OF GREEN CUBES " .
DO YOU MEAN:
1 - DIRECTLY ON THE SURFACE
2 - ANWHERE ON TOP OF ?
Person: 2
Computer: THREE OF THEM.
Now, if you've seen discussions with ChatGPT and earlier large language models you'll know that the above is beyond the capabilities of modern systems; including ones trained specifically to manipulate robotic arms etc. ChatGPT, faced with instructions like the above, will soon start to hallucinate objects that don't exist, then hallucinate moving them, and make a pig's meal of the entire state of the blocks world. SHRDLU, confined as it was in its virtual, toy world, could still follow the instructions of its user with absolute precision. We still have nothing that can repeat this feat. Why? Because "Chomskyism" was abandoned, and everyone turned to statistical NLP, after the AI winter of the '90s crushed AI research funding, that's why.
> In a 1991 interview, Winograd said about SHRDLU:[3]
> [...] the famous dialogue with SHRDLU where you could pick up a block, and so on, I very carefully worked through, line by line. If you sat down in front of it, and asked it a question that wasn't in the dialogue, there was some probability it would answer it. I mean, if it was reasonably close to one of the questions that was there in form and in content, it would probably get it. But there was no attempt to get it to the point where you could actually hand it to somebody and they could use it to move blocks around. And there was no pressure for that whatsoever. Pressure was for something you could demo. Take a recent example, Negroponte's Media Lab, where instead of "perish or publish" it's "demo or die." I think that's a problem. I think AI suffered from that a lot, because it led to "Potemkin villages", things which - for the things they actually did in the demo looked good, but when you looked behind that there wasn't enough structure to make it really work more generally.
Unlike "Chomskyian efforts," statisical NLP actually allows the machine to interact with natural language.
In that passage you quoted, Winograd is saying that SHRDLU was essentially an academic demo, something created only for the purpose of passing a thesis examination, even as a tech demo is something created only to show off at a showroom. That's well known.
But SHRDLU, academic demo that it was, was a program hand-crafted by a single PhD student in a few months. It didn't have behind it a team of 30+ people, or a budget of hundreds of thousands of dollars, like GPT-3 -and who knows about ChatGPT!
And yet it still worked better than ChatGPT has ever worked, even though just for the purpose of moving virtual blocks around a virtual world.
>> Unlike "Chomskyian efforts," statisical NLP actually allows the machine to interact with natural language.
Sure. And do what? Go off the rails like a crazy train, most of the time.
> SHRDLU, confined as it was in its virtual, toy world, could still follow the instructions of its user with absolute precision.
It seems clear that, in light of the quote from Winograd, that simply isn't true.
> But SHRDLU, academic demo that it was, was a program hand-crafted by a single PhD student in a few months. It didn't have behind it a team of 30+ people, or a budget of hundreds of thousands of dollars, like GPT-3 -and who knows about ChatGPT!
Do you have some idea in mind how we might scale up from SHRDLU's approach? Does Winograd? It seems pretty clear to me that no one does; it was a one-off toy that didn't solve any fundamental problems.
> And yet it still worked better than ChatGPT has ever worked, even though just for the purpose of moving virtual blocks around a virtual world.
There are plenty of questions remaining concerning LLMs. We don't know what they will be used for, where they will succeed and fail. But they are absolutely a huge step forward from earlier attempts and it's utterly bizarre to deny that.
A huge step forward- to what? To building an uncontrollable firehose of bullshit? What, exactly, is that going "forward" towards?
In any case, I'm no clairvoyant and I can't see the future. What I see right now is a huge mess and a gigantic marketing bubble. So I'll just sit tightly and wait to see what will happen in a few years.
I do have to note though that, so far, all the large hype models that have come out of OpenAI's research -GPT-2, GPT-3, now ChatGPT- have not really been useful in anything but parlor tricks: generating silly poems and malfunctioning code, and the like. They're great entertainment, but not very useful all.
As indeed have been earlier overhyped results. I wonder if anyone remembers the Neural Translation Machine in Google Translate and how it invented its own "interlingua"?
>> It seems clear that, in light of the quote from Winograd, that simply isn't true.
You're right, my comment exaggerates the achievement in SHRDLU. So let's say, not "absolute precision" but "unprecedented precision" and precision not since achieved again. Certainly not by any statistical language model.
Or, I mean, I don't know. People have controlled robots with code created by GPT-3 etc (I've been in a couple of conferences where people from MIT were showing off some research to that point). Can you show me an example of an interaction like the one in Winograd's thesis? Personally, I haven't seen any so far.
Edit: I'm re-reading my comments to make sure the tone is civil. Please let me know if you don't think it's OK.
I've used ChatGPT once for work to write a regular expression. The expression needed to use unicode properties (like \p{L}, which I didn't know existed). I had googled around and couldn't find the solution. ChatGPT gave me something close that I was able to quickly verify and use with small changes. That was enough to convince me that it has potential.
Whether LLMs are more or less revolutionary than google search remains to be seen. I predict they'll be somewhere in that ballpark.
> Or, I mean, I don't know. People have controlled robots with code created by GPT-3 etc (I've been in a couple of conferences where people from MIT were showing off some research to that point). Can you show me an example of an interaction like the one in Winograd's thesis? Personally, I haven't seen any so far.
I don't know of anything. Will be interesting to see how far they get with that.
I don't mind your tone. I think it's civil but, frankly, I enjoy a little incivility now and then.
Is this complete blog post or got trimmed? Because it ends with a claim about Galileo's telescope thus dismissing the critics via an analogy but doesn't offer much concrete chain of reasoning to reach there.
Chomsky has been extremely derisive towards anyone with whom he disagrees. You can see that in the NYT piece but it's been a decades-long pattern where dissenters are automatically morons. That's not a manner befitting a scientist.
I agree that "Chomskyism" has been a "false promise" in some sense. It's not clear to me what Chomsky's linguistics have actually accomplished. Perhaps they gave us a new way to think about language. In that sense, they were a philosophical achievement. But we didn't build anything with them, and that's ultimately the test of science. Meanwhile neural nets got us Google translate and ChatGPT (so far). Now, I don't think philosophy is useless, but Chomsky going after machine learning is a bit like "Jesuit astronomers declining to look through Galileo’s telescope" in that Chomsky is a bit like a philosopher immersed in metaphysics who is dismissive towards empirical scientists who accomplished something much more concrete.
Then there's the question of whether ChatGPT is conscious and whether it could lead to AGI. I agree with Chomsky that it's not actually intelligent in some sense. But I'm not sure how much that matters. If you can build a Q&A machine, whether it's intelligent or conscious is an interesting philsophical question but ultimately beside the point. Anyway, that debate has been raging for 50 years (see Searle) and isn't especially interesting at this point. As to whether LLMs can scale into AGIs, I have no idea, that depends on how we define AGI. To me, one of the lessons of ChatGPT is that we don't need consciousness to build useful AIs.
Chomsky is a humanist. I believe his critique of behaviorialism was based on humanism: he wanted to place humans in a separate category (as philosophers often do). I think his criticism of ChatGPT is ultimately similar. But the worry is misplaced. ChatGPT is in some sense more human than we are. It isn't some foreign, disembodied intelligence. It's based entirely on text we, as a species, generated and in that sense it's a culmination of human potential.
Regarding Chomsky's characterization of LLMs "gorging on hundreds of terabytes of data" compared to the "miniscule data" required for a child to learn a language: The brain is evolved, so the the "gorging" already happened in animals for 100s of millions of years. The brain has a lot of evolved sequence processing, visual processing, language (the authors are linguists and they admit this though it undercuts their point). Only fine tuning of this pretrained model is needed for a child to grow up speaking, say, English vs. French. This requires only a relatively miniscule amount of data.
Moreover, it doesn't matter that LLMs work differently from the human brain. Per Larry Wall, TIMTOWTDI ("There is more than one way to do it").
Most animals don't acquire language, and the non-humans who are the best candidates for it all appear to have developed it independently; specifically in humans the potential is at most around 10mil years ago since we don't find it in other closely-related primates. Actual language is ~3 orders of magnitude younger, and during most of the time since then humans didn't have such high-density ways to ingest it. As I mentioned elsewhere in the comments, GPT-3 consists of around a million modern-human-years of linguistic intake. It seems to have the strong lead here.
> it doesn't matter that LLMs work differently from the human brain.
It does if your goal is to learn how human language acquisition works, rather than grind through another trillion in VC cash via your mechanical parrots.
Chomsky's article really seems to show the results of decades of living in an echo chamber.
>On the contrary, the human mind is a surprisingly efficient and even elegant system that operates with small amounts of information;
True, but then
> it seeks not to infer brute correlations among data points but to create explanations.
This is such an odd dichotomy, and feels really wrong to me. Explanations are to me just further correlations.
> a young child acquiring a language is developing — unconsciously, automatically and speedily from minuscule data — a grammar, a stupendously sophisticated system of logical principles and parameters.
Chomsky apparently has children, but clearly may have forgotten what it's like for them to learn language! It's years of trying to communicate, constantly, failing, and learning.
Now, go ask ChatGPT to Write an ode to <obscure topic> in iambic pentameter and I bet that it will be as astonishing, creative and fluent as a native speaker.
> Chomsky apparently has children, but clearly may have forgotten what it's like for them to learn language! It's years of trying to communicate, constantly failing, and learning.
You are correct that children learning language are constantly failing in some manner. There has been a lot of study of the language acquisition of children. However, the way in which children fail and do not fail is specific. Some things children will fail at a number of times until they learn. On the other hand some rules of grammar, logical principles and parameters children seem to pick up immediately, and in Chomsky's view are even innate.
So you can divide theoretically possible failures into two categories - ones theoretically possible that are not made, and ones that are theoretically possible and are made. The first category is what he is talking about.
Also - there are different languages with different rules, and people learning a language have to learn the rules of English which are different from Spanish which are different than Japanese. Although in Chomsky's view, these languages are all so similar (compared to say the recursively enumerable language a Turing machine uses), they are virtually identical from a theoretical point of view.
Also - children make grammar errors such as first saying "I go to the store" (correct) and then "I goed to the store" (incorrect). These errors were not made by data they acquired! The errors children make are showing they are beginning to understand grammar rules, it's a sign they are going off a series of rules and not just regurgitating data they have acquired.
Chomsky is a fool and this article demonstrates it clearly.
First we don't really know how the mind works and therefore we really have no idea if the language models are path toward AGI or not.
He doesn't have an idea how a child obtains a language. You really need a lot of data to acquire the grammar, no, minuscule data is not enough.
I have observed it personally with my multilingual children, their mother tongue, to what they have the least exposure and what is the most complex, is not still not fluent for them and yes, they make obvious grammar mistakes that demonstrates that they have not fully internalized it.
His claims are simply false, easily refuted by empirical data.
I would call it fallacy of English. English is stupid simple language. Most languages are not.
I have to say I don't find "I have kids" a compelling dismissal of a lifetime of work, nor do I agree that English (or any other language people speak natively) could be described as "stupid simple" (and really, do you think Chomsky did not consider any other language before positing a universal grammar?).
It is not my fault that he made his claims too easily refutable by claiming that "young child acquiring a language is developing speedily from minuscule data". I have a comparison between two languages that are both not simple as English and there has been clear correlation between the exposure time (the amount of data) and fluency and the difference between the exposure has not been minuscule. I have noticed the same by other similar children.
I honestly don't know what he did or did not consider considering that he has made very bad judgement about two aggressor states: Serbia and Russia based on his anti-US stance. It clearly demonstrates that he is able to not think clearly.
He didn't claim that it is irrelevant, he said that it can be very small. This is clearly wrong based on empirical evidence that can be quantified based on hours of exposure to one or another language. It is enough to show a single counter example to his claim.
I'm not here to argue about Chomsky's basic claims (which I do not find useful), but I am here specifically to argue against the idea that children easily learn language with somehow way less data than ChatGPT. Those who claim otherwise simply have not spent a couple years with a baby.
GPT3's input is something like 45TB of data. It's a significant fraction of all text humans ever produced, to the point researches are concerned we may run out of text within the next few years.
Normal healthy language exposure for children is around 100KB/day, we can round up to 40MB/year.
No, I have not spent ~1000000 years around a baby.
ChatGPT is trained on some healthy fraction of the internet. You're arguing that the amount of information a baby consumes to learn language is comparable to the size of the internet?
(1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted.
(2) It lacks the capacity to learn. Learning is, in part, explaining; it is, in part, causal contact with an environment. It's output has the property of seeming-to-be-generated-by-a-learner.
(3) It lacks the capacity to speak either truthfully or otherwise. Lacking the capacity to explain, because at least it lacks the capacity to imagine (counterfactuals), it cannot thereby determine whether a claim accords with the way the world is. It's output has the property: seeming-to-be-true.
(4) It lacks the capacity to be moral. For all the above, and more: it lacks the capacity to care. It's output has the property: seeming-to-take-moral-stands.
The "Jesuit" here, I fear, is the AI fanatic who have turned their telescopes away from reality (intelligence as it actually exists), and have stolen the lenses to read scripture (intelligence as theorised by partisan academics).
One has to agree with Chomsky here at the end, "given the amorality, faux science and linguistic incompetence of these systems, we can only laugh or cry at their popularity."
The morality point I think should hit home most extremely: how horrifying to treat generated text output as-if it were constructed by an agent who cares.
Apparent properties of the output are not properties of the system; ChatGPT does not think, know, care, intend, speak, commnuicate, etc.
One can only "laugh or cry" at how absurd this sales pitch: what a horror to be invited to treat ChatGPT as possessing any of these capacities, simply because correlations across a whole internet of text seems to.