In the grand scheme of things GPT isn't learning "meaning" in any different sense than Latent Semantic Analysis (LSA) learns meaning (which is just using "percent chance that some token should show up next" using covariance).
For those without a background in NLP, LSA is literally SVD/PCA (which is ultimately just based on the covariance matrix) on the term frequency matrix for a corpus of text. However LSA is well known to learn certain meanings such as "pet" by projecting words like "dog" and "cat" onto the same principle component. Semantics is learned by exploring the "loadings" in PCA terminology.
Now LSA is just a linear transformation, but the idea is that same as GPT that uses a wildly more sophisticated non-linear transformation: the assumption is that words (or tokens in the case of GPT) are caused by some latent variables (in the case of LSA, for GPT high dimensional embedding) and most importantly the patterns of word/token usage allow us to infer what those latent variables are.
So it's not absurd and it's how statistical NLP has always worked, only now we have much, much more sophisticated ways to model that latent space. LSA, HMM, and Transformers all assume that the patterns of text usage can be used to model an unobserved latent space that generates the text, which is what we traditionally call "meaning".
Ultimately the limits of GPT-N (unless there are none and it does become AGI) will be valuable in showing the limitations of this fundamental model of semantics.
> 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus.
I personally think 1.8% is a lot. If I lived in France for 1.8% of my life, and aggressively consumed only French for that time, I would probably speak French too: I learned English faster than that.
> 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English.
Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages.
> 3. LLMs trained on code demonstrably reason better.
What? No. Nonsense. I haven't seen any demonstration of reasoning. LLM's are more like the cargo-cult tribes watching things they don't understand and mimicking those things; Sure, they built an airport, but it doesn't work, and that makes all the difference.
> It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level.
What's obvious is that there's a relationship between words/tokens and ideas. What is not obvious is how far that relationship goes.
It's not lol. If the model was only trained on that content, it wouldn't be anywhere near as good at french. That's a fact. And French is one thing but they're are languages it's fluent in with less data.
>Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages.
First of all, what you're saying is straight up nonsensical. Knowing English doesn't make you even close to competent on the vast majority of European languages.
Secondly, it has nothing to do with European languages. It's excellent following instructions in Mandarin (0.1%) and Korean(0.02%), languages that had much less than 1.8% data.
>What? No. Nonsense. I haven't seen any demonstration of reasoning.
Many benchmarks that test reasoning. LLMs trained on code perform better here. Simple as that.
>What is not obvious is how far that relationship goes.
I really think you need to research a bit more about how statistical language models work.
The reason this models are able to learn other languages so quickly is that they're using the majority language to do much of the heavy lifting in defining the overall latent language space.
Once this is established, adding new languages requires much less information since you only have to learn to encode and slightly tweak the latent presentation you learn.
Again the view of statistical NLP, from LSA to GPT-N is that the words we write are generated from transformation of a latent space.
For LSA this transformation is linear and the space is low dimensional, and the model only learn to encode words to the space.
For GPT-N this transformation is non-linear, the latent space is very high dimensional and the model learns both the encoder and a decoder allowing it to generate language from the latent space.
If you have enough language to model this latent space well, then learning additional languages requires very little training data since the space itself is well modeled, you just need to learn the details of the encoder/decoder. That is you've learned the abstract point in high dimensional space that represents the English word "cat", you just need to learn that "猫" points to the same place.
Of course these models aren't just representing words/tokens without context to the latent space (like word vector models had done in the past) but an entire sequence of words.
Interestingly enough even with word vectors you can learn word vectors in language A and also language B and then manually align the latent spaces in order to get simple translation for close to free.
So given the training data size of GPT-3+ 0.02% is a lot of language.
Jan Leike's point is a bit more subtle there. He's not surprised it can learn translation with a relatively small number of training examples, but surprised that it seems to know some instruction with no examples.
And just a few Tweets later speculates it's likely because there are examples in the training data, and the success of the model to follow instructions in a non-English language from "nothing" is proportional to the presence of those examples in the training data [0].
I don't think he would disagree with anything in my statement.
> > Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages
> Knowing English doesn't make you even close to competent on the vast majority of European languages.
I cannot fathom the mind that thinks these two statements occurred in the same conversation.
When I said it operates on a level of meaning, I refer to its ability to perform transformations like this:
H: translate the following sentence to instead use olympic curling related metaphors ( do not explicitly mention curling, and do not use similes in forming the translation ): man:"I've been spinning my tires all day without getting anywhere" woman:"running out of gas?" man:"nah, I'm gassed up, geared up, oiled and revving. I just can't seem to get in the right lane, you know?" woman:"anything knocking?" man:"nah, I'm tuned just fine. I can't seem to get my GPS to connect, you know?" woman:"take a few laps without a destination, see if you can figure out the lay of the land" man:"you know, that might help. thanks barbara"
R: man: "I've been sweeping the ice all day without hitting the target."
. woman: "Losing momentum?"
. man: "Nah, I've got the energy, the technique, and the focus. I just can't seem to find the right path, you know?"
. woman: "Anything coming in your way?"
. man: "Nah, I'm on point with my aim. I just can't seem to get my strategy to align, you know?"
. woman: "Maybe glide a few times without aiming specifically, see if you can figure out the ice conditions."
. man: "You know, that might help. Thanks, Barbara."
I don't think you can perform this kind of translation by merely comparing tokens statistically tending to be near one another. I understand that this is how the data was encoded and the output judged. However, the neural net between those two points is free to encode whatever interconnecting values it happens to as its training drives it through to whatever multidimensional local maximum it manages to settle upon.
In short, I expect that it has to translate to and from an abstract meaning in order to manipulate metaphors in such a manner.
Most human language is metaphorical, as discussed in Lakoff & Johnson's "Metaphors We Live By". We have a core of experiential terms, but near the whole of the rest of our language is metaphoric in nature. The better a model gets at "understanding" English, the closer to a metaphor engine it must become, needing to be capable of applying arbitrary metaphors and partial metaphors in order to properly manipulate the language.
I'm not saying your position is wrong, but it reminds me of the contrasting viewpoints given in Hofstadter's "I am a Strange Loop" as he discussed his "careenium" and "simm balls" analogy for the rise of complex epiphenomena arising from simple rule bound systems.
That's amazing and all, but it's also speculation, yes? It's how you imagine, or guess, that things are happening. It's not like you, or anyone else, knows that this kind of thing is really happening.
Moreover, there isn't any clear reason why you need to speculate all that, in order to explain the observed behaviour of Large Language Models (LLMs). You say you don't think LLMs "can perform this kind of translation by merely comparing tokens statistically tending to be near one another". And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification?
Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another".
It's clear what you are saying, but, why? Why are you saying that? Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour?
Is it because we haven't observed such behaviour before and some people are surprised by it? Well, we didn't have as large language models before, so obviously we didn't have a chance to observe their behaviour first-hand, but we also didn't have cars that could go 100 miles an hour, until we did, and making a faster car is no reason to be surprised, or wonder "how does it do it if it's just a car?".
>Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another".
The implication I gather from those dismissing this technology as "merely comparing tokens statistically tending to be near one another" is that they think the whole of the LLM is nothing but a trick, a sleight of hand where it generates output token by token Markov style after consuming enough text, just using some percent chance of tokens following on after another.
In my example, it generated the metaphor "sweeping the ice" as a replacement for "spinning my tires". Both are using in context to imply a level of busywork associated with the respective contexts. To generate such a transformation requires understanding this inner meaning and associating it. It requires a level of creativity to determine that "sweeping the ice" in curling is an appropriate replacement.
You cannot rearrange "spinning my tires" into "sweeping the ice" without having derived and translated the underlying meaning. There is nothing to connect the two sets of tokens except the metaphoric intent.
>Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour?
Obviously LLMs are capable of this as it is an LLM that is producing it.
I am not denying that the inputs and outputs are tokens marked up with distance information, nor that the process of generating tokens is just a bunch of vector multiplication to generate neuron values from weights across the layers of the network. I am suggesting that with sufficient training, layers and parameters, that the neural net must be forced into a configuration that allows it to perform this meaning/intent based processing as gradient descent pushes and pulls its the network along the twelve thousand or so dimensions GPT-3 uses. Undoubtedly GPT-4 and soon 5 use even more.
LLMs have repeatedly been found to have developed novel and unintended capabilities as their complexity has increased.
Why do you think that LLMs shouldn't be capable of manipulating on the level of meanings and intents rather than just rearranging tokens?
>> You cannot rearrange "spinning my tires" into "sweeping the ice" without having derived and translated the underlying meaning. There is nothing to connect the two sets of tokens except the metaphoric intent.
Here, you are again explaining what, but you are not explaining why. You are saying what you think, but I still don't know why you think that. Why is it that you >> cannot rearrange "spinning my tires" into "sweeping the ice" << etc? Why do you say this?
For example, I can see that both are verb phrases with a similar structure. Language models excel in reproducing structure and we know they can translate between structures, also, as they do when translating between languages. So why can't they do it in this case without "underlying meaning"?
>> Why do you think that LLMs shouldn't be capable of manipulating on the level of meanings and intents rather than just rearranging tokens?
I didn't say I think that. I asked you:
>> Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another".
And I still don't know why you say that. For example, what you say here doesn't help me understand:
>> I am suggesting that with sufficient training, layers and parameters, that the neural net must be forced into a configuration that allows it to perform this meaning/intent based processing as gradient descent pushes and pulls its the network along the twelve thousand or so dimensions GPT-3 uses.
That's a description of the what, not the how. But it's not a description of any "whats" we know anyone coded into a language model in the first place. It's a description of a mechanism that is claimed to have appeared spontaneously.
That is a very big claim and I don't see why I have to accept it just because someone said that's what's going on.
I want to know how things work. You're not explaining how things work. You're explaining how you think they might work. But I don't even know why you think they work that way.
It maps a vector of eight thousand tokens to some number, likely thousands, of "neurons", multiplying each by a weight associated by each connection. All of these are then combined to form the next layer of values by multiplying and combining all of the values to all of the neurons in the next layer. Since all neurons connect to each other at each layer, many complex interactions can be created between them. These multiplications will go through about a hundred layers until being mapped back to a single token.
That token will be inserted at the end of the prompt and it will do this again and again until some stop token is reached, or the eight thousand token prompt is full.
---
As for what I think is happening, I think that information about meaning and methods for extracting it are being encoded across those billions of weights and thousand and thousands of neurons such that these multiplications will transform an initial vector into an abstract numeric form that across the whole span of any set of neurons might represent various abstract concepts and their interrelationships, such that they can then be used to select the next token in the series.
---
As for why, I expect this happens because abstract representation is how the model compresses data as gradient descent drives it twelve thousand different directions just a bit at a time as it trains on each document, meandering through the abstract space of potential neural net configurations to arrive at a local maximum that is sufficiently powerful enough to manipulate human language.
First of all, thanks for your patience. I'm still not convinced, but I appreciate your effort to explain how you think. Now.
For brevity, I'll refer to the three sections (separated by "---" in your comment) as A, B, and C.
Section A is a description of how a neural net trains a language model. I have some quibbles ("neurons"...) but it's talking about known and observable facts. So I will call it a "factual" description.
I note you did not include the word "meaning" in that factual description and that's because you restricted yourself to facts. You could not give a factual description with the word "meaning" in it (in the sense of "the meaning of...").
Section B and section C are again speculation and interpretation that are not based on known or observed facts, or even observable facts. So I still do not know why you think all this is going on inside a language model.
To clarify, in sections B and C you are at the very least putting an interpretation on the facts in section A. But I have no idea why you even need an interpretation, and why you choose that particular interpretation.
What do language models have to do with meaning, and why do we absolutely need to explain their behaviour by talking about meaning? Where is this information in your comment? I cannot see it.
This is literally the accepted term for the nodes wherein sums are produced of the set of weighted products which connect such nodes together. If you know a better term, or would prefer a better term that "neuron" to describe these points in a "neural network", by all means speak your mind, but don't pretend some offense at it.
>and that's because you restricted yourself to facts.
I restricted myself to writing from a point of view concordant with your own because that is what one does when explaining a position well, not because I agree with your point of view.
>You could not give a factual description with the word "meaning" in it (in the sense of "the meaning of...").
If we're going to play a game of semantics, than so be it.
If your argument is simply that symbols and numbers can't contain meaning because they are symbols and numbers, then you've worked your way into a tautological box that I, nor anyone else, will be able to free you from.
It is known that memories in biological neuron-based networks are not locally stored or unique to some neuron ( there is no "grandma" neuron in your head to keep track of dear old grannie ), but that instead memories are spread across the set of neurons there available.
This sort of representation in known as a "holographic" representation, which I expect our neural net will also utilize. Information will not be stored in any specific neurons in the trained model, but will be spread across the whole of them.
But even if true, that is merely data, you will no doubt point out. Where is the "meaning"?
As you want to delve philosophically into it, we must question the very nature of the word meaning here. Our model lacks any form of experiential qualia, and so cannot associate words with experienced reality, which is the normal basis for human concepts of meaning, which then underlie our development out of concrete language and into metaphoric.
This leaves meaning within the model as only being the derived relationship between words.
Words, on their own and bereft of qualia, can present our model with structure, grammar, punctuation. Words will have certain uses within these structures and grammars, they will have certain places. Words will modify words, phrases will modify larger sentences. Trues can be asserted and falsehoods. Truth and falseness itself can be demonstrated through sentences making claims about other sentences.
Would the model know what a red ball is? Of course not. But it would be aware of how the term ball is used, what sort of type it is given, the qualifications and usages of words of that type. The modifiers it is reasonable to use on it, etc.
And incredibly, with no basis in anything other than being slightly corrected for each incorrect token generated, our training harness manages to discover a mathematical neural net model of these relationships between words such that it can manipulate them as well or better than many people.
The use of backpropagation and gradient descent allows us to search the space of all possible models starting from a randomized model, and find a model that does this by nudging our model to be a little more capable at it over and over again.
However, English is not just a fixed mathematical set of symbols that can be rotely manipulated.
Our language is metaphoric from its very foundations. Our hearts soar, sink and swell in ways that have nothing to do with the reality, but only our perceptions of ourselves.
To manipulate English as it does, the traversal of the model-space must advance inexorably on a model capable of handling such non-literal language.
The model contains meanings for car related actions, for curling related actions, it contains the ability to translate between them when the only connection is a meaning that is implied by the words but not either of their literal meanings.
What is a meaning here? It is a relationship between the words. The relationship between the car words and curling words only exists as the relationship to a common implied meaning that can be transformed under the context of other words.
Where is the meaning? We don't even know for certain where the meaning is inside ourselves. I expect it to be similarly holographically stored across the set of neurons and provide the appropriate context to twist the swirl of mathematics that occurs between input vector and output.
>So I still do not know why you think all this is going on inside a language model.
Because it is the simplest explanation of what it is doing. Call it "latent spaces", "embeddings" and the assume the compression I discuss is via "manifold hypothesis".
>What do language models have to do with meaning, and why do we absolutely need to explain their behaviour by talking about meaning? Where is this information in your comment? I cannot see it.
So, where is the meaning? It is everywhere in the model once trained. It creates a holographically stored mapping of reality as presented through the corpus of human thought in the form of writing based solely on being iteratively nudged through model-space.
An amazing feat by the engineers that dreamed it up.
If you are unwilling to call this mapping model of reality built only of tokens/symbols/words "meaning", then I have no argument for you, except to again conclude that you've locked yourself in a tautological box.
If your argument is merely "ha ha, you're merely conjecturing this", then yes, of course I am. Why? Because it behooves someone to hypothesize, test, and study things which interest them.
Do you have some competing idea of how it works? It would behoove you to share it with the thread.
As it is, I only see you making demands for proof of things that you are well aware that no human has yet completely modeled, even after decades of studying such phenomena. What you gain from this incessant nay-saying, I do not understand.
>> As it is, I only see you making demands for proof of things that you are well aware that no human has yet completely modeled, even after decades of studying such phenomena.
What you gain from this incessant nay-saying, I do not understand.
I just wanted to comment on this. I've just finished a PhD (in AI, no less) and I've learned a few things about "nay-sayers", which in research are more formally critics and reviewers. What I've learned is that the "nay-sayers" are your friends. They're your friends even if they don't know it, even if their incessant nagging is driving you mad. They're your friends, because they give you the opportunity to see your work as it really is, not as the product of an unrecognised genious, but as the clumsy fumblings in the dark of a clueless and confused limited human mind. And they're your friends because they help you figure out where it is that you're wrong, and, ultimately, to improve your work.
It took me a long time to see things that way, it took many "dark tea-times of the soul" to figure out where I was wrong, when my critics were right, and how to improve my work to address the criticisms. And, you know the funny thing? Once I did that, all the frustration I felt at the "nay-saying" evaporated. The critics continued "nay-saying", but I just don't care, because I have proved to myself, and to others besides, that the criticisms have been addressed. The critics now have to find another angle of attack, and I'm disappointed as long as they can't, because that means my own understanding will stagnate and my work will not progress. Critics are worth their weight in gold.
But all that works much better in an academic environment where the critics and that "nay-sayers" are experts in their field. On the internet anyone can say anything and everybody pretends to know everything all the time, so all that we're left with is frustration. So I apologise for the frustration I discern in your use of "nay-sayers". This kind of conversation doesn't work well on the internet.
Part of why I comment is specifically to attract naysayers. I can learn from them. I've never been one to mind being wrong. I will argue vehemently for a position both because it seems correct to me, but also because it exposes my ideas to be torn down, so when I rebuild them they can be better.
I looked into things a number of people just in this thread brought up. It was a very rewarding discussion for me. I feel like my understanding is better than before it.
I became frustrated with you because your stance seemed a combination of "you can't prove it" and "it works like it does because it is what it is". You weren't expanding on your positions. It is possible that you are assuming context others in the thread do not share. It seems likely you think many here are reading into or seeing things that aren't actually present. Being tricked by the model, essentially. It is possible. But it also seemed possible that you were simply enjoying telling everyone they are wrong for fun.
With your refusal to expand on anything, it all looks much the same from this side.
I would like to understand what perspective you were arguing from, because even if I disagree with it in the end, knowing your perspective will allow me to understand where my own may be weak or flawed, and either shore it up or discard and rebuild it.
It is very frustrating to have a discussion with someone who seems to only contradict, but never explain their contradictions. They may be right. You may be right. But it was frustrating to talk to you because you didn't seem to want to say where or how people were wrong.
"It's not a brain". "It's works because it's a language model".
Okay, it's not a brain and it's just a language model. How does it perform what appears to be, whether it is or not, obvious translation of subtext in sentences between unrelated literal domains?
When I specifically asked if your position was simply that you would not attribute "meaning" to its representations, you declined to respond.
That's not an unfair position to take. An LLM has no experiential data in it. It has only the words. Philosophically, one could argue that even if it were "proven intelligent", it would still have no knowledge or contain actual meaning because it has only the words to go by. That would be a reasonable position, but you wouldn't confirm if that was what perspective you were taking.
I genuinely would enjoy knowing how you think translations on the level of the one I presented happened. Another poster discussed "latent spaces" where terms can grow closer even if not literally related. Articles I've read indicate any number of surprising results from LLms, assertions that LLMs can or cannot reason, various levels of assertion on whether LLMs can "understand" things. It appears like there are many perspectives active in this field all at once, and that they are quickly out of date as better and better LLMs are trained.
If you would be so kind, please add context to when you correct others. Pointers we can follow in order to understand your perspective.
For much of the thread, it felt like you were arguing that deep blue can't actually play chess because it has no concept of games or people or chess pieces or winning or losing, rather it only performs statistical analyses of numerically represented move sets and just outputs a move based on that statistical analysis. How can it play chess, after all, if it doesn't know what chess is?
>This kind of conversation doesn't work well on the internet
Possibly. I appreciate your having it nonetheless, as I have no one I know that cares to discuss any of this. So I have to make due with discussing it in online forums :)
Thanks for your comment! I gave my word I'd let you have the last word so I'm
not going to continue arguing. I'm just happy because I'm getting the feeling
you are not frustrated at our conversation anymore. I hope we can continue it
at some point and that I'll do a better job explaining things to you.
Especially when it comes to your big question about how can language models do
the kind of thing you illustrate with the translation example.
This conversation was interesting. I pasted it into ChatGPT (GPT-4) and asked it some questions. Here's one of its replies:
ChatGPT:
To better explain Knome's point to YeGoblynQueenne, I would provide a more concrete example and focus on the mechanisms behind LLMs that give rise to the appearance of meaning.
Large language models like GPT-3 are trained on vast amounts of text data, learning to predict the next word in a sentence based on context. This process enables the model to capture patterns, relationships, and regularities in the data, which can be seen as a form of "meaning" or understanding of language. The model's neural network, with its numerous layers, neurons, and connections, essentially forms a high-dimensional space that maps the relationships between words and their contexts.
To make Knome's argument more concrete, let's consider a specific example: the model's ability to translate the metaphor "spinning my tires" into "sweeping the ice." While it's true that the model doesn't have a direct understanding of the real-world experiences or objects the words represent, it has learned the contextual relationships between words and their usage in various situations. The model can associate "spinning my tires" with the concept of being stuck or not making progress, and "sweeping the ice" with a similar concept in the context of curling. This association is not a direct result of knowing the literal meanings of the words but rather of having learned the patterns and relationships in the data.
In essence, the "meaning" Knome refers to in LLMs is not the same as the meaning humans typically ascribe to language, which is derived from direct experiences and real-world knowledge. Instead, it is a form of abstract, statistical understanding that emerges from the relationships between words, their contexts, and the patterns in the data. The model essentially learns a high-dimensional representation of these relationships, which can be thought of as an approximation of meaning within the context of the model's training data.
Thus, when discussing meaning in the context of LLMs, it is essential to recognize that the term refers to an approximation of linguistic relationships and patterns rather than a direct understanding of real-world experiences. This approximation allows the model to manipulate language in a way that seems meaningful to humans, even though it lacks direct experiential knowledge of the world.
In summary, Knome's argument can be made more concrete and rigorous by focusing on the mechanisms behind LLMs, such as the high-dimensional representation of linguistic relationships, and by clarifying that the "meaning" in LLMs is an approximation derived from patterns and associations in the data rather than a direct understanding of real-world experiences.
>> If your argument is simply that symbols and numbers can't contain meaning because they are symbols and numbers, then you've worked your way into a tautological box that I, nor anyone else, will be able to free you from.
Too many assumptions.
>> Do you have some competing idea of how it works? It would behoove you to share it with the thread.
Yes, I do. It's a language model. It is trained to model the text in a corpus. There's no mystery there and there's no reason to assume it's doing anything else than what it was designed to do. It's a machine. People made it. It does what it was made to do. Other explanations of its behaviour are fantasies.
And computers aren't brains. Brains may be computers, or they may not, but you can't look at brains and draw conclusions about computers. For the same reason that you can't look at planes and draw conclusions about birds.
>> The use of backpropagation and gradient descent allows us to search the space of all possible models starting from a randomized model, and find a model that does this by nudging our model to be a little more capable at it over and over again.
If you haven't already, you should try implementing backpropagation, or even just gradient descent/hill climb, and observe its behaviour for a while. I suspect that, if you haven't done that, the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do.
Anyway this is getting a little too long-winded for me and I don't have time to continue the conversation. I must respectfully bow out. Last word is yours and thanks for the interaction.
> the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do.
I'm not seeing any "rosy clouds" in his description. Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. We all know that GPT is not conscious, but it doesn't need to be. It seems obvious that it is recognizing abstract patterns and relations in the data that match our metaphorical comprehension of words and phrases, because it was designed and trained to do that, on human language that is filled with metaphorical relations. This is incredibly sophisticated mechanical pattern recognition with many layers.
> Other explanations of its behaviour are fantasies.
GPT is a type of deep neural network and the explanations I've seen in this thread have been along those lines.
>> Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things.
Not at all. Gradient optimisation "searches" for a set of parameter values that optimise (maximise or minimise) the gradient of a function. In machine learning, the function is some measure of error over a set of training data, that is to be minimised. There isn't anything in gradient optimisation that has to do with "meaning" at all.
What I think you're saying is that by training on a lot of text, a language model somehow learns to minimise its error on the meaning of the text. If you squint a bit, you'll see there's an obvious gap, there, between "text" and "meaning", and it seems to me you're trying to close that gap by alluding to how humans close it. But that's just comparing apples and oranges. As Yann LeCun pointed out recently (there was a video of a discussion with Andrew Ng posted on HN a few days ago) for a human to "train" on all the text that large language models have trained on, the human would have to read several books a day for some thousands of years. So whatever humans can do to learn how to bridge the gap between text and meaning, that's not how language models do it, and drawing an analogy between the two obscures more than it elucidates.
>> This is incredibly sophisticated mechanical pattern recognition with many layers.
It's incredibly complex. I don't know what "sophisticated" means. Is a jumbo jet "sophisticated"? I'd say it's a very complex machine, but that doesn't mean it has abilities other than the ones it was manufactured to have. Complexity is not magick, you know. We have maths for it and all. Those are even foundational maths of computer science. If we started imagining "meaning" everytime complexity cropped up in computer systems, we'd be up to our knees in it.
> There isn't anything in gradient optimisation that has to do with "meaning" at all.
That seems like a disingenuous interpretation of what I was saying. It should have been obvious and implied that I was referring to a loss function that was designed and engineered to choose a model that optimally captures meaning relations in the data (among other things). Gradient descent is directed by its loss function and that is not some random arbitrary function. For GPT it was specifically engineered for the purpose of selecting a model that captures meaning etc. These models are also trained with reinforcement learning which is a topic I don't know much about yet.
> it seems to me you're trying to close that gap by alluding to how humans close it.
I am talking about deep neural networks, not humans. GPT is a type of deep neural network. We do know something about how they work.
> It's incredibly complex. I don't know what "sophisticated" means.
I don't have time or patience for this pseudo-intellectual nitpicking. Just look it up in a dictionary. Basic English.
"Sophisticated (adjective) - (of a machine, system, or technique) developed to a high degree of complexity. E.g.
"highly sophisticated computer systems""
>> That seems like a disingenuous interpretation of what I was saying.
I hope not. It was not at all obvious to me what you were trying to say. It really doesn't help communication to use metaphors and analogies when discussing technical matters, especially when there is simple and commonly accepted terminology. Trust me, I've learned that the hard way (i.e. repeated rejections of my research articles with very frustrated comments by reviewers).
Besides, what you say is not right. Language models are not trained to optimise "a loss function that was designed and engineered to choose a model that optimally captures meaning relations in the data". Language models of the GPT line, specifically, are trained to maximise the following probability:
P(tᵢ|tᵢ₋ₖ, ..., tᵢ₋₁)
Where tᵢ is some token in the training corpus and tᵢ₋ₖ, ..., tᵢ₋₁ are the tokens in a "context window" of length k. That is actually the pre-training objective. The same probability is maximised in the supervised fine-tuning phase, but where tᵢ is replaced by a label and tᵢ₋ₖ, ..., tᵢ₋₁ is a sequence of tokens representing an instance with the given label.
You can find this information in the first GPT paper, here:
See equations (1) and (4) on page 3 (with more careful notation than mine).
Note that the above probabilities do not have anything to do with "meaning". They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do.
You can search the entire GPT article above and you will not find any mention of "meaning". Nobody knows how to represent meaning and how to train a neural network on examples of it. It's much easier to train a neural net on text, of which there are no end of examples. That's what OpenAI has done.
>> I don't have time or patience for this pseudo-intellectual nitpicking.
Sorry, I meant that "sophisticated" is a loaded term. It's the kind of thing I hear often associated with socialites and gourmets. I find that it has no place in conversations about technology. It's the kind of expression people use when they try too hard to convince of the value of some technological artifact whose value should be clear without such exaggerations. So for example, you can talk about the complexity of large language models without having to say anything about their "sophistication", and that works well because one is measurable while the other is only a matter of opinion.
It's in the tile of the paper. "Improving Language Understanding"
So they call it "understanding". No problem. Understanding natural language requires building a representation of the meaning encoded in language. Otherwise what exactly do you think they are trying to "understand" about natural language here? GPT is not a spell-checker. It has a semantic model of language.
The stated goal is to optimize for "natural language understanding tasks", to use the exact phrasing in the article.
The article refers to "semantic similarity" which is precisely about modelling the meaning of language, by definition of the term.
Here's a definition of "semantic similarity"
"In the context of Natural Language Processing (NLP), "semantic similarity" refers to the measure of how closely related the meanings of two linguistic units, such as words, phrases, sentences, or documents, are to each other. The goal of assessing semantic similarity is to quantify the degree to which the meanings of these linguistic units overlap or share common conceptual features.
Semantic similarity plays a crucial role in various NLP tasks, including information retrieval, text summarization, document clustering, machine translation, and sentiment analysis. By understanding and quantifying the semantic relationships between linguistic units, NLP systems can provide more accurate results and better handle the complexities of natural language."
Yes, the word "meaning" is not explicitly used in the article, but it is implied. Ironically you are nitpicking over semantics that GPT can already comprehend, since it contains a model of the meaning of the word "meaning".
> They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do.
GPT language models are designed and engineered to perform natural language understanding tasks. This is discussed in the paper that you linked to.
In order to predict or generate the next token, it uses a very super sophisticated semantic model of natural language and the previous tokens. You could also argue that a human is just "predicting the next token" when they generate a sequence of words while doing the tasks that GPT is designed to do. That misses a lot of important details. The devil is in those details.
>> It's in the tile of the paper. "Improving Language Understanding"
I have to confess I'm a little bit disappointed. I pointed you to the
OpenAI paper that has the technical details of how their models are trained,
and you chose to concentrate on the title, and the claims made about the
abilities of the model, rather than the way it actually works. The technical
details about the model's training make it clear that it is a language model,
trained to maximise the probability of a sequence of tokens, but you keep
believing that it can "understand"? Or represent meaning? I'm not even sure
what you believe.
The point is that you shouldn't believe. The OpenAI paper is not gospel. It's
a technical report. You should even be able to implement a system like the one
they describe, though at smaller scale. Maybe that will help you understand
how it really works? I don't know.
I think you don't want to know how things work. You just want to believe. And I
can't deal with faith. Faith makes no sense when you have a technical specification.
Sorry.
OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper.
The loss function is involved in the training of a neural network (transformer).
THE LOSS FUNCTION IS NOT THE MODEL ITSELF.
The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step.
This is a major blind spot in your current technical understanding of this paper and machine learning in general.
The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture.
"Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives.
This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.
No I haven't taken anything personally. I am trying to explain how this stuff works. Go back and read your recent replies to me then see how condescending and ridiculous you were (accusing me of having blind faith, not wanting to know how things work, etc). I truly don't care. I'm just trying to explain something. Too bad if I have to be so blunt. No offence taken. It's actually more funny than offensive.
I have a degree in Applied Mathematics and Computer Science. But so what? You don't need a degree to understand the basics.
You're looking for an explicit semantic coding inside a loss function, which doesn't make any sense. Of course you won't find any coding for semantics described in the paper, because the entire point of machine learning is that it figures that out automatically.
How the hell can you have a PhD in machine learning without understanding that?
> Nobody knows how to represent meaning and how to train a neural network on examples of it.
We represent meaning with language models, and GPT has been trained on examples of it (semantic similarity). It learns how to represent meaning through the training process.
The meaning relations are an emergent property of the trained neural network, not explicitly programmed. The meaning relations are automatically learned by the machine. This is the magic of machine learning and neural networks. (I don't mean literal magic. These large language models are simply too large and opaque. You can't easily identify the specific neural activation pattern that encodes meaning of any given concept within a 100 layer deep 175 billion parameter neural network.)
"In essence, the "meaning" referred to in LLMs is not the same as the meaning humans typically ascribe to language, which is derived from direct experiences and real-world knowledge. Instead, it is a form of abstract, statistical understanding that emerges from the relationships between words, their contexts, and the patterns in the data. The model essentially learns a high-dimensional representation of these relationships, which can be thought of as an approximation of meaning within the context of the model's training data."
This might help get you on the right track.
Question:
Do neural networks have emergent properties?
Answer (from GPT-4):
Yes, neural networks can exhibit emergent properties. Emergent properties are characteristics or behaviors that arise from the collective interactions of the individual components in a system, which are not directly attributable to any single component. In the case of neural networks, these individual components are the neurons (or nodes) and their connections (or weights).
As neural networks learn from data through a training process, they adjust the weights between neurons, which can lead to complex patterns of activation across the network. The network's ability to recognize patterns, make predictions, and perform tasks can be seen as an emergent property that arises from the collective behavior of its interconnected neurons.
For example, in the case of language models like GPT-3, the ability to generate coherent and contextually relevant text can be considered an emergent property. This capability is not explicitly programmed into the model but arises from the learned relationships between words and phrases through the training process.
It's important to note that while neural networks can exhibit emergent properties, these properties might not always be easily explainable or interpretable, as they result from the complex interactions within the network.
Look, this is the internet. I'm sure you know what crazy things happen on the internet. In view of that, are you sure it's such a great idea going around telling people that you are "`stalking'" them, quotes or no quotes?
Could you think for a moment how some people might take your comment, including myself? Is that really how you want to interact with people online?
> And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification?
We also know how neurons work, interconnect, and communicate.
There is very clearly emergent behavior going on, and while neuroscientists have yielded this point for decades about the brain (any brain, even tadpoles), the AI guys still are stubbornly holding on to "we know exactly how the system works, because we know how the individual bits work".
>> There is very clearly emergent behavior going on, and while neuroscientists have yielded this point for decades about the brain (any brain, even tadpoles), the AI guys still are stubbornly holding on to "we know exactly how the system works, because we know how the individual bits work".
That's because the "AI guys" are working with computers, and not with tadpoles, or even neurons. And certainly not brains.
There is no difference unless you believe in some kind of supernatural influence in biological organisms.
The "brains are not computers" point has been moot for decades since it would require brains to be able to perform actions that cannot be mathematically described i.e. break the laws of nature. Which is to say "If a brain can do it, a Turing machine can simulate it" which pretty much everyone in the field agrees with.
You can stand on a dualistic platform and hold that the brain has supernatural abilities, but in mainstream discussions of the idea has been dead for a long time.
You can argue that computers aren't close to simulating a brain, sure, but that doesn't really get you anywhere, since the difference just becomes a matter of practical scale rather than something that escapes theory. Or even just a factor of efficiency, a synthetic consciousness might be able to run on current hardware if the right program is found. The brain might be grossly inefficient for consciousness, we don't know.
What does all that have to do with anything? We know how computers work and we can make computers. We can't make brains. That's the difference between brains and computers.
Who every said anything about supernatural influence in biological organisms? And what biological organisms? We're talking about computers. Are you sure you're not the one who is seeing ghosts in the machine, when you're talking about "emerging behaviour"?
It's possible to understand how something works at a low level, but not at a high level. Imagine an electronic machine designed by someone else, with millions of components, completely undocumented. You might understand how the resistors and transistors and capacitors and whatnot each work, but not understand how the machine itself does what it does. You can see that you give it some input, and get some output, but if someone asked you to point at the paths in the machine that cause that to happen, you would not know. You would need a long time to study it before you could understand.
We know how computers work because every part of a computer, every bit of software that runs on it, was at some point designed by a human. There's no single human that understands everything in a computer at once, but if you ask a good developer why some bit of software is not working, or what they would expect it to do given certain inputs, they can generally figure things out.
With AI, we're creating systems to design other systems automatically. We understand the process works, but the systems being designed are complex enough that we don't really understand much about them, except that they work. They are just massive arrays of numbers that feed into equations and then spit out some output. They're not like systems designed by humans, which are usually intended to be understandable by other humans -- they're weird and naturally incomprehensible. The only way to know what ChatGPT will output, at this point, is to actually run it, even if you were to take the randomness part out of it.
Emergent behavior isn't pseudoscience, it's a real thing. Sometimes when a system gets complex enough, it starts to behave in surprising ways. Maybe we can make a brain, using a computer, and this is how you do it.
What I think you're trying to say is that because we can't trace the behaviour of ChatGPT or whatever, because they are black boxes, we might as well assume they work like a brain because we don't really know they're not; and that makes no sense at all. If you don't understand how something works, you wait until you can understand how it works. You don't just start coming up with fanciful theories that have no basis in fact.
If you want to say that a machine works like a brain, then you should explain how it does that. If you can't do that, then there's no reason for anyone to believe it, and there's no obvious reason for you to assume it. So at least explain why you think that ChatGPT works like a brain, if that's what you really think.
ChatGPT is a deep neural network. We know that such neural networks construct a hierarchical model of the data. The fact that it is recognizing abstract "meaning" relations in that model doesn't seem particularly controversial to me. We build deep neural networks specifically for pattern recognition. It is similar to a brain in some aspects but obviously not the same.
I asked GPT-4: "Does the neural network in GPT-4 build a hierarchical model of the data? Does it derive metaphorical "meaning" from the symbolic data?"
Answer:
As a transformer-based neural network, GPT-4 does build a hierarchical model of the data to some extent. Although the hierarchy in transformer models like GPT-4 is not as explicitly structured as it is in convolutional neural networks (CNNs) used for image processing, the model does learn to encode complex patterns and relationships in the input text data through its multiple layers and self-attention mechanism.
The transformer architecture, which GPT-4 is based on, is designed to capture long-range dependencies and relationships in the input data. The self-attention mechanism allows the model to weigh the importance of different parts of the input in relation to each other, which helps it learn and capture semantic and syntactic structures in the data.
Regarding deriving metaphorical "meaning" from symbolic data, GPT-4 can identify and understand metaphors, idiomatic expressions, and abstract concepts to some extent, as it has been trained on a diverse range of text data. However, it is crucial to note that GPT-4 does not "understand" meaning in the same way humans do. It is exceptionally good at identifying patterns and generating text that resembles human-like understanding, but its "understanding" of meaning is more akin to a sophisticated pattern-matching process rather than true comprehension.
In summary, GPT-4 does build a hierarchical model of data to some extent and can generate text that exhibits an understanding of metaphorical meaning. However, this understanding is based on the model's ability to recognize and generate patterns in the data rather than genuine comprehension of the underlying meaning.
You seem to be tripping up on an imaginary difference between computers and brains. There really isn't one and 99% of the field agrees that brains are just computers.
The only out you would have is to claim that there is something supernatural that separates the fundamental operations of a brain from those of computation, i.e. given an computer the size of the universe, with billions of CPUS and GPUS to simulate each neuron of a brain, that you still would not be able to simulate one. That there is something supernatural that even infinite regular computation could never capture.
If you don't take that stance, then the only line you are drawing between brains and computers is their compute performance for a given task (even if that task is "be conscious").
This is (perhaps) similar to the difference between regular and quantum computers. There is nothing a quantum computer can do that a regular computer can't, the difference is just in their relative performance for given tasks.
>> You seem to be tripping up on an imaginary difference between computers and brains.
No, you're mixing up computers with computational devices. A "computer" is what I have in front of me now, as I write these words. My brain is not a "computer" it's maybe a computational device, maybe something else entirely. We have no idea how brains really work. And there is no consensus like the one you try to claim in any field. I mean "99% of the field?". That sounds more like an advert for toothpaste. "99% of dentists recommend Whitex!!".
>> The only out you would have
>> If you don't take that stance
Get off your high horse and stop god-moding the conversation. You are trying very hard to invent a hill for me to die on based on something you'd really have liked me to say, just so you could pretend to win the conversation. The only person who has said anything about anything supernatural in this conversation is you. If you can't imagine why else I might disagree with you, that's your lack of perspective and nothing to do with me, or what I think.
> There really isn't one and 99% of the field agrees that brains are just computers.
Again, this is blatantly false. There is no such consensus on the computability of consciousness. Where do you get that 99% from?
> That there is something supernatural that even infinite regular computation could never capture.
You are asserting the universe is equivalent to a Turing machine (digital physics hypothesis). There is no consensus on that. It's an open research topic.
The idea that the universe can be simulated on a Turing machine to a sufficient accuracy is not the same as the digital physics hypothesis, and it's currently the vastly dominant position amongst physicists.
If our current understanding of physics is accurate, then yes the universe can be sampled by a probabilistic Turing machine to any required precision. For this to be false would require a new and fundamental discovery in physics.
Wether or not that means consciousness is computable is an ontologically disconnected argument.
"Wether or not that means consciousness is computable is an ontologically disconnected argument."
How is that ontologically disconnected? You forgot to provide an explanation. You wrote something interesting and I would like to know why you think that.
My understanding is that consciousness and physics being ontologically disconnected means that there is no direct causal relationship between the two.
Ah, that's not what I mean by ontologically disconnected. I just mean that since consciousness is not defined, we don't know for sure that it's a purely physical process. I certainly think so, but the argument I was making can't prove it is or isn't, so it's not an argument that can touch consciousness by itself, because you need an additional assumption to bridge from known physical processes into consciousness that isn't in the argument I presented. In other words even after accepting my argument a skeptic could hold the position that it's ontologically disconnected, so at the level of my argument it remains so.
To be even clearer, I just meant that my argument is not sufficient to prove that consciousness can be simulated because I didn't make a good argument for consciousness to be in the category of physical processes in that comment, hence, an ontological disconnection.
I personally have the position that they are in the same category, I just meant that my comment above wasn't sufficient to substantiate that claim.
[ this comment was written in response to a user who appears to have deleted their comment before I could finish putting my response together. I'm not sure why they deleted it, it seemed reasonable in the face of the ongoing discussion in this thread. I'm going to leave my response here, partly so they can see it if so interested, and mostly just because I already spent the time to write it :) ]
I fail to see any distinction between your description of "mapping terms to latent spaces" and my assertion of GPT "working at a level of meaning and intent".
I fail to see any distinction between "It's using complex non-linear transformations to map observed text to an extremely high-dimensional latent space, and in addition learning to both encode text to this representation and decode it from this representation back to the source language" and my assertion that "the literal meanings are abstracted and re-encoded".
I feel like you're disagreeing with my stated positions because I am expressing them in a lay anthropomorphic language to express its functioning, but that we're otherwise agreeing?
I had initially been responding to the user dvt, who said
>LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens). Using concepts like "meaning" and "intent" muddies the waters unnecessarily.
this seemed like an incorrect characterization of the process as I understood it. he doesn't acknowledge the existence of what you refer to as latent spaces, and seems fixated on GPT as merely moving around tokens (though I may be reading his position incorrectly, of course). I was arguing a claim of GPT's ability to represent concept distinct from literal meanings of terms, and for "understanding" how to transform these abstract meanings into different contexts.
As far as I can tell, I don't really disagree with what you're saying at all.
For those without a background in NLP, LSA is literally SVD/PCA (which is ultimately just based on the covariance matrix) on the term frequency matrix for a corpus of text. However LSA is well known to learn certain meanings such as "pet" by projecting words like "dog" and "cat" onto the same principle component. Semantics is learned by exploring the "loadings" in PCA terminology.
Now LSA is just a linear transformation, but the idea is that same as GPT that uses a wildly more sophisticated non-linear transformation: the assumption is that words (or tokens in the case of GPT) are caused by some latent variables (in the case of LSA, for GPT high dimensional embedding) and most importantly the patterns of word/token usage allow us to infer what those latent variables are.
So it's not absurd and it's how statistical NLP has always worked, only now we have much, much more sophisticated ways to model that latent space. LSA, HMM, and Transformers all assume that the patterns of text usage can be used to model an unobserved latent space that generates the text, which is what we traditionally call "meaning".
Ultimately the limits of GPT-N (unless there are none and it does become AGI) will be valuable in showing the limitations of this fundamental model of semantics.