Given that large language models don't have any actual knowledge (in a human sense) of the data they've been trained on other than raw statistics, can they be said to "reason" about anything?
How sure are we that knowledge in humans is fundamentally different from a language model? How do you generally represent knowledge? Graphs? Patterns with typed place holders? Could those structures be embedded into a language model?
>How sure are we that knowledge in humans is fundamentally different from a language model?
Because humans learn about language in real world contexts, accompanied by multiple sensory streams. LLMs learn about it solely by being exposed to text. Imagine a child kept in a closet who learns about the world solely by reading books, with no indication which are fact and which are fiction, and no exposure to the physical world outside. LLMs cannot reason, only regurgitate probabilistically.
What does grok semantics mean, how would you test this?
I am not trying to argue that language model are close to human level reasoning or whatever, but it is not obvious to me that a language model of some sort is fundamentally unable to to achieve this.
The question of whether language models can "reason" ultimately comes back to the chinese room argument. There aren't any universally accepted resolutions to that argument either, nor whether those limitations (if they exist) are fundamental.
However, the current incarnation of LLMs like ChatGPT fail at straightforward reasoning around things like categorizing sailfish and mathematical proofs. I'm incredibly impressed at how powerful the simplistic methods they use are, but the deficiencies make it clear that it's not abstractly reasoning from definitions in the same way humans are able to.
A fun thing to do: Ask ChatGPT a question, and ask for references. Then try to track down the references. I just asked a simple question about the Thirty Years War, and with the answer it gave me two valid references (including C.V. Wedgewood, yay), and one that was completely made up.
As far as I understand, LLMs have no model of reality to base results on; they're just an enormously complex statistical model of "what is the next word in the series".
> is this all just a game of syntax that _feels_ like semantics from the inside?
The point is that in case of LLMs, there is no mystery here: it is syntactic templates (x) statistics. But do we know that 'syntax' is all there is to our mental reasoning? We don't know this (yet). I have flirted with the notion - heretical for me since I 'believe' in primacy of meaning, not form - and can kinda squint and see meaning possibly being a 'virtual' phenomena and possibly simply a 'narrative' of 'arcs' connecting 'facts' mapped to a syntactic template. That meaning is simply a process. [So, not even "42" ..]
Could be. Does not even have to feel like anything, this could be a independent additional component. This goes into the direction of the Chinese room, how do you figure out if someone understands something? I would say you ask questions and judge by the answers.
The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. There is not much difference, honestly.
My understanding is that it basically interpolates between the most frequent sequences of words, based on context. There is no concept of expressing a rational reasoning underpinning the words. I don't think it is comparable at this stage. It is just something very good at throwing you back segments of text it has found on the internet.
What you are probably missing is that for the model to be competent at predicting the next the word IT HAS to learn how to reason as a human being would.
I don't see why this is. Humans aren't trained to perform a statistical analysis of what the next word is likely to be. We learn language from other humans in social and environmental context. The LLM is just being trained on text.
Nobody is saying these models are trained the same way humans are. I am just saying that during their training task (predicting the next word) they end up developing human reasoning. That of course wasn't obvious before these large models appeared. But now it's pretty clear that's the case. It wouldn't be able to perform so well in so many tasks if it didn't have something resembling reasoning capabilities.
I asked ChatGPT about the Chinese Room and asked for references. The first it came up with was "Minds, Brains, and Programs" by John Searle (1980).
The next two were, "The Chinese Room: Searle and Artificial Intelligence" edited by John Preston and Mark Bishop (2002) (Preston and Bishop did publish Views into the Chinese Room in 2002, but I can't find anything by the given title) and "The Chinese Room Argument" by David Chalmers (2006) (Chalmers has written on the topic, and published three articles in 2006, that's not one of them.
Also:
"Two cities: one, a city of “knights”, and the other of “knaves”. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. He decides to ask the first person he meets where he is. The answer he receives is that he is in the “knaves” city."
produced the result:
"The man cannot be sure that the person he met was telling the truth, because the person could be a knight from the city of knights, who always lie, or a knave from the city of knaves, who always tell the truth.
"To determine which city he is in, the man could ask the second person he meets the same question. If the first person was a knight, then the second person would also be a knight, and would lie about being in the city of knights. If the first person was a knave, then the second person would also be a knave, and would tell the truth about being in the city of knaves.
"By asking the same question to multiple people and analyzing their responses, the man can eventually determine which city he is in. This problem, known as the "Knights and Knaves" puzzle, is a classic example of logical reasoning and problem-solving."
It seems to recognize the "knights and knaves" form of puzzle, but doesn't get the knights and knaves right. And it's not getting the reasoning part at all even given varying prompts.
I don't understand what point you are making. Are you saying that it lacks any reasoning capability because it fails this test? I say try a bunch more tests and then get the average to really have a better sense; one test is meaningless.
How many opportunities would you give someone who, when asked to justify their answer, routinely made up references, and who ignored important parts of a question and whose answer wouldn't work anyway?
As it turns out, we kinda do know what is going on in a LLM, and there's no model of reality for it to base reasoning on in there.
Does that mean that's impossible? No. Does it mean seeing reasoning there is making the same mistake as those who enjoyed chatting with Parry and Eliza? Probably.
I hear this type of opinion expressed frequently by coders. I use ChatGPT pretty much every day (to help with various tasks) and there's just no way to hold on to this kind of belief after you get to "know" it at that level. It clearly can reason, maybe not in exactly the same way as a person. It is superior to the average person just in terms of following a logical argument in many cases. Maybe that is a low bar but I can't help but think that it is much higher than you think.
Try asking it one of Ray Smullyan's logic puzzles, like "There are two cities: one, a city of “knights” who always tell the truth, and the other of “knaves” who always lie. The residents of the two cities like to visit each other.
One day, a man enters one of the cities, but doesn’t know which. What question should he ask to find out?"
I had to add the part about the behavior of knights and knaves, because ChatGPT seems to recognize the puzzle but gets them wrong, and it seems to be completely ignoring the visiting behavior. Further, the answer doesn't really work.
Interpolation is a leaky analogy. With scale these models acquire increasingly sparse discrete-like activation patterns, and there are techniques[1] amounting to pinpointing the exact locations of the knowledge and updating it freely.
It seems that nonlinear interpolation in a rich enough latent space can give you a sufficiently good approximation to reasoning.
This must be about the 50th time I've seen the exact same comment with the exact same response. It's getting very dull. At some point somebody will say "it gets stuff confidently wrong all the time" and someone else will say "so just like people do?"
To call the model "just statistics" is meaningless. To call it "just like a brain" is unsupported. There are better ways to debate this.
Not exactly unsupported. There are a bunch of papers showing a correlation between patterns of brain activation and weights activations inside an AI model when they are given the same task or stimuli. And this correlation increases with AI model accuracy. Doesn't that suggest that AI models are partly mimicking the same human reasoning pathways?
This is an open and active question, but there really isn't any literature that's strongly trusted to show that there is a truly deep relationship between how current machine learning models operate (at the mathematical level) and how brains work. The suggestion is there, no doubt, but it's unclear where that suggestion will ultimately lead. There certainly has been quite a deal of health cross-fertilization between neuroscience and computer science.
It seems save to say that ML has been moving further and further away from what neuroscience suggests, and towards efficient execution on fast GPUs and TPUs. That's in part b/c neuroscience is currently not equipped to explain how human high-level intelligence works.
Agreed! I think one useful direction to take this debate is to admit that in order for you or me to interact with the tool we need to have a mental model of the tool (so we have some idea of how to interact with it).
So which mental model is superior when evaluated under the criterion of "ability to interact productively with the tool"? (a) It's just statistics / stringing together phrases from the internet or (b) it's more like a human brain with vastly superior general knowledge?
Personally, I would put money on (b) resulting in better interactions. This is probably something we could study empirically.
But ultimately I don't really care that much about the philosophical issues, my mental model of it (a more nuanced version of b) has enabled me to use it productively almost every day to do things that I couldn't do before.
Is that what neuroscientist and machine learning people are saying? Because I've heard there are significant differences between actual neurons and digital neural networks. Anyway, I think analyzing knowledge at the level of neurons misses the bigger picture. We learn from interacting with the world and other people via bodies, not vast amounts of textual tokens.
See my comment in this thread about how ChatGPT thinks that sailfish are mammals. I possibly have never read the classification of sailfish explicitly in a book or a documentary, but I could have classified it as a fish any time of day. For one it's in the name, and furthermore I can imagine with high probability that baby sailfish do not drink milk from their mothers.
I don't know if the knowledge in an LLM is stored like in my neurons - I know nothing about neurobiology. But after having used ChatGPT extensively I can certainly tell that ChatGPT does not mobilize knowledge like humans.
It's definitely missing a "module" that humans have: something like an ability to know & admit when it's not certain about something. The way I think of it is: imagine if you constitutionally unable to ever admit that you didn't know something and you were forced to continue talking about that topic. You too would have no choice in this situation but to start spouting nonsense. Once we add this model of uncertainty, I think this type of problem will go away and it will say something like "I'm not sure but ...".
>The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons.
We don't know that this is true. But even if it is, there is a critical difference in the knowledge stored. Human knowledge incorporates direct experience of the world through multiple senses, while an LLM's knowledge is solely derived from text and so misses some critical physical constraints and contexts.
> The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. There is not much difference, honestly.
Do we know this to be true? I guess I'm not aware that we understand very well how memories are stored/recalled in a human brain.
