This is just a variation on the "can submarines swim" question. Unask the question.
Can an LLM produce output that matches the written aspect of what humans call reasoning? Obviously yes. Are there limits? Yes, but individual humans have limits too.
It's a funny phrase people use to somehow discredit AI, like it isn't really 'thinking', AI 'thinking' is like a submarine 'swimming'. But doesn't really provide much insight.
Let's say a submarine can't swim. Thus, by inference, AI can't think.
So what, the submarine still beats every human in every swim competition. The submarine dominates the humans, literally destroys them, churns them into chum. No human can beat a submarine swimming.
And thus also with AI 'thinking'. OK by funny analogy, AI does not 'think'.
Again, so what, if whatever it is doing is completely dominating humans. We can argue all day that AI is not 'thinking' can't 'reason', but if it is doing whatever it is doing better than humans, then it still destroys the humans.
LLMs reason to the extent they are allowed to. You could say that they are overfitting when it comes to reasoning. They weren't trained to reason to begin with, so the bigger surprise is that they can do it within limits.
I agree about the surprise -- LLMs can do many surprising things that frankly are astonishing given their architecture. The fact that they can produce output that is difficult to distinguish from the output of an actor that we "know" can reason is pretty astonishing too.
I don't agree with the idea of "the extent they are allowed to"; that's giving the creators of LLMs waaay more agency than they have in reality. These things have already escaped the bounds of what we thought they could do, I don't think we have a realistic way of constraining their behavior in a deterministic way (other than maybe cutting down the context length).
I mean, since you're a human and can, by assumption, perform new reasoning that hasn't been done before, let's fire up the old noggin and come up with some examples and we'll see how the old LLM does compared to humans.
For preregistration purposes, do not try any of your new reasoning examples in an LLM before putting them in your response. Once you have, say, five examples of new reasoning and have posted them, then we'll all try plugging them in and/or answering them using our human-brains-that-are-capable-of-pure-reason.
One is a mimic the other is actually applying the laws of physics in both cases. You are arguing based on utility and similarities with actual reasoning. But can it really reason when presented with novel complicated cases? That's the question.
> One is a mimic the other is actually applying the laws of physics in both cases. You are arguing based on utility and similarities with actual reasoning. But can it really reason when presented with novel complicated cases? That's the question.
This is a confusing reply. What's not the same as what? One is a mimic -- are we talking about fish or submarines or minds or LLMs? What are both cases? What is applying the laws of physics?
A bird and a submarine both apply the same underlying laws of physics.
Just because LLMs seems to reason does not necessarily imply that it is able to actually reason. An airplane can't fake flying, it either flys or it doesn't. With text, reasoning can be faked.
My argument here is that the discussion of LLMs reasoning is a semantic question, not a technical or scientific one. I specifically didn't use "flying" because both airplanes and birds fly, and "fly" generally means "move through the air" regardless of method.
Swim means "move through water" but with the strong connotation is "move through water in the way that living things move through water". Submarines move through water but they do not swim.
Reason means what -- something like "arrive at conclusions", but with a strong connotation of "arrive at conclusions as living things do", and a weak connotation of "use logic and step-by-step thinking to arrive at conclusions". So the question is, what aspect of "reasoning" is tied to the biological aspect of reasoning (that is, how animals reason) vs. a general sense of arriving at conclusions. Don't try to argue a definition of "reason" that is different than mine -- doing so makes it immediately apparent that we're just playing with semantics. The question is "what observable behavior does a thing that we all agree can 'reason' have that LLMs do not have?". And the related question is "to what degree does humans' ability to 'reason' reflect our ideal conception of what it means to 'reason' using logic".
Both the statements "LLMs can reason" and "LLMs cannot reason" are "not even wrong"[1]
I use LLMs daily in coding and it very clear to me (as a humble average thinking machine) that it is an approximation of reasoning and very close one. But the mistakes it makes clearly shows that this system does not really understand, it is not very different from the mistakes those who memorized text do. Humans, when they really think, they think differently. That is why, I would never expect the current architecture of LLMs to come up with a something like special relativity or any novel idea, because again it doesn't not really reasonn the same way deep thinkers, philosophers do. However, most knowledge work does not require that much depth in reasoning, hence LLMs wide adoption.
Can an LLM produce output that matches the written aspect of what humans call reasoning? Obviously yes. Are there limits? Yes, but individual humans have limits too.