I am extremely alarmed by the number of HN commenters who apparently confuse "is able to generate text that looks like" and "has a", you guys are going crazy with this anthropomorphization of a token predictor. Doesn't this concern you when it comes to phishing or similar things?
I keep hoping it's just short-hand conversation phrases, but the conclusions seem to back the idea that you think it's actually thinking?
How do you know my cat isn't constantly solving calculus problems? I also can't come up with a "mechanistic model" for what it means to do that either.
Further, if your rubric for "can reason with intelligence and have an opinion" is "looks like it" (and I certainly hope this isn't the case because woo-boy), then how did you not feel this way about Mark V. Shaney?
Like I understand that people live learning about the Chinese Room thought experiment like it's high school, but we actually know it's a program and how it works. There is no mystery.
> but we actually know it's a program and how it works. There is no mystery.
You're right, we do know how it works. Your mistake is concluding that because we know how LLMs work and they're not that complicated, but we don't know how the brain works and it seems pretty complicated, therefore the brain can't be doing what LLMs do. That just doesn't follow.
You made exactly the same argument in the opposite direction, asking if my rubric for "can reason with intelligence and have an opinion" is "seems like it", and your rubric for "thinking is not a token predictor driven by matrix multiplications" is "seems like it".
You can make a case for the plausibility of each conclusion, but that's doesn't make it a fact, which is how you're presenting it.
Dude it's a token predictor. This all sounds very nice until you snap back to reality and remember it's a token predictor and you're not a scientist. You're a web developer. You have no evidence, you have no studies, you have no proof. You're making a claim on the basis that everyone has as much understanding of the field as you and that's just wrong.
I'll take your silence as indication that you realize that I'm not making any claims beyond: we have no evidence to support your claims because, as I said from the very beginning, we lack a robust and detailed mechanistic model for what it means to think, so any claims that depend on the assumption that we do have that knowledge are speculation at best.
In fact, I think an even stronger case could be made that prediction is central to how our brains work, and the evidence is the rise of predictive coding models in neuroscience. It's too early still to say what form that prediction takes, but clearly your dismissal of "token prediction" as somehow meaningless or irrelevant to human thinking seems frankly silly.
The "stochastic parrot" crowd keeps repeating "it's just a token predictor!" like that somehow makes any practical difference whatsoever. Thing is, if it's a token predictor that consistently correctly predicts tokens that give the correct answer to, say, novel logical puzzles, then it is a reasoning token predictor, with all that entails.
Then please go ahead and explain how something can solve novel logical puzzles (i.e. ones that are not present in its training set) without some capacity for reasoning. You're claiming that it is "generating texts that looks like ..." - so what is the "..." in this case? I posit that the word that should be placed there is solution, and then you need to explain why that is not ipso facto a demonstration of the ability to reason.
I couldn't agree more. It is shocking to me how many of my peers think something magic is happening inside an LLM. It is just a token predictor. It doesn't know anything. It can't solve novel problems.
I keep hoping it's just short-hand conversation phrases, but the conclusions seem to back the idea that you think it's actually thinking?