I think it does not matter in this case. ALL current LLMs are prone to confabulation. AFAIK, there is not a single model which is able to tell “I don’t know” instead of making things up.
> ALL current LLMs are prone to confabulation. AFAIK, there is not a single model which is able to tell “I don’t know” instead of making things up.
I can't help but feel that people who keep parroting these lines haven't even bothered to take 30 days out of their lives to actually learning the basics of how LLMs work. LLMs are not "prone" to confabulation. They confabulate by design. That's how they work. They predict the next token based on probabilities derived from training data, not some magical ability to discern truth from falsehood.
The critical step people seem to ignore is that after the LLM generates a response, you need a filter to determine whether it's correct or not. But this is easier said than done! It can be done but in a very narrow range of cases, like generating code that passes a specific set of test cases or producing a Lean proof. The tests or proof verifiers act as that filter. But even here, you're far from guaranteed perfection. Test cases don't cover everything and proofs might be valid but still irrelevant to the problem.
Expecting an LLM to just say "I don’t know" fundamentally misunderstands what these models are. They're not epistemic agents. They don't "know" or "not know" anything. They just generate statistically plausible sequences. If that seems like a flaw to you, the problem isn't the LLM, it's your unrealistic expectations.
> The critical step people seem to ignore is that after the LLM generates a response, you need a filter to determine whether it's correct or not.
And therein lies the problem. A vast swath of human users tend to turn off their brain when interacting with LLMs. They expect correctness from computers and thus do not understand they produce plausible looking text, not the truth.
I agree with your thesis, your comment is correct in all its technical details, but making everyone understand those very important points and act accordingly is a continuous uphill struggle.
| ... "but making everyone understand those very important points and act accordingly is a continuous uphill struggle."
And that right there is exactly why people who know keep writing articles and getting in arguments with people who don't know but think they do because some "expert" who also doesn't actually know anything about "AI" beyond the hype they've been sold said so... Until the "bubble pops" and the hype dies down, folks are gonna keep parroting various mistakes and misunderstandings and folks who actually understand the technology are gonna have to keep repeating the same old tired arguments... Seen it time and again over my decades in the "tech" industry. This time is once again proving to be a carbon copy of the same old script we've seen played out countless times before with each new "next big thing". Surprise!
