Mainstream LLMs are trained on tests that contain vast amounts of falsehood, so even if they did work by memorizing facts, they would be memorizing counter-facts.
Pick any topic in which misinformation is spread by crackpots. (Especially a topic that is not "policed" by the system prompt). Ask questions using terminology, phrases and ideas that the crackpots use, and you will tend to get crackpot answers; the AI will confirm crackpot theories, backd by links to crackpot forum posts.
What we have is a technology that can reproduce sentences that are grammatical with a very high probability, paragraphs that are coherent with a high probability and true with a lower probability if all the input training data is true. Then, true with an even lower probability when the input data contains vast numbers of untrue sentences, as is the case for all the mainstream, cloud-hosted AI public offerings.
LLMs are not (and ever were) expected to produce truth. Just like humans. Most of the time humans have no idea about what they are talking about, they are just repeating patterns. Just like LLMs.
LLMs are doing what they are expected to be doing -- imitating humans, and they are very good at it, because humans are essentially doing the same thing -- imitating other humans. Kids imitate their parents, students imitate their teachers, employers imitate their boss, fans imitate their popstar.
It's hard to even have this conversation, because it's so bloody obvious that it is hard to understand why this even needs an explanation.
Just remember when was the last time you actually read the manual for a program rather than tried to fit your use-case into an example.
Generating original content and understanding anything from the first principles is not just hard, it's prohibitively hard. Nobody learns arithmetic by reading a book on Peano arithmetic, people just imagine putting some apples in a pot, which is the same principle -- analogy/ imitation.
Many people believe in crackpot theories, therefore an LLM has to faithfully imitate them.
Pick any topic in which misinformation is spread by crackpots. (Especially a topic that is not "policed" by the system prompt). Ask questions using terminology, phrases and ideas that the crackpots use, and you will tend to get crackpot answers; the AI will confirm crackpot theories, backd by links to crackpot forum posts.
What we have is a technology that can reproduce sentences that are grammatical with a very high probability, paragraphs that are coherent with a high probability and true with a lower probability if all the input training data is true. Then, true with an even lower probability when the input data contains vast numbers of untrue sentences, as is the case for all the mainstream, cloud-hosted AI public offerings.