People seem to get really hung up on the fact that words have meaning to them, in regards to thinking about what an LLM is doing.
It creates all sorts of illusions about the model having a semantic understanding of the training data or the interaction with the users. It's fascinating really how easily people suspend disbelief just because the model can produce output that is meaningful to them and semantically related to the input.
It's a hard illusion to break. I was discussing usage of LLM by professors with a colleague who teaches at a top European university, and she was jarred by my change in tone when we went from "LLMs are great to shuffle exam content" (because it's such a chore to do it manually to preclude students trading answers with people who have already taken a course) to "LLMs could grade the exam". It took some back and forth for me to convince her that language models have no concept of factuality and that some student complaining about a grade and resulting in "ah ok I've reviewed it and previously I had just used an LLM to grade it" might be career ending.
I think there's a strong case to be made that the detailed map is indeed the land it maps.
Or that one can construct a surprisingly intuitive black box out of a sufficiently large pile of correlations.
Because what is written language, if not an attempt to map ideas we all have in our heads into words? So inversely, should there not be a statistically-relevant echo of those ideas in all our words?
It creates all sorts of illusions about the model having a semantic understanding of the training data or the interaction with the users. It's fascinating really how easily people suspend disbelief just because the model can produce output that is meaningful to them and semantically related to the input.
It's a hard illusion to break. I was discussing usage of LLM by professors with a colleague who teaches at a top European university, and she was jarred by my change in tone when we went from "LLMs are great to shuffle exam content" (because it's such a chore to do it manually to preclude students trading answers with people who have already taken a course) to "LLMs could grade the exam". It took some back and forth for me to convince her that language models have no concept of factuality and that some student complaining about a grade and resulting in "ah ok I've reviewed it and previously I had just used an LLM to grade it" might be career ending.