>Students need to understand what AI is and is not.
Do we really understand what AI is and is not? For example we all know AI can hallucinate, but how does that knowledge help us in practice? If I'm leaning something new I would have to fact check everything which would take about as much time as learning from verified sources.
I just watched a youtube about cruise ships. A near-riot broke out during a movie quote quiz because the AI that made the questions confused Casablanca and Gone With The Wind. If it aint ready for quiz night, it aint ready to teach kids.
People formed their opinions on LLMs in late 2022/early 2023 when ChatGPT was first released and taking the world by storm and then never updated them after nearly 4 years of improvements.
Hell, just being able to search the Internet did MASSIVE things for reducing hallucinations.
Ask a color blind person to spell "red" and "green", then ask them how many red apples are on the table.
LLMs work on tokens. They literally cannot see individual letters. Yet, unlike humans, they can instantly create a tool to solve this task for them and never be stumped by it again.
It is important to understand how this technology works. It isn't magic. Your example is solid. Similar to the Chinese Room.
Do you think your legislators or most users understand tokens and predictions and what is going on? Do you think that those who bombed a school using it know this?
Until frontier models can account for the disparity between what they can do and what the person (or other agent) is asking, we will continue to suffer bad information used like good information.
That tokens cannot count letters like that is not a brain-dead gotcha. Ignoring how humans are using these models is lacking in critical understanding of the real world. People are literally going to die due to humans not understanding how this works (and already have). True brain-dead, and body-dead.
Do we really understand what AI is and is not? For example we all know AI can hallucinate, but how does that knowledge help us in practice? If I'm leaning something new I would have to fact check everything which would take about as much time as learning from verified sources.