Incredible description. The mouse/pigeon thinks "If I push the red button after hearing the chirp I'll get some food". And the human thinks "If I add 'do not guess' the AI will lie less to me".
Yeah. LLMs are great at convincing people they could be amazing, that they're so close to being amazing, if only they just figure out how to bridge that last gap... But that last gap is fundamentally unbridgeable because of the way LLMs work.
It's a very effectively designed human trap, in this and other ways.
