>The comment is written in such a way that it assumes this is the nature of reality and at some point, we will learn that the human mind is no different from the parlor tricks of LLMs.
Again your whole point seems to boil down to simply denying physicalism (unless you have something more subtle in mind about the limits of physical science). It's a valid position to take but not very fruitful for further conversation.
No, I'm not talking about physicalism at all. I'm talking about reductionism and emergence. Even if it were shown that human thought processes operate at lower levels in ways similar to machines, that does not imply that human beings are equivalent to machines. That is a reductive argument.
Now I don't understand at all anymore why you're disagreeing so strongly. I don't think anyone has proven that it's inherent to emergent properties that they can't be understood or explained in detail. One just has to explain the mechanisms of emergence in addition to explaining the component parts reductively (and if that is your objection to reductionist arguments I would be inclined to agree with you). But such an explanation of how intelligence can emerge from simple component parts is exactly what could potentially be provided by a better understanding of AI systems.
Again your whole point seems to boil down to simply denying physicalism (unless you have something more subtle in mind about the limits of physical science). It's a valid position to take but not very fruitful for further conversation.