Isn’t it fundamentally impossible to model a highly entropic system using deterministic methods?
My point is that animal brains are entropic and “designed” to model entropic systems, where as computers are deterministic and actively have to have problems reframed as deterministic so that they can solve them.
All of the issues mentioned in the article boil down to the fundamental problem of trying to get deterministic systems to function in highly entropic environments.
LLMs are working with language, which has some entropy but is fundamentally a low entropy system, and has orders of magnitude less entropy than most peoples’ back garden!
As the saying goes, to someone with a hammer, everything looks like a nail.
Not fundamentally, at least I doubt it : pseudo-random number generation is technically deterministic.
And it's used for sampling these low information systems that you are mentioning.
(And let's not also forget how they are helpful in sampling deterministic but extremely high complexity systems involving a high amount of dimensions that Monte Carlo methods are so good at dealing with.)
Isn’t it fundamentally impossible to model a highly entropic system using deterministic methods?
My point is that animal brains are entropic and “designed” to model entropic systems, where as computers are deterministic and actively have to have problems reframed as deterministic so that they can solve them.
All of the issues mentioned in the article boil down to the fundamental problem of trying to get deterministic systems to function in highly entropic environments.
LLMs are working with language, which has some entropy but is fundamentally a low entropy system, and has orders of magnitude less entropy than most peoples’ back garden!
As the saying goes, to someone with a hammer, everything looks like a nail.