I suspect when it comes to training data, it may need to be general enough to allow the architecture a chance to learn the meta concept of "learning". Ie identify the latent gestalt within a text corpus that we most identify as "reasoning ability".
If the training data is not rich enough, then these more refined emergent abilities will not be discovered through our current algorithms/architecture. Maybe in the future when more efficient algorithms are found (we know the lower bound must be at least as efficient as our human brains for example) then we won't need as much/as rich data. Or use Multi modal data.
From what we're seeing I believe we can already discount the tainted training data as likely hypothesis, and trend to the suspicion there is something deeper at play.
For instance, what if LLMs through pattern recognition of text alone may have built a coherent enough world model that it yields answers indistinguishable from human intelligence?
It may also suggest there to be nothing special functionally about the human brain; the ability for a system to recursively identify, model, and remix concepts may be sufficient to give rise to the phenomenology we know as intelligence.
Qualia, goals, "feelings", that sounds more nebulous and complicated to define and assess though.
I suspect when it comes to training data, it may need to be general enough to allow the architecture a chance to learn the meta concept of "learning". Ie identify the latent gestalt within a text corpus that we most identify as "reasoning ability".
If the training data is not rich enough, then these more refined emergent abilities will not be discovered through our current algorithms/architecture. Maybe in the future when more efficient algorithms are found (we know the lower bound must be at least as efficient as our human brains for example) then we won't need as much/as rich data. Or use Multi modal data.
From what we're seeing I believe we can already discount the tainted training data as likely hypothesis, and trend to the suspicion there is something deeper at play.
For instance, what if LLMs through pattern recognition of text alone may have built a coherent enough world model that it yields answers indistinguishable from human intelligence?
Nothing about that seems improbable from current neuroscience theories https://en.m.wikipedia.org/wiki/Predictive_coding
It may also suggest there to be nothing special functionally about the human brain; the ability for a system to recursively identify, model, and remix concepts may be sufficient to give rise to the phenomenology we know as intelligence.
Qualia, goals, "feelings", that sounds more nebulous and complicated to define and assess though.