As someone who admittedly belongs more to the "AI believer" side, I find the vagueness of the training data increasingly frustrating.
The thing that impressed me most about LLMs so far is less the factual correctness or incorrectness of its output but the fact that it appears (!) to understand the instructions that are given. I.e., even if you give it an improbable and outlandish task ("write a poem about kernel debugging in the style of Edgar Allan Poe", "write a script for a Star Trek TNG episode in which Picard only talks in curse words "), it always gives a response which is a valid fulfillment of the task.
Of course it could be that the tasks weren't really as outlandish as they seemed and somewhere in the vast amounts of training data there was already a matching TNG fanfic which just needed some slight adjustments or something.
But those kinds of arguments essentially shift the black box from the model to the training data: Instead of claiming the model has magical powers of intelligence, now the training data magically already contains anything you could possibly ask for. I personally don't find that approach that much more rational that believing in some kind of AI consciousness (or fragments of it).
...but of course it could be. This is why I'd wish for foundation models with more controlled training data, so we can make more certain statements about which responses could be reasonably be pulled from the training data and which would be truly novel.
I suspect when it comes to training data, it may need to be general enough to allow the architecture a chance to learn the meta concept of "learning". Ie identify the latent gestalt within a text corpus that we most identify as "reasoning ability".
If the training data is not rich enough, then these more refined emergent abilities will not be discovered through our current algorithms/architecture. Maybe in the future when more efficient algorithms are found (we know the lower bound must be at least as efficient as our human brains for example) then we won't need as much/as rich data. Or use Multi modal data.
From what we're seeing I believe we can already discount the tainted training data as likely hypothesis, and trend to the suspicion there is something deeper at play.
For instance, what if LLMs through pattern recognition of text alone may have built a coherent enough world model that it yields answers indistinguishable from human intelligence?
It may also suggest there to be nothing special functionally about the human brain; the ability for a system to recursively identify, model, and remix concepts may be sufficient to give rise to the phenomenology we know as intelligence.
Qualia, goals, "feelings", that sounds more nebulous and complicated to define and assess though.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
The thing that impressed me most about LLMs so far is less the factual correctness or incorrectness of its output but the fact that it appears (!) to understand the instructions that are given. I.e., even if you give it an improbable and outlandish task ("write a poem about kernel debugging in the style of Edgar Allan Poe", "write a script for a Star Trek TNG episode in which Picard only talks in curse words "), it always gives a response which is a valid fulfillment of the task.
Of course it could be that the tasks weren't really as outlandish as they seemed and somewhere in the vast amounts of training data there was already a matching TNG fanfic which just needed some slight adjustments or something.
But those kinds of arguments essentially shift the black box from the model to the training data: Instead of claiming the model has magical powers of intelligence, now the training data magically already contains anything you could possibly ask for. I personally don't find that approach that much more rational that believing in some kind of AI consciousness (or fragments of it).
...but of course it could be. This is why I'd wish for foundation models with more controlled training data, so we can make more certain statements about which responses could be reasonably be pulled from the training data and which would be truly novel.