You're essentially arguing that AGI can't exist, and you're assuming when people say "AI will turn the world to paperclips" they mean a really smart go-playing neural net could do that. Of course it can't.
Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI.
It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.
Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits.
We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before.
And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.
What you say is true for silicon, but Turing complete molecular machinery is already out there, we just have yet to master its intricacies.
The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.
This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?
That's more worrying than silicon AGI, which I think is a non-starter.
The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.
We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.
Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.
> We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.
This is mostly true because of current economical realities in the IT sector.
If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.
So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").
Even software developed with effectively unlimited resources is rather buggy. The closest humanity has ever come to truly reliable software was probably the Space Shuttle flight control system, and that code was relatively simple in a highly constrained problem domain. There is zero evidence that "serious methodologies" would get us any closer to a working AGI.
Oh, definitely. I am not saying that not doing the normal agile crap for development brings us closer to AGI. What I would venture to say though is that trying stuff like formal verification and finding a way to integrate it in a daily workflow would definitely help with the average quality of software developed through such a process. And that maybe the private labs are more liberal and let people's creativity achieve results.
Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.
Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.
>Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.
in theory, sure.
But we don't know much at all about how brains operate, as far as I know, or how to construct artificial brain-like machines; for that matter, as far as I know, the only ways we've been able to connect a brain made in the traditional way to a computer is through existing senses or through emulating existing senses in a way that isn't higher bandwidth.
(Like cochlear implants are fucking amazing.... but it turns out, as far as I understand, that they aren't any higher bandwidth than regular ears. There's also been similar work that is semi-successful at giving people something like sight... but none of these technologies give me the ability, say, to acquire textual data faster than reading a book.)
I think you need to distinguish between AGI and the singularity.
We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.)
The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.
If we take humans as our model for AGI, it's worth noting that well-established networks of humans have proved efficient at ahem... deleting more-intelligent but potentially threatening human minds, that status and EQ has tended to prevail over IQ when it comes to manipulating humans into acting against their interests, that relatively close physical copies of human minds often have very dissimilar goals and abilities so self-replicating AGIs might ultimately be self-defeating... but above all that we've already got multiple threats to humanity in the form of human intelligences, augmented by technology that's often as dumb as it is deadly.
While that is true, also consider the destruction we have caused to other species.
AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.
And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.
Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.
(6) It is easier to move a problem around (for example, by moving
the problem to a different part of the overall network
architecture) than it is to solve it.
In other words, saying "it's the definition that we don't have, not the implementation" just pushes the magic smoke around.
I don't think GP means "magic smoke" as in actual magic, some new physics. It's more like we just have few potential building blocks (e.g. pattern matching), but we're not sure if we have all of them, and we definitely don't know how to connect them together to create sentience.
(And no, more GPUs running faster DNNs doesn't seem like an answer.)
I thought the idea was to create an AI that was way smarter than a human? I mean, if you just have human level AI, that's not gonna cause the singularity unless there aren't any physical limits preventing it from scaling itself up, and scaling itself up a lot
A group of persons can do many more than a person can do, even with relatively low bandwidth interconnections (speech, text, diagrams, gestures), coordination problems, and economics that has to waste resources on members' incentives (it's not wasted by majority opinion of the said members, of course).
Even in off-chance that you can't improve individual AIs beyond top humans, AI community can be way more intelligent and efficient than humanity, as it is possible to remove the limitations I wrote above. AI society can function as very efficient war economy. War economy shows that even humans can be persuaded/indoctrinated/brought up/whatever to set aside their inessential needs and work for common goal. And AIs will be necessarily more malleable.
So I can't see the idea that AI will be safe by default other than wishful thinking.
>So I can't see the idea that AI will be safe by default other than wishful thinking.
I... don't think anyone is arguing that AI is safe in any way? I mean, the pattern matching tools we call AI now are already deployed in very dangerous weapons.
My point is not that AI is going to be safe and warm and fuzzy, or even that it won't be an existential threat. My point is that we're already facing several existential threats that don't require additional technological development to destroy society, and because the existing threats don't require postulating entirely new classes of compute machinery, we should probably focus on them first.
There are still enough nuclear weapons in the world for a global war to end society and most of us. A conflict like we saw in the first half of the 20th century using modern weapons would kill nearly all of us, and we are seeing a global resurgence of early 20th century political ideas. I think this is the biggest danger to the continuation of humanity right now.
We haven't killed ourselves yet... but the danger is still there. we still have apocalyptic weapons aimed and ready to go at a moment's notice. We still face the danger of those weapons becoming cheaper and easier to produce as we advance industrially.
> pattern matching, logical inference and post-hoc justification.
I think humans do way more than that. As the saying goes, "the easiest person to fool is yourself." Humans are able to lie to themselves to the point that they don't realize they are lying to themselves. They're able to create realities that no one else sees but themselves.
There is a lot in 'logical inference' that we still don't really know how to describe well enough to tell computers how to do it. (I mean, I think it's super interesting how we figured out how to teach computers to do pattern matching... but I think logical inference is a different sort of thing that won't fall to the same tools)
But really, I think your idea that we're mostly 'post-hoc justification' is interesting. I mean, I've read some research to that effect, too; but it's... creepy and doesn't line up with anything reasonable about free will or really about long-term planning.
Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI.
It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.