> We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction.
That's easy. Stop training your AIs on cheesy old sci-fi that talks about robot uprisings. In fact, maybe y'all should just stop talking about robot uprisings altogether. Putting a stochastic parrot in charge of an agentic function-calling REPL doesn't somehow make it super-dangerous, except to the extent that dumb mistakes might result in danger. And you can't prevent an AI from making dumb mistakes with burdensome regulation.
The biggest existential risk from AI is its contribution to global climate change. The second biggest risk from AI is the potential for AI-generated disinformation and propaganda to spark, or to manufacture consent for, a world war. The risk of superintelligent paperclip maximizers is so low as to be negligible.
> The risk of superintelligent paperclip maximizers is so low as to be negligible.
Literal paperclips, sure.
But the point of the example was never literal paperclips.
The point is that maximising *any* goal, if it doesn't include what you care about, will annihilate what you care about.
If you don't believe me, consider what you yourself just said about climate change, and why this is a consequence from maximising money spent on data centres.
show me an agent that persists productively in a goal without stopping. Does not exist. LLMs run on gradient descent. The agent is looking for the most efficient way to halt. AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
> show me an agent that persists productively in a goal without stopping. Does not exist.
The stories about agents bankrupting their owners by running too long passed you by?
> LLMs run on gradient descent.
They were *trained on*, they don't run on it.
Know what else is? DNA. A/B testing. Capitalism. Democracy.
> The agent is looking for the most efficient way to halt.
No. They are looking to produce an answer most likely to get a high score on a rating system which itself is another AI, created either manually or by yet another AI but in both cases to approximate what the creators think is "good", which may or may not be what anyone else thinks is "good", hence Grok calling itself Mecha Hitler because Musk is an edgelord.
> AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
Do billionaires ever get satisfied with how much money they have?
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy their constraints.
whether billionaores get stified is imaterial to the fsct they still are constraint bound.
> persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
So, your standard for how risky it is, is simply how competent it is? (And if so, is the paperclip maximiser scenario really it being "successful"?)
That's fine right up until the thing passes an unknown threshold, one which will only be visible in the rear view mirror.
This is a problem in two directions:
1. We have a trend of the maximum complexity of a tasks they can handle growing faster than Moore's law did at its peak.
2. The threshold can be quite small, e.g. the covid-19 virus itself is not what anyone would call smart, ditto HIV, smallpox, and bubonic plague, but a genome is much the same learning system, and they still killed millions each.
> superficially satisfy their constraints.
The "superficially" part is one of the reasons these things can be dangerous. e.g. hopefully nobody at OpenAI actually wanted their wildly-sycophantic version, but yet they created it.
This is in fact the whole reason for the paperclip maximiser scenario: some idiot specifies the constraint "maximise paperclips" and it (in the hypothetical) superficially satisfies this without any consideration of why someone might ask for it.
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy tjeir constraints.
whether billionaores get stified is imaterial to the fsct they still are constraint bound.
> except to the extent that dumb mistakes might result in danger
That "except" goes all the way up to starting WW3. Or a leak from a viral research lab, and by "leak" I mean "mail order" and by "research lab" I mean "the companies who already ship custom DNA and RNA retroviruses": https://duckduckgo.com/?q=companies+who+already+ship+custom+...
If you can prove that simply not training on horror stories would work, it would make a lot of people very happy.
Unfortunately, I don't think it does a single thing to solve, for example, Elon Musk just plain asking some future version of Grok to take over the world for him.
Nor would merely failing to include them in traing data stop certain entire fictional scenarios such as that Doctor Who episode where the android repair bots weren't told that the crew were off-limits as spare parts, or the other Doctor Who episode where the utilitarian robots started killing everyone who was upset because they calculated net positive utility from upset people ceasing to exist. Well, except for the bit where the Doctor saves the day, because they are not real.
That's easy. Stop training your AIs on cheesy old sci-fi that talks about robot uprisings. In fact, maybe y'all should just stop talking about robot uprisings altogether. Putting a stochastic parrot in charge of an agentic function-calling REPL doesn't somehow make it super-dangerous, except to the extent that dumb mistakes might result in danger. And you can't prevent an AI from making dumb mistakes with burdensome regulation.