If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people who think as you think indeed have left or never joined.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively?
If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
> Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction.
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
> It’s a bit of a self-serving argument, don’t you think?
I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again?
> And how does that relate to the ask for oligopoly licensing within global democracy?
They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism.
> “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant.
Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
They allegedly believe the technology itself is a nuclear weapon tier threat or greater, so why does it matter which lab they are trying to achieve it at?
They're trying to make it not be a threat, and are all scared and afraid that their best efforts to make it harmless are not enough.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator
is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
> No you see, actually I’m the guy that makes sure that only the Palantir and Mossad get the model that can discover iOS 0 days. I’m standing between us and ruin. Ted over there across the open office, he’s the one actually manufacturing the weapon. You’re thinking of him
Ok, so they're uniquely careful, and they also get to that threshold first (whatever it is).
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork.
Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this.
God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture.
This is exactly the opposite of what Dario proposes.
There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry.
What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies with obscure money trails. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested.
With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today.
I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice.
And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
Who should I be more scared of? China, which has been doubling down on open transparent research, or the secretive US companies who are in bed with the most unhinged administration we've ever had and has been actively starting wars?
They're not proposing anything concrete, and when they do, what do you think the proposal will be? Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies? Or are their proposals going to be aimed at everyone running open Chinese models?
And if a threat to the human race does come from AI, it's going to come from OpenAI/Anthropic. Hypercapitalist, secretive, in bed with the government, plus multiple real documented hackings of open source infrastructure already.
We're a hell of a lot safer with China doing the same research out in the open and making it available to anyone. The choice might well be: one or two superintelligent autonomous AIs at OpenAI/Anthropic - or a lot of smaller ones, unable to be controlled but also coming out of a diverse set of environments.
One of those leads to a stable ecosystem where we can all coexist, the other is genuinely terrifying. But make no mistake, from OpenAI/Anthropic this is all motivated by their stock price - when you're in the silicon valley mindset, it distorts your reality. They've convinced themselves that everyone's safer if they stay on top and in control, conveniently ignoring how that benefits them, and I don't believe them for a minute.
Funny you should try and spin the conversation off into climate change to avoid answering the question. Big AIs contribution to climate change likely has a much more tangible route to killing millions of people and destroying society and there's actual evidence and a tangible mechanism for that. But for some strange reason the business media and all the outlets owned by big AI investors aren't interested in long think pieces about that risk. But that would be a lot more credible if that's what all these social media posts about potential apocalypse were referring to
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
> Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from
denial as you can get).
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.