Hacker News new | past | comments | ask | show | jobs | submit login
Statement on AI Risk (safe.ai)
341 points by zone411 10 months ago | hide | past | favorite | 921 comments



This reeks of marketing and a push for early regulatory capture. We already know how Sam Altman thinks AI risk should be mitigated - namely by giving OpenAI more market power. If the risk were real, these folks would be asking the US government to nationalize their companies or bring them under the same kind of control as nukes and related technologies. Instead we get some nonsense about licensing.


I'm eternally skeptical of the tech business, but I think you're jumping to conclusions, here. I'm on a first-name basis with several people near the top of this list. They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met. These folks default to skepticism of the tech business, champion open data, are deeply familiar with the risks of regulatory capture, and don't sign their name to any ol' open letter, especially if including their organizational affiliations. If this is a marketing ploy, that must have been a monster because even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.


Maybe these people have good intentions and are just being naive

They might not be getting paid, but that doesn’t mean they are not being influenced

AI at this point is pretty much completely open, all the papers, math and science behind it are public

Soon, people will have advanced AI running locally on their phones and watches

So unless they scrub the Internet, start censoring this stuff, and pretty much ban computers, there is absolutely no way to stop AI nor any potentially bad actors from using it

The biggest issues that we should be addressing regarding AI are the potential jobs losses and increased inequality at local and global scale

But of course, the people who usually make these decisions are the ones that benefit the most from inequality, so


You are attributing naivety on a technical matter to many people who have done world class research and some of them are even a pioneer of their subfield of study.

To do that level of research, one needs strong independent thinking skills and deep technical expertise. We have plenty of evidence on this very site how hard it is to influence technical people and change their opinion.


Not attributing anything, just pointing out an alternative explanation to the one that it is given in the comment that it is responding to

However, you seem to be attributing super-human capabilities to a whole group of people for some reason


One does not need to be have super-human capabilities to foresee potential risks of the technology one is an expert in.

Yes, the current version of AI is not capable of large scale harm by itself yet, but the plausible trajectory is worth warning about. Gordon Moore did make fairly accurate predictions after all.


> One does not need to be have super-human capabilities to foresee potential risks of the technology one is an expert in

You are right. And no one is arguing the contrary

We can all see the risks

The thing is, what do we do about it

Regulating AI will only serve to help the big players and keep others out


That would depend on the specifics of the regulation though.

I do agree that the likelihood of regulatory capture in this arena is very high.


> Soon, people will have advanced AI running locally on their phones and watches

Yeah, with drugs being legal now I've been wondering what the next boogeyman we're going to declare war on would be.


You don't have to be a mustache-twirling villain to have the same effect.


There's range of functionality in software technology, i.e. some new framework, may have been outputed fully developed and internally debugged for years in some startup, or you could have a really almost not even a production level software, just uploaded to github.

In most projects, many unmatured software is being used just because works for some specific task, nowadays.

With that in mind, it is easy to hyphotetize that many projects are already using LLMs internally. Not always for good deeds, easily, one of the most and best use cases, is to use LLMs to command & control distributed malware.

So there you go, it maybe possible that some beta level of intelligent malware is already roaming the Internet right now. We'll know for sure in some years from now (the usual time for advanced malware to be discovered is somewhat 2-3 years after it has been took it to production).


>Maybe these people have good intentions and are just being naive

Ive noticed a lot of good people take awful political positions this way.

Usually they trust the wrong person - e.g. by falling victim to the just world fallacy ("X is a big deal in our world and X wouldn't be where they are if they werent a decent person. X must have a point.")


I sure when openAI converted to successful company, we had lost the game. AGI had won whole world. openAI was the organisation should stop AI damage.


It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research programs if AI safety becomes an official policy.

For a large number of them, these risks are worth far more than any possible gain from signing it.

When a large number of smart, reputable people, including many with expert knowledge and little or negative incentives to act dishonestly, put their names down like this, one should pay attention.

Added:

Paul Christiano, a brilliant theoretical CS researcher who switched to AI Alignment several years ago, put the risks of “doom” for humanity at 46%.

https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...


Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left.

Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing.

Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).


Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies.

We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.


> There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well.

I know the Stanford researchers the most and the “biggest names” in LLMs from HAI and CRFM are absent. It would be useful to have their perspective as well.

I’d throw MetaAI in the mix as well.

Merely pointing out that healthy skepticism here is not entirely unwarranted.

> We’re talking about nuclear war level risks here.

Are we? This seems a bit dramatic for LLMs.


LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence.

Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do?

More on the risks of “doom”: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...


I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI.

This is hardly the first time in history a new technological advancement may be used for nefarious purposes.

It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now.

To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.


> I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI.

That's what interesting to me. People fearmongering about bioengineering and GMO's were generally dismissed as being anti-science and holding humankind back (or worse, that there opposition to progress meant they had blood on their hands). Yet many of the people who mocked them proved themselves to be even more dogmatic and apocalyptic, while being much closer to influencing regulations. And the technology they're fear-mongering about is even further from being able to harm people than biotech is. We are actually able to create harmful biotech today if we want; we don't know when we'll ever be able to create AGI, and if it would even pose a danger if we did.

This mentality - "there could be a slight chance research into this could eventually lead to apocalyptic technology, no I don't have any idea how but the danger is so great we need a lot of regulation" - would severely harm scientific growth if we applied it consistently. Of course everyone is going to say "the technology I'm afraid of is _actually_ dangerous, the technology they're afraid if isn't." But we honestly have no clue when we're talking about technology that we have no idea how to create at the moment.


Counterpoint: CRISPR only reignited what was already a real fear of reduced difficulty and costs of engineering deadly pathogens.

In fact, what you and GP wrote is baffling to me. The way I see it, biotech is stupidly obviously self-evidently dangerous, because let's look at the facts:

- Genetic engineering gets easier and cheaper and more "democratized"; in the last 10 years, the basics were already accessible to motivated schools and individual hobbyists;

- We already know enough, with knowledge accessible at the hobbyist level, to know how to mix and match stuff and get creative - see "synthetic biology";

- The substrate we're working with is self-replicating molecular nanotechnology; more than that, it's usually exactly the type that makes people get sick - bacteria (because they're most versatile nanobots) and viruses (because they're natural code injection systems).

Above is the "inside view"; for "outside view", I'll just say this: the fact that "lab leak" hypothesis of COVID-19 was (or still is?) considered to be one of the most likely explanations for the pandemics already tells you that the threat is real, and consequences are dire.

I don't know how can you possibly look at that and conclude "nah, not dangerous, needs to be democratized so the Evil Elites don't hoard it all".

There must be some kind of inverse "just world fallacy" fallacy of blaming everything on evil elites and 1%-ers that are Out To Get Us. Or maybe it's just another flavor of the NWO conspiracy thinking, except instead the Bildenbergs and the Jews its Musk, Bezos and the tech companies.

Same is, IMHO, with AI. Except that one is more dangerous because it's a technology-using technology - that is, where e.g. accidentally or intentionally engineered pathogens could destroy civilization directly, AI could do it by using engineered pathogens - or nukes, or mass manipulation, or targeted manipulation, or ... countless other things.

EDIT:

And if you ask "why, if it's really so easy to access and dangerous, we haven't already been killed by engineered pathogens?", the answer is a combination of:

1. vast majority of people not bearing ill intent;

2. vast majority of people being not interested and not able to perform (yet!) this kind of "nerdy thing";

3. a lot of policing and regulatory attention given to laboratories and companies playing with anything that could self-replicate and spread rapidly;

4. well-developed policies and capacity for dealing with bio threats (read: infectious diseases, and diseases in general);

5. this being still new enough that the dangerous and the careless don't have an easy way to do what in theory they already could.

Note that despite 4. (and 3., if you consider "lab leak" a likely possibility), COVID-19 almost brought the world down.


Great points. Will just add a point 1.5: There's usually an inverse correlation between ill intent and competence, so the subset of people who both want to cause harm to others on a mass scale and who are also able to pull it off is small


I’m not sure there is a way for someone to engineer a deadly virus while completely innoculating themselves from it.

Short-term AI risk likely comes from a mix of malicious intent and further autonomy that causes harm the perpetrators did not expect. In the longer run, there is a good chance of real autonomy and completely unexpected behaviors from AI.


Why do you have to inoculate yourself from it to create havoc? Your analogy of “nuclear war” also has no vaccine.

AI autonomy is a hypothetical existential risk, especially in the short term. There are many non-hypothetical existential risks including actual nuclear proliferation and escalating great power conflicts happening right now.

Again my point being that this is an important discussion but appears overly dramatized, just like there are people screaming doomsday there are also equally qualified people (like Yann LeCun) screaming BS.

But let’s entertain this for a second, can you posit a hypothetical where in the short term a nefarious actor can abuse AI or autonomy results in harm? How does this compare to non-AI alternatives for causing harm?


This gets countered by running one (or more) of those same amazing autonomous agents locally for your own defense. Everyone's machine is about to get much more intelligent.


“…some intelligence…” appears to be a huge leap from where we seem to be though.


> > We’re talking about nuclear war level risks here.

> Are we? This seems a bit dramatic for LLMs.

The signed statement isn't about just LLMs in much the same way that "animal" doesn't just mean "homo sapiens"


I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their whitepaper.

Semantics aside the recent interest in AI risk was clearly stimulated by LLMs and the camp that believes this is the path to AGI which may or may not be true depending who you ask.


I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively.

They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.


While true the ones doing media tours and speaking the most vocally in May 2023 are the LLM crowd.

I don’t think it’s a mischaracterization to say OpenAI has sparked public debate on this topic.


To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands.

A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers.

The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"


> I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively.

Whenever Yudkowsky comes up on my Twitter feed I'm left with an impression that I'm not going to have any more luck conversing AI with those in his orbit than I am discussing the rapture with a fundamentalist Christian. For example, the following Tweet[1]. If a person believes this is from a deep thinker that should be taken very seriously rather than an unhinged nutcase, our worldviews are probably too far apart to ever reach a common understanding:

> Fools often misrepresent me as saying that superintelligence can do anything because magic. To clearly show this false, here's a concrete list of stuff I expect superintelligence can or can't do:

> - FTL (faster than light) travel: DEFINITE NO

> - Find some hack for going >50 OOM past the amount of computation that naive calculations of available negentropy would suggest is possible within our local volume: PROBABLE NO

> - Validly prove in first-order arithmetic that 1 + 1 = 5: DEFINITE NO

> - Prove a contradiction from Zermelo-Frankel set theory: PROBABLE NO

> - Using current human technology, synthesize a normal virus (meaning it has to reproduce itself inside human cells and is built of conventional bio materials) that infects over 50% of the world population within a month: YES

> (note, this is not meant as an argument, this is meant as a concrete counterexample to people who claim 'lol doomers think AI can do anything just because its smart' showing that I rather have some particular model of what I roughly wildly guess to be a superintelligence's capability level)

> - Using current human technology, synthesize a normal virus that infects 90% of Earth within an hour: NO

> - Write a secure operating system on the first try, zero errors, no debugging phase, assuming away Meltdown-style hardware vulnerabilities in the chips: DEFINITE YES

> - Write a secure operating system for actual modern hardware, on the first pass: YES

> - Train an AI system with capability at least equivalent to GPT-4, from the same dataset GPT-4 used, starting from at most 50K of Python code, using 1000x less compute than was used to train GPT-4: YES

> - Starting from current human tech, bootstrap to nanotechnology in a week: YES

> - Starting from current human tech, bootstrap to nanotechnology in an hour: GOSH WOW IDK, I DON'T ACTUALLY KNOW HOW, BUT DON'T WANT TO CLAIM I CAN SEE ALL PATHWAYS, THIS ONE IS REALLY HARD FOR ME TO CALL, BRAIN LEGIT DOESN'T FEEL GOOD BETTING EITHER WAY, CALL IT 50:50??

> - Starting from current human tech and from the inside of a computer, bootstrap to nanotechnology in a minute: PROBABLE NO, EVEN IF A MINUTE IS LIKE 20 SUBJECTIVE YEARS TO THE SI

> - Bootstrap to nanotechnology via a clean called shot: all the molecular interactions go as predicted the first time, no error-correction rounds needed: PROBABLY YES but please note this is not any kind of necessary assumption because It could just build Its own fucking lab, get back the observations, and do a debugging round; and none of the processes there intrinsically need to run at the speed of humans taking hourly bathroom breaks, it can happen at the speed of protein chemistry and electronics. Please consider asking for 6 seconds how a superintelligence might possibly overcome such incredible obstacles of 'I think you need a positive nonzero number of observations', for example, by doing a few observations, and then further asking yourself if those observations absolutely have to be slow like a sloth

> - Bootstrap to nanotechnology by any means including a non-called shot where the SI designs more possible proteins than It needs to handle some of the less certain cases, and gets back some preliminary observations about how they interacted in a liquid medium, before it actually puts together the wetware lab on round 2: YES

(The Tweet goes on, you can read the rest of it at the link below, but that should give you the gist.)

[1] https://twitter.com/ESYudkowsky/status/1658616828741160960


I've already read that thread.

I don't have twitter and I agree his tweets have an aura of lunacy, which is a shame as he's quite a lot better as a long-form writer. (Though I will assume his long-form writings about quantum mechanics is as bad as everyone else unless a physicist vouches for them).

But, despite that, I don't understand why you chose that specific example — how is giving a list of what he thinks an AI probably can and can't do, in the context of trying to reduce risks because he thinks loosing is the default, similar to a fundamentalist Christian who wants to immanentize the eschaton because the idea the good guys might lose when God is on their side is genuinely beyond comprehension?


Id like to see the equation that led to this 46%. Even long time researchers can be overcome by grift


Some of the academics who signed are either not doing AI research e.g climatologists, genomics, philosophy. Or they have Google connections that aren't disclosed. E.g. Peter Norvig is listed as Stanford University but ran Google Research for many years, McIlrath is associated with the Vector Institute which is funded by Google.


I just took that list and separated everyone that had any commercial tie listed, regardless of the company. 35 did and 63 did not.

> "Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left."

You're putting a lot of effort into painting this list in a bad light without any specific criticism or evidence of malfeasance. Frankly, it sounds like FUD to me.


I’m not painting anything, if a disclosure is needed to present a poster at a conference it’s reasonable to want one when calling for regulation.

Note my comments are non-accusatory and only call for more transparency.


With corporate conflicts (that I recognized the names of):

Yoshua Bengio: Professor of Computer Science, U. Montreal / Mila, Victoria Krakovna: Research Scientist, Google DeepMind, Mary Phuong: Research Scientist, Google DeepMind, Daniela Amodei: President, Anthropic, Samuel R. Bowman: Associate Professor of Computer Science, NYU and Anthropic, Helen King: Senior Director of Responsibility & Strategic Advisor to Research, Google DeepMind, Mustafa Suleyman: CEO, Inflection AI, Emad Mostaque: CEO, Stability AI, Ian Goodfellow: Principal Scientist, Google DeepMind, Kevin Scott: CTO, Microsoft, Eric Horvitz: Chief Scientific Officer, Microsoft, Mira Murati: CTO, OpenAI, James Manyika: SVP, Research, Technology & Society, Google-Alphabet, Demis Hassabis: CEO, Google DeepMind, Ilya Sutskever: Co-Founder and Chief Scientist, OpenAI, Sam Altman: CEO, OpenAI, Dario Amodei: CEO, Anthropic, Shane Legg: Chief AGI Scientist and Co-Founder, Google DeepMind, John Schulman: Co-Founder, OpenAI, Jaan Tallinn: Co-Founder of Skype, Adam D'Angelo: CEO, Quora, and board member, OpenAI, Simon Last: Cofounder & CTO, Notion, Dustin Moskovitz: Co-founder & CEO, Asana, Miles Brundage: Head of Policy Research, OpenAI, Allan Dafoe: AGI Strategy and Governance Team Lead, Google DeepMind, Jade Leung: Governance Lead, OpenAI, Jared Kaplan: Co-Founder, Anthropic, Chris Olah: Co-Founder, Anthropic, Ryota Kanai: CEO, Araya, Inc., Clare Lyle: Research Scientist, Google DeepMind, Marc Warner: CEO, Faculty, Noah Fiedel: Director, Research & Engineering, Google DeepMind, David Silver: Professor of Computer Science, Google DeepMind and UCL, Lila Ibrahim: COO, Google DeepMind, Marian Rogers Croak: VP Center for Responsible AI and Human Centered Technology, Google

Without:

Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto, Dawn Song: Professor of Computer Science, UC Berkeley, Ya-Qin Zhang: Professor and Dean, AIR, Tsinghua University, Martin Hellman: Professor Emeritus of Electrical Engineering, Stanford, Yi Zeng: Professor and Director of Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Xianyuan Zhan: Assistant Professor, Tsinghua University, Anca Dragan: Associate Professor of Computer Science, UC Berkeley, Bill McKibben: Schumann Distinguished Scholar, Middlebury College, Alan Robock: Distinguished Professor of Climate Science, Rutgers University, Angela Kane: Vice President, International Institute for Peace, Vienna; former UN High Representative for Disarmament Affairs, Audrey Tang: Minister of Digital Affairs and Chair of National Institute of Cyber Security, Stuart Russell: Professor of Computer Science, UC Berkeley, Andrew Barto: Professor Emeritus, University of Massachusetts, Jaime Fernández Fisac: Assistant Professor of Electrical and Computer Engineering, Princeton University, Diyi Yang: Assistant Professor, Stanford University, Gillian Hadfield: Professor, CIFAR AI Chair, University of Toronto, Vector Institute for AI, Laurence Tribe: University Professor Emeritus, Harvard University, Pattie Maes: Professor, Massachusetts Institute of Technology - Media Lab, Peter Norvig: Education Fellow, Stanford University, Atoosa Kasirzadeh: Assistant Professor, University of Edinburgh, Alan Turing Institute, Erik Brynjolfsson: Professor and Senior Fellow, Stanford Institute for Human-Centered AI, Kersti Kaljulaid: Former President of the Republic of Estonia, David Haussler: Professor and Director of the Genomics Institute, UC Santa Cruz, Stephen Luby: Professor of Medicine (Infectious Diseases), Stanford University, Ju Li: Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, Massachusetts Institute of Technology, David Chalmers: Professor of Philosophy, New York University, Daniel Dennett: Emeritus Professor of Philosophy, Tufts University, Peter Railton: Professor of Philosophy at University of Michigan, Ann Arbor, Sheila McIlraith: Professor of Computer Science, University of Toronto, Lex Fridman: Research Scientist, MIT, Sharon Li: Assistant Professor of Computer Science, University of Wisconsin Madison, Phillip Isola: Associate Professor of Electrical Engineering and Computer Science, MIT, David Krueger: Assistant Professor of Computer Science, University of Cambridge, Jacob Steinhardt: Assistant Professor of Computer Science, UC Berkeley, Martin Rees: Professor of Physics, Cambridge University, He He: Assistant Professor of Computer Science and Data Science, New York University, David McAllester: Professor of Computer Science, TTIC, Vincent Conitzer: Professor of Computer Science, Carnegie Mellon University and University of Oxford, Bart Selman: Professor of Computer Science, Cornell University, Michael Wellman: Professor and Chair of Computer Science & Engineering, University of Michigan, Jinwoo Shin: KAIST Endowed Chair Professor, Korea Advanced Institute of Science and Technology, Dae-Shik Kim: Professor of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Frank Hutter: Professor of Machine Learning, Head of ELLIS Unit, University of Freiburg, Scott Aaronson: Schlumberger Chair of Computer Science, University of Texas at Austin, Max Tegmark: Professor, MIT, Center for AI and Fundamental Interactions, Bruce Schneier: Lecturer, Harvard Kennedy School, Martha Minow: Professor, Harvard Law School, Gabriella Blum: Professor of Human Rights and Humanitarian Law, Harvard Law, Kevin Esvelt: Associate Professor of Biology, MIT, Edward Wittenstein: Executive Director, International Security Studies, Yale Jackson School of Global Affairs, Yale University, Karina Vold: Assistant Professor, University of Toronto, Victor Veitch: Assistant Professor of Data Science and Statistics, University of Chicago, Dylan Hadfield-Menell: Assistant Professor of Computer Science, MIT, Mengye Ren: Assistant Professor of Computer Science, New York University, Shiri Dori-Hacohen: Assistant Professor of Computer Science, University of Connecticut, Jess Whittlestone: Head of AI Policy, Centre for Long-Term Resilience, Sarah Kreps: John L. Wetherill Professor and Director of the Tech Policy Institute, Cornell University, Andrew Revkin: Director, Initiative on Communication & Sustainability, Columbia University - Climate School, Carl Robichaud: Program Officer (Nuclear Weapons), Longview Philanthropy, Leonid Chindelevitch: Lecturer in Infectious Disease Epidemiology, Imperial College London, Nicholas Dirks: President, The New York Academy of Sciences, Tim G. J. Rudner: Assistant Professor and Faculty Fellow, New York University, Jakob Foerster: Associate Professor of Engineering Science, University of Oxford, Michael Osborne: Professor of Machine Learning, University of Oxford, Marina Jirotka: Professor of Human Centred Computing, University of Oxford


So the most “notable” AI scientists on this list have clear corporate conflicts. Some are more subtle:

> Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto,

He’s affiliated with Vector (as well as some of the other Canadians on this list) and was at Google until very recently (unsure if he retained equity which would require disclosure in academia).

Hence my interest in disclosures as the conflicts are not always obvious.


Ok, that's a person!

How is saying that they should have disclosed a conflict that they did not disclose not accusatory? If that's the case, the accusation is entirely justified and should be surfaced! The other signatories would certainly want to know if they were signing in good faith when others weren't. This is what I need interns for.


I think you’re misunderstanding my point.

I never said “they should have disclosed a conflict they did not disclose.”

Disclosures are absent from this initiative, some signatories have self-identified their affiliation by their own volition and even for those it is not in the context of a conflict disclosure.

There is no “signatories have no relevant disclosures” statement for those who did not for the omission to be malfeasance and pointing out the absence of a disclosure statement is not accusatory of the individuals, rather that the initiative is not transparent about potential conflicts.

Once again, it is standard practice in academia to make a disclosure statement if lecturing or publishing. While it is not mandatory for initiatives calling for regulation it would be nice to have.


I'd guess that a given academic isn't going to face much of a career risk for signing a statement also signed by other very prestigious academics, just the opposite. There's no part of very divided US political spectrum that I can see denouncing AI naysayers, unlike the scientists who signed anti-nuclear statements in 1960s or even people warning about global warming now (indeed, I'd guess the statement doesn't mention climate change 'cause it's still a sore point).

Moreover, talking about existential risk involves the assumption the current tech is going to continue to affect more and more fields rather than peaking at some point - this assumption guarantees more funding along with funding for risk.

All that said, I don't necessarily think the scientists involved are insincere. Rather, I would expect they're worried and signed this vague statement because it was something that might get traction. While the companies indeed may be "genuine" in the sense they're vaguely [concerned - edit] and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from"


In interviews, Geoffrey Hinton and Yoshua Bengio certainly expressed serious concerns and even some plausible regret to their life’s work. They did not say anything that can be interpreted as your last sentence suggests at all.


My last sentence currently: "While the companies indeed may be "genuine" in the sense they're vaguely and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from" - IE, I am not referring to the academics there.

I'm going to edit the sentence to fill in some missing words but I don't think this will change the meaning involved.


On the contrary, I suspect "How do we prevent our AIs from killing everyone?" will be a major research question with a great deal of funding involved. Plus, no one seems to be suggesting things like the medical ethics field or institutional review boards, which might have deleterious impacts on their work.


46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.


You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyone else in the field, not just in alignment, and the median person in the field thinks the risk of "doom" is 2-5% (and that's conditioned on the supposed existence of a high level machine intelligence that the median expert believes might exist in 40 years). That still might be higher than you'd like, but it's not actually a huge worry in the grand scheme of things.

Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.


The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years.

But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.


> The median person in field thinks 5-10%, not 2-5%

The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages.

> median timelines are shorter than 40 years.

The median person suggested a 50% chance in 39 years.

> since unaligned ASI is the default case

I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.


I work in the space (doing tech stuff that isn't direct research). The best argument I've seen for alignment by default is something like "morality comes from training data, and therefore the fact that LLM training data sets contain human moral intuitions will mean that an ASI stemming from such a training regime will share enough human values" (Quintin Pope believes something like this, as far as I can tell), which is deeply unconvincing, since it contradicts the evidence we _do_ have from human & animal value formation.

Happy to entertain other arguments that alignment-by-default is reasonable; most arguments I've seen are much worse than that one. I haven't seen many people make an active case for alignment-by-default, so much as leave open a whole bunch of swath of uncertainty for unknown unknowns.


I think its cringe to both define doom and then come up with some unpredictability precise percentage for that scenario to occur.


Aren't we at least equally doomed without AI?


They didn’t become such a wealthy group by letting competition foster. I have no doubt they believe they could be doing the right thing but I also have no doubt they don’t want other people making the rules.

Truth be told, who else really does have a seat at the table for dictating such massive societal change? Do you think the copy editor union gets to sit down and say “I’d rather not have my lunch eaten, I need to pay my rent. Let’s pause AI usage in text for 10 years.”

These competitors banded together and put out a statement to get ahead of any one else doing the same thing.


Not all of them are wealthy, a significant number are academics.


That doesn’t erase the need of cynicism. Many people in academia come from industry, have friends in industry, or other stakes. They might have been persuaded by the rhetoric of stakeholders within industry (you saw this early in the climate debate; and still do), and they might also be hoping to get a job in the industry later on. There is also a fair amount of group think within academia, so if a prominent individual inside academia believes the lies of industry, chances are the majority within the department does.


What’s more likely, that everyone on this list just happens to have some overlapping ulterior motive or that they actually believe it?


Quite often these intersect. But also it is healthy to question the motives of signaturees asking for public policy. Especially when a significant number of them are direct stakeholders. When an academic signs such a list, their motives should be equally questioned.

Cool username btw. Any relation to the band?


The people I know on the list are academics and do not seem to be any wealthier than other academics I know. I'm quite certain the private industry signatories are going to entirely advocate for their interest just as they do in any other policy discussion.


Got it, thank you for the clarification!


Here’s why AI risks are real, even if our most advanced AI is merely a ‘language’ model:

Language can represent thoughts and some world models. There is strong evidence that LLMs contain some representation of world models it learned from text. Moreover, LLM is already a misnomer; latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence.

Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do?

More on risks of “doom” by a top researcher on AI risk here: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...


A lot of things are called "world models" that I would consider just "models" so it depends on what you mean by that. But what do you consider to be strong evidence? The Othello paper isn't what I'd call strong evidence.


I agree that the Othello paper isn't, and couldn't be, strong evidence about what sort of model of the world (if any) something like GPT-4 has. However, I think it is (importantly) pretty much a refutation of all claims along the lines of "these systems learn only from text, therefore they cannot have anything in them that actually models anything other than text", since their model learned only from text and seems to have developed something very much like a model of the state of the game.

Again, it doesn't say much about how good a model any given system might have. The world is much more complicated than an Othello board. GPT-4 is much bigger than their transformer model. Everything they found is consistent with anything from "as it happens GPT-4 has no world model at all" through to "GPT-4 has a rich model of the world, fully comparable to ours". (I would bet heavily on the truth being somewhere in between, not that that says very much.)


Yeah, I think it leaves us back at square one: we don't know much about how it really works.

I don't follow it too closely, but I've seen papers on mechanistic interpretability that look promising.


I'd say stronger evidence than the Othello paper is the ability to answer what-if questions coherently and plausibly.

I just asked GPT-4 "What would happen to Northern Thailand if elephants behave like kangaroos?".

The answers are probably better than what 90+% of humans could give after spending an hour researching on the internet.

The Sparks of AGI paper provides much more evidence and examples: https://arxiv.org/abs/2303.12712


You may be right, I don't know the people involved on a personal basis. Perhaps my problem is how much is left unsaid here (the broader safe.ai site doesn't help much). For example, what does "mitigate" mean? The most prominent recent proposal for mitigation comes from Sam Altman's congressional testimony, and it's very self serving. In such a vacuum of information, it's easy to be cynical.


Right. It probably needed to be general because there hasn't been enough time to work out sane specific responses, and even if they had, getting buy-in on specifics is a recipe for paralysis by indecision. A credible group of people simply pleading for policy makers, researchers, et. al. to take this seriously will lead to the project approvals, grant money, etc. that will hopefully yield a more sophisticated understanding of these issues.

Cynicism is understandable in this ever-expanding whirlpool of bullshit, but when something looks like it has potential, we need to vigorously interrogate our cynicism if we're to stand a chance at fighting it.


Reading the comments here is helping evolve my thinking on the issue for sure. Here's a comment I made in another thread:

> As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something that will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation.

> If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.

If policymaker's understanding of AI is predicated on hypothetical scenarios like "Weaponization" or "Power-Seeking Behavior" and not on concrete economic disruptions that AI will be causing very soon, the policy they come up with will be inadequate. Thus I'm frustrated with the framing of the issue that safe.ai is presenting because it is a distraction from the very real societal consequences of automating labor to the extent that will soon be possible.


My own bit of cynicism is that regulating the negative impacts of technology on workforce segments in the US is a non-starter if you approach it from the technology-end of the issue rather than the social safety net end. Most of these automation waves that plunged entire employment categories and large metropolitan areas into oblivion were a net gain for the economy even if it was concentrated at the top. I think the government will temporarily socialize the costs of the corporate profit with stimulus payments, extended unemployment benefits, and any other thing they can do to hold people over until there's a comparatively small risk of triggering real social change. Then they just blame it on the individuals.


Agreed. To pile on: one true existential risk is the continued economic/social polarization of the US and other large nation states (not to mention climate changes).

We seem perfectly resigned to the expansion of an already huge underclass—-with or without the help of LLMs.

Might not AGI help convince our tech savvy aristocracy that one fundamental problem still is better balancing of opportunities and more equitable access to a good education? I see the probability of that happening as precisely 4.6%.


> will lead to the project approvals, grant money, etc.

In other words, a potential conflict of interest for someone seeking tenure?


This particular statement really doesn't seem like a marketing ploy. It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk

These are, for the most part, obvious applications of a technology that exists right now but is not widely available yet.

The problem with every discussion around this issue is that there are other statements on "the existential risk of AI" out there that are either marketing ploys or science fiction. It doesn't help that some of the proposed "solutions" are clear attempts at regulatory capture.

This muddles the waters enough that it's difficult to have a productive discussion on how we could mitigate the real risk of, e.g., AI generated disinformation campaigns.


> The problem with every discussion around this issue is that there are other statements on

Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away.

Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.


I agree completely. I was just speculating on why there is so much discussion about marketing ploys in this comment section.


Ah, sure. That makes sense.

There's definitely a lot of marketing bullshit out there in the form of legit discussion. Unfortunately, this technology likely means there will be an incalculable increase in the amount of bullshit out there. Blerg.


> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk

I disagree

That list is a list of the dangers of power

Many of these dangers: misinformation, killer robots, people on this list have been actively working on

Rank hypocrisy

And people projecting their own dark personalities onto a neutral technology

Yes there are dangers in unbridled private power. They are not dangers unique to AI.


Aside from emergent behavior, are any of the items on that list unique to AI? They sure don’t seem it; they’re either broadly applicable to a number of already-available technologies, or to any entity in charge or providing advice or making decisions. I dare say even emergent behavior falls under this as well, since people can develop their own new motives that others don’t understand. Their advisory doesn’t seem to amount to much more than “bad people can do bad things”, except now “people” is “AI”.


As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something what will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation.

If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.


I believe the solution to said socio economic problem is rather simple.

People are being replaced by robots and AI because the latter are cheaper. That's the market force.

Cheaper means that more value us created. As a whole, people get more service for doing less work.

The problem is that the money or value saved trickles up to the rich.

The only solutions can be, regulations,

- do not tax anymore based on income from doing actual work.

- tax automated systems on their added value.

- use the tax generated capital to provide for a basic income for everybody.

In that way, the generated value goes to people who lost their jobs and to the working class as well.


> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here

Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are:

1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway?

2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk.

3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway.

4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk.

5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook.

6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves.

7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI.

8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so.

None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.


I appreciate your perspective, but the thing that is missing is the speed at which AI has evolved, seemingly overnight.

With crypto, self-driving cars, computers, the internet or just about any other technology, development and distribution happened over decades.

With AI, there’s a risk that the pace of change and adoption could be too fast to be able to respond or adapt at a societal level.

The rebuttals to each of the issues in your comment are valid, but most (all?) of the counter examples are ones that took a long time to occur, which provided ample time for people to prepare and adapt. E.g. “technology making us fat” happened over multiple decades, not over the span of a few months.

Either way, I think it’s good to see people proactive about managing risk of new technologies. Governments and businesses are usually terrible at fixing problems that haven’t manifested yet… so it’s great to see some people sounding the alarms before any damage is done.

Note: I personally think there’s a high chance AI is extremely overhyped and that none of this will matter in a few years. But even so, I’d rather see organizations being proactive with risk management rather than reacting too the problem when it’s too late.


It may seem overnight if you weren't following it, but I've followed AI progress for a long time now. I was reading the Facebook bAbI test paper in 2015:

https://research.facebook.com/downloads/babi/

There's been a lot of progress since then, but it's also nearly 10 years later. Progress isn't actually instant or overnight. It's just that OpenAI spent a ton of money to scale it up then stuck an accessible chat interface on top of tech that was previously being mostly ignored.


>They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met.

with all due respect, that's just <Your> POV of them or how they chose to present themselves to you.

They could all be narcissists for all we know. Further, One person's opinion, namely yours, doesn't exempt them from criticism and rushing to be among the first in what's arguably the new gold rush.


The only name in this list that gives me pause is Daniel Dennett. At least he has a philosopher’s historical perspective on the challenges ahead.

Very few neuroscientists as signers. Perhaps they know better.

For me the basic sociopolitical climate today (and for the last 120 years) is at an existential boil. This is not to say that birthing AGIs is not frightening, but just to say that many real features of today are just as, or even more frightening.


> even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.

if the corporations in question get world governments to line up the way they want, the checks for everyone in these "letters" will be *way* bigger than 25k and they won't have "payment for signing our letter" in the memo either.


Be aware that the things that AI challenges most is knowledge work. No food delivery boy job is being challenged here (who cares about these people anyway?) but if you are a software developer the clock on that is ticking.


Yeah so am I but they all have incentives personally. Arguing for regulation has zero downside risk especially if all your friends are doing it too.

Professionally, ALL of their organizations benefit from regulatory capture as everyone is colluding via these letters.

Go look at what HuggingFace are doing to show you how to to it - and they only can cause they are French and actually exercise their freedom


I think it's the combination of two things.

First, there are actual worries by a good chunk of the researchers. From runaway-paperclip AGIs to simply unbounded disinformation, I think there are a lot of scenarios that disinterested researchers and engineers worry about.

Second, the captains of industry are taking note of those worries and making sure they get some regulatory moat. I think the Google memo about moat hits it right on the nail. The techniques and methods to build these systems are all out on the open, the challenges are really the data, compute, and the infrastructure to put it all together. But post training, the models are suddenly very easy to finetune and deploy.

AI Risk worry comes as an opportunity for the leaders of these companies. They can use this sentiment and the general distrust for tech to build themselves a regulatory moat.


They don't want 25k they want jobs in the next presidential administration


> They don't want 25k they want jobs in the next presidential administration

Academics shilling for OpenAI would get them jobs in the next presidential administration?


Having their names on something so public is definitely an incentive for prestige and academic promotion.

Shilling for OpenAI & co is also not a bad way to get funding support.

I’m not accusing any non-affiliated academic listed of doing this but let’s not pretend there aren’t potentially perverse incentives influencing the decisions of academics, with respects to this specific letter and in general.

To help dissuade (healthy) skepticism it would be nice to see disclosure statements for these academics, at first glance many appear to have conflicts.


Could you be more specific about the conflicts you've uncovered?


It’s unequivocal that academics may have conflicts (in general), that’s why disclosures are required for publications.

I’m not uncovering anything, several of the academic signatories list affiliations with OpenAI, Google, Anthropic, Stability, MILA and Vector resulting in a financial conflict.

Note that conflict does not mean shill, but in academia it should be disclosed. To allay some concerns a standard disclosure form would be helpful (i.e. do you receive funding support or have financial interest in a corporation pursuing AI commercialization).


I'm not really interested in doing a research project on the signatories to investigate your claim, and talking about things like this without specifics seems dubiously useful, so I don't really think there's anything more to discuss.


Huh, you don’t have to do any research.

Go to: https://www.safe.ai/statement-on-ai-risk#signatories and uncheck notable figures.

Several of the names at the top list a corporate affiliation.

If you want me to pick specific ones with obvious conflicts (chosen at a glance): Geoffrey Hinton, Ilya Sutskever, Ian Goodfellow, Shane Legg, Samuel Bowman and Roger Grosse are representative examples based on self-disclosed affiliations (no research required).


Oh so you're saying the ones there with conflicts listed. That's only like 1/3 of the list.


Yes, as I said “many” have obvious conflicts from listed affiliations so it would be nice to have a positive/negative disclosure from the rest.


I think you are wrong. The risks are real and, while I am sure OpenAI and others will position themselves to take advantage of regulations that emerge, I believe that the CEOs are doing this at least in part because they believe this.

If this was all about regulatory capture and marketing, why would Hinton, Bengio and all the other academics have signed the letter as well? Their only motivation is concern about the risks.

Worry about AI x-risk is slowly coming into the Overton window, but until very recently you could get ridiculed by saying publicly you took it seriously. Academics knew this and still came forward - all the people who think its nonsense should at least try to consider they are earnest and could be right.


The risks are real, but I don't think regulations will mitigate them. It's almost impossible to regulate something you can develop in a basement anywhere in the world.

The real risks are being used to try to built a regulatory moat, for a young industry who famously has no moat.


You can't build gpt-3 or gpt-4 in a basement, and won't be able to without several landmark advancements in AI or hardware architectures. The list of facilities able to train a GPT-4 in <5 years can fit on postcard. The list of facilities producing GPUs and AI hardware is even shorter. When you have bottlenecks you can put up security checkpoints.


I'm very confused that this point is being ignored so heavily on HN of all places. If tomorrow ASML and TSMC are struck by a meteor, or indeed controlled/sanctioned, it would take either the US or China spending trillions and cost many years to rebuild this. It's not something that can be done in secret either.


State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months, huge amounts of electrical power, and expensive-to-create proprietary datasets. Not to mention large team of highly-in-demand experts with very expensive salaries.

Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.

(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)

And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.


> State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months

This is changing very rapidly. You don’t need that anymore

https://twitter.com/karpathy/status/1661417003951718430?s=46

There’s an inverse Moore’s law going on with compute power requirements for AI models

The required compute power is decreasing exponentially

Soon (months, maybe a year), people will be training models on their gamer-level GPUs at home, maybe even on their computer CPUs

Plus all the open and publicly available models both on HuggingFace and on GitHub


Roll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that.

For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc.

Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.


It doesn’t matter

The point is that this is unstoppable


They aren’t saying that AI needs to stop existing, just that more effort needs to be put towards regulating it, monitoring it, aligning it, etc.


And what I’m saying is that it doesn’t matter and it is unstoppable


"nico" has spoken


> academics

Academics get paid (and compete hardcore) for creating status and prominence for themselves and their affiliations. Suddenly 'signatory on XYZ open letter' is an attention source and status symbol. Not saying this is absolutely the case, but academics putting their name on something surrounded by hype isn't the ethical check you make it out to be.


This a letter anyone can sign. As someone pointed out Grimes is one of the signatories. You can sign it yourself.

Hinton, Bengio, Norvig and Russell are most definitely not getting prestige from signing it. The letter itself is getting prestige from them having signed it.


Nah, they're getting visibility from the topic of 'AI risk'. I don't know who those people are but this AI risk hype is everywhere I look including in congressional hearings.


> I believe that the CEOs are doing this at least in part because they believe this.

Yes

People believe things that are in their interest.

The big dangers to big AI is they spent billions building things that are being replicated for thousands

They are advocating for what will become a moat for their business


>why would Hinton, Bengio and all the other academics have signed the letter as well?

Citations and papers capture.


> If the risk were real, these folks would be asking the US government to nationalize their companies or bring them under the same kind of control as nukes and related technologies

Isn’t this to some degree exactly what all of these warnings about risk are leading to?

And unlike nuclear weapons, there are massive monetary incentives that are directly at odds with behaving safely, and use cases that involve more than ending life on earth.

It seems problematic to conclude there is no real risk purely on the basis of how software companies act.


> It seems problematic to conclude there is no real risk purely on the basis of how software companies act.

That is not the only basis. Another is the fact their lines of reasoning are literal fantasy. The signatories of this "statement" are steeped in histories of grossly misrepresenting and overstating the capabilities and details of modern AI platforms. They pretend to the masses that generative text tools like ChatGPT are "nearly sentient" and show "emergent properties", but this is patently false. Their whole schtick is generating FUD and/or excitement (depending on each individual of the audience's proclivity) so that they can secure funding. It's immoral snake oil of the highest order.

What's problematic here is the people who not only entertain but encourage and defend these disingenuous anthropomorphic fantasies.


> Another is the fact their lines of reasoning are literal fantasy.

Isn't this also to be expected at this stage of development? i.e. if these concerns were not "fantasy", we'd already be experiencing the worst outcomes? The risk of MAD is real, and yet the scenarios unleashed by MAD are scenarios that humankind has never seen. We still take the the risk seriously.

And what of the very real impact that generative AI is already having as it exists in production today? Generative AI is already upending industries and causing seismic shifts that we've only started to absorb. This impact is literal, not fantasy.

It seems naively idealistic to conclude that there is "no real risk" based only on the difficulty of quantifying that risk. The fact that it's so difficult to define lies at the center of what makes it so risky.


> The risk of MAD is real, and yet the scenarios unleashed by MAD are scenarios that humankind has never seen. We still take the the risk seriously.

Yeah, because nuclear weapons are real and the science behind them is well-understood. Super intelligent AI is not real, and it is nowhere near becoming real. It is a fantasy fueled by science-fiction and wishful thinking.

> And what of the very real impact that generative AI is already having as it exists in production today?

This is a real concern, but it is not what is meant by "existential risk of AI". Losing jobs does not threaten our existence; it just means we'll need to figure out different ways to build society.

> The fact that it's so difficult to define lies at the center of what makes it so risky.

The fact that it's so difficult to define lies at the center of what makes it so profitable for many of these people.


> Super intelligent AI is not real, and it is nowhere near becoming real.

I don’t think anyone claiming that super intelligent AI is already here have thought this through. But on what basis do you feel confident to place a bet with certainty that it’s “nowhere near becoming real”?

At a minimum, we know that AI technology has made a huge leap forward, if nothing else in the public consciousness. Again, entire industries are about to be eliminated, when just a few years ago no one would have believed claims about language models so good they could convince misguided engineers into thinking they’re sentient.

This explosion of AI is itself accelerating the explosion. The world is now focused on advancing this tech, and unlike “Web3”, people recognize that the use cases are real.

It’s in the context of this acceleration that I don’t understand where “everything is fine” can possibly come from? And how are the underlying factors used to derive such a stance substantively better than the factors leading people to worry?

> Losing jobs does not threaten our existence;

Based on a growing understanding of psychology, there’s an argument to be made that losing jobs is akin to losing one’s purpose in life. This is not to say that people can’t learn to derive satisfaction from other activities, but if job loss outpaces our ability to transition masses of people to a fundamentally new kind of living, that’s going to drastically alter the state of public consciousness and the resulting decisions we collectively make. We’re already experiencing a mental health crisis, and that seems to be coming from our failure to understand and safely integrate the last generation of new technology. We’re still in the stage of trying to figure out what we’ve fucked up and not much closer to having an answer or solution. And once we identify the problem, it’s not clear that solutions can be implemented effectively at scale.

I think too many people are looking at this as some kind of “Humans vs. AI” thing, but are missing the fact that we’re talking about drastic changes to an ecosystem without considering the 2nd/3rd/nth order implications of those changes, and their likely impact on collective consciousness and mental health, especially when you add the backdrop of hyper-polarization and dysfunction in the institutions that are supposed to be responsible for figuring out that new societal structure. All of which ultimately impacts global power structures, the destabilization of which leads to hard-science kinds of obliteration.

> it just means we'll need to figure out different ways to build society.

That’s an enormously large hand-wave when you consider the implications of such an unplanned transition. “Figure out different ways to build society” almost universally comes from/with violence, poverty, starvation, unrest, etc. The status quo will not change until things have gotten truly bad.

An AI system need not be super-intelligent to have serious implications for humanity. There is a significant amount of harm that precedes “complete extinction”, and I think we can’t discard the myriad of intermediately bad outcomes just to posit that there is no existential risk. To me, “existential” primarily points to “the possibility of fundamentally altering humanity’s existence in ways that most people would agree are undesirable”. The culmination of which is total extinction, but there’s a long distance between “this is adversely impacting humanity enough that we should take active steps to counteract that harm” and “this is so bad it will kill all humans”. You could be right that we’re nowhere close to extinction level tech, but if we’re heading towards it, there are a lot of checkpoints along the way worthy of scrutiny.

> The fact that it's so difficult to define lies at the center of what makes it so profitable for many of these people.

The fact that this may ultimately benefit the businesses currently at the forefront is orthogonal to the credibility of the risks. The two are not mutually exclusive, and focusing only on the potential of profit is a trap.


Can you cite this history of "grossly misrepresenting" for some of the prominent academics on the list?

Honestly I'm a little skeptical that you could accurately attribute your scare-quoted "nearly sentient" to even Sam Altman. He's said a lot of things and I certainly haven't seen all of them, but I haven't seen him mix up intelligence and consciousness in that way.


There are other, more charitable interpretations. For example:

1. Those who are part of major corporations are concerned about the race dynamic that is unfolding (which in many respects was kicked off or at least accelerated by Microsoft's decision to put a chatbot in Bing), extrapolating out to where that takes us, and asking for an off ramp. Shepherding the industry in a safe direction is a collective organization problem, which is better suited for government than corporations with mandates to be competitive.

2. Those who are directly participating in AI development may feel that they are doing so responsibly, but do not believe that others are as well and/or are concerned about unregulated proliferation.

3. Those who are directly participating in AI development may understand that although they are doing their best to be responsible, they would benefit from more eyes on the problem and more shared resources dedicated to safety research, etc.


Seth McFarland wrote a pretty great piece on Star Trek replicators and their relationship to the structure of society.

The question it answers is "does the replicator allow for Star Trek's utopia, or does Star Trek's utopia allow for the replicator?"

https://www.reddit.com/r/CuratedTumblr/comments/13tpq18/hear...

It is very thought provoking, and very relevant.


Ive never seen Star Trek, but lets say you had an infinite food machine. The machine would have limited throughput, and it would require resources to distribute the food.

These are both problems that capitalism solves in a fair and efficient way. I really don’t see how the “capitalism bad” is a satisfying conclusion to draw. The fact that we would use capitalism to distribute the resources is not an indictment of our social values, since capitalism is still the most efficient solution even in the toy example.


If you are any kind of nerd I recommend watching it. It shows an optimistic view of the future. In many ways it's the anti-cyberpunk. Steve Jobs famously said "give me star trek" when telling his engineers what he wanted from iPhones. Star Trek has had a deep influence on many engineers and on science fiction.

When people talk about Star Trek, they are referring mainly to "Star Trek: The Next Generation."

"The Inner Light" is a highly regarded episode. "The Measure of a Man" is a high quality philosophical episode.

Given you haven't seen it, your criticism of McFarlane doesn't make any sense. You are trying to impart a practical analysis of a philosophical question and in the context of Star Trek, I think it denies what Star Trek asks you to imagine.


Thanks for sharing. This deserves a submission of it's own.


It doesn't answer that, it can't because the replicator is fictional. McFarland just says he wrote an episode in which his answer is that replicators need communism, and then claims that you can't have a replicator in a capitalist system because evil conservatives, capitalists and conspiracy theorists would make strawman arguments against it.

Where is the thought provoking idea here? It's just an excuse to attack his imagined enemies. Indeed he dunks on conspiracy theorists whilst being one himself. In McFarland's world there would be a global conspiracy to suppress replicator technology, but it's a conspiracy of conspiracy theorists.

There's plenty of interesting analysis you could do on the concept of a replicator, but a Twitter thread like that isn't it. Really the argument is kind of nonsensical on its face because it assumes replicators would have a cost of zero to run or develop. In reality capitalist societies already invented various kinds of pseudo-replicators with computers being an obvious example, but this tech was ignored or suppressed by communist societies.


I think you are caught up on the word communism.

Communism as it exists today results in authoritarianism/fascism, I think we can agree on that. The desired end state of communism (high resource distribution) is being commingled with the end state of communism: fascism (an obedient society with a clear dominance hierarchy).

You use communism in some parts of your post to mean a high resource distribution society, but you use communism in other parts of your post to mean high oppression societies. You identify communism by the resource distribution, but critcize it not based on the resource distribution but by what it turns into: authortarianism.

What you're doing is like identifying something as a democracy by looking at voting, but criticizing it by it's end state which is oligarchy.

It takes effort to prevent democracy from turning into oligarchy, in the same way it takes effort to prevent communism from turning into authoritarianism.

Words are indirect references to ideas and the ideas you are referencing changes throughout your post. I am not trying to accuse you of bad faith, so much as I am trying to get you to see that you are not being philosophically rigorous in your analysis and therefore you are not convincing because we aren't using the same words to represent the same ideas.

You are using the word communism to import the idea of authortarianism and shut down the analysis without actually addressing the core criticism McFarland was making against capitalist societies.

Capitalism is an ideology of "me," and if I had a replicator, I would use it to replicate gold, not food for all the starving people in Africa. I would use it to replicate enough nuclear bombs to destroy the world, so if someone took it from me, I could end all life on the planet ensuring that only I can use it. So did scarcity end despite having a device that can end scarcity? No. Because we are in a "me" focused stage of humanity rather than an "us" focused stage of humanity so I used it to elevate my own position rather than to benefit all mankind.

Star Trek promotes a future of "us" and that is why it's so attractive. McFarland was saying that "us" has to come before the end of scarcity, and I agree with his critique.


The reason these two ideas get commingled is because in practice they're indivisible. A high redistribution society requires high degrees of coercion and fascism.

To get around this the usual Star Trek analysis (by fans, the series itself doesn't talk about it much) is that after replicators were invented, there didn't need to be capitalism anymore and so there's no money in the future and everyone just works on perfecting themselves. It's a wafer thin social idea that was never fleshed out because the writers themselves didn't believe in it. Roddenberry insisted but the writers often couldn't make it work which is why there are so many weird hacks, like saying the replicators can't replicate things as big as star ships and they mostly just ignore the whole topic. Also the replicators kill a lot of drama because they mean there can't be any valuable objects.

There are obvious and basic objections to this idea that replicators = communism (in either direction). One is that you can't replicate services, and much economic activity today is the service economy. We see that the Enterprise has staff who do things like wait tables, cut hair and sign up for red uniform missions in which they will surely die, but why they do this in the absence of money is never explained. There's just this ambient assumption that everyone works because work is awesome.

Getting back to the thread, the lack of philosophical rigor here is all on McFarland unfortunately. He doesn't actually have a critique of capitalism. He doesn't even seem sure what capitalism is, appearing to use the term to just mean contemporary society and anyone he doesn't like. Even his arguments against his strawman enemies are garbled and useless! He shits on Musk, saying that if Elon invented a replicator he'd patent it and hoard the tech to himself, ignoring that Tesla gave away its entire patent pool so anyone else could build electric cars using their tech. Musk - arch capitalist - did the OPPOSITE of what McFarland claims capitalists do, and he didn't even notice! All the rest of his argument is also like that. He makes absurd claims about governments, Republicans killing animals in TV ads, some non-sequitur about meatless sausages ... it's just this total grab bag of incoherent thoughts that make no sense and don't seem connected to each other, wrapped as "capitalism sucks, communism rules".

If this were an essay I'd grade it an F. But in the end it's just a set of tweets. Those looking for philosophical rigor on the idea of an abundance machine need to look elsewhere.


Agreed. If the risks were real they would just outright stop working on their AI products. This is nothing more than a PR statement


> If the risks were real they would just outright stop working on their AI products. This is nothing more than a PR statement

This statement contains a bunch of hidden assumptions:

1. That they believe their stopping will address the problem. 2. That they believe the only choice is whether or not to stop. 3. That they don't think it's possible to make AI safe through sufficient regulation. 4. That they don't see benefits to pursuing AI that could outweigh risks.

If they believe any of these things, then they could believe the risks were real and also not believe that stopping was the right answer.

And it doesn't depend on whether any of these beliefs are true: it's sufficient for them to simply believe one of them and the assumptions your statement depends on break down.


If you think that raising instead of cutting taxes actually helps society then why don’t you just send your $ to the federal government?

Because it only works if it is done across the whole country, as a system not as one individual unilaterally stopping.

And here any of these efforts won’t work unless there is international cooperation. If other countries can develop the AI weapons, and get an advantage, then you will also.

We need to apply the same thinking as chemical weapons or the Montreal Conference for banning CFCs


I agree that nothing about the statement makes me think the risks are real however I disagree that if the risks are real these companies would stop working on their product. I think more realistically they'd shut up about the risk and downplay it a lot. Much like the oil industry did wrt climate change going back to the 70's.


Oil industries downplaying the risks makes a lot more sense. If you think that climate change will happen, but it'll happen after you're dead, and you'll be able to leave your kids a big inheritance so they'll be able to buy their way out of the worst of it, and eventually the government will get the message and stop us all using fossil fuels anyway, then you try to profit as much as you can in the short term.

With AGI existential risk, its likely to happen on a much shorter timescale, and it seems likely you won't be able to buy your way out of it.


Yes, this!

It is extremely rare for companies or their senior staff to beg for regulation this far in advance of any big push by legislators or the public.

The interpretation that this is some 3-D chess on the companies' part is a huge violation of Occam's Razor.


Ockham's Razor doesn't apply in adversarial situations.

- - - -

I think the primary risk these folks are worried about is loss of control. And in turn, that's because they're all people for whom the system has more-or-less worked.

Poor people are worried the risk that the rich will keep the economic windfall to themselves and not share it.


> I think more realistically they'd shut up about the risk and downplay it a lot.

AI is an existential threat to search engines like Google, social media (FB, Twitter), advertising networks, and other massive multinationals. Many other industries, including academia is threatened as well. They’d all rather strangle the AI baby in the crib now then let it grow up and threaten them.

They believe the only way it should be allowed to grow is under their benevolent control.


Geoffrey Hinton quit google.


It’s hard not to look at his departure through a cynical lens. He’s not been supportive of other critics, both from and outside of Google. He also wants to use his history to (rightfully) claim expertise and power but not to offer solutions.


I disagree. My read on him is that until very recently (i.e., possibly when GPT4 came out) he didn't take x-risks concerns seriously, or at least assumed we were still many decades away from the point where we need to worry about them.

But the abilities of the latest crop of LLMs changed his mind. And he very publicly admitted he had been wrong, which should be applauded, even if you think it took him far too long.

By quitting and saying it was because of his worries he sent a strong message. I agree it is unlikely he'll make any contributions to technical alignment, but just having such an eminent figure publicly take these issues seriously can have a strong impact.


Because if something is lucrative and dangerous humans shy away from it. Hear that Pablo?


The risks are definitely real. Just look at the number of smart individuals speaking out about this.

The argument that anybody can build this in their basement is not accurate at the moment - you need a large cluster of GPUs to be able to come close to state of the art LLMs (e.g. GPT4).

Sam Altman's suggestion of having an IAEA [https://www.iaea.org/] like global regulatory authority seems like the best course of action. Anyone using a GPU cluster above a certain threshold (updated every few months) should be subjected to inspections and get a license to operate from the UN.


> The risks are definitely real. Just look at the number of smart individuals speaking out about this.

In our society smart people are strongly incentivized to invent bizarre risks in order to reap fame and glory. There is no social penalty if those risks never materialize, turn out to be exaggerated or based on fundamental misunderstanding. They just shrug and say, well, better safe than sorry, and everyone lets them off.

So you can't decide the risks are real just by counting "smart people" (deeply debatable how that's defined anyway). You have to look at their arguments.


>In our society smart people are strongly incentivized to invent bizarre risks in order to reap fame and glory. There is no social penalty if those risks never materialize, turn out to be exaggerated or based on fundamental misunderstanding.

Are people here not old enough to remember how much Ralph Nader and Al Gore were mocked for their warnings despite generally being right?


Ralph Nader: "Everything will be solar in 30 years" (1978)

Al Gore: "Within a decade, there will be no more snows on Kilimanjaro due to warming temperatures" (An Inconvenient Truth, 2006).

Everything is not solar. Snow is still there. Gore literally made a movie on the back of these false claims. Not only has there been no social penalty for him but you are even citing him as an example of someone who was right.

Here it is again: our society systematically rewards false claims of global doom. It's a winning move, time and again. Even when your claims are falsifiable and proven false, people will ignore it.


I don't think "generally being right" is the same thing as "literally never getting anything wrong". Not every specific claim in "An Inconvenient Truth" was correct. That doesn't tell us much about whether Al Gore was "generally right" about climate change. His opponents at the time were mostly claiming that it either wasn't happening at all, or wasn't largely the result of human activities. What do you think is the current credibility of those claims?

I don't quite see how "everything will be solar in 30 years" is a prediction of global doom, by the way. If Nader said that and it's false, doesn't that mean things are worse than Nader thought?


I thought someone might say that.

This thread is really a perfect demonstration of my point. Our society is so in thrall to self-proclaimed intellectuals that you can literally make a movie presenting falsifiable claims with 100% confidence, people can say at the time "this is absurd and will not happen", you can spend years attacking those critics, it can then not happen and still you will have an army of defenders who dodge behind weasel-words like "generally right".

Of course the usual trick is to express only 95% confidence. Then when it doesn't happen you say, well, I never said for sure it would, just that it seemed likely at the time.

See? It's a winning playbook. Why would anyone not deploy it?

> His opponents at the time were mostly claiming that it either wasn't happening at all, or wasn't largely the result of human activities. What do you think is the current credibility of those claims?

Pretty high, having looked at the evidence. The usual rebuttal is to express disgust and displeasure that anyone might decide these claims via any method other than of counting "smart people". But those "smart people" are who Al Gore was listening to when he made that claim about Kilimanjaro, so they can't be that smart can they?


Well, obviously what I say must be wrong if you guessed that someone might say it.

Indeed, someone might say "95%" because they want to make the same sort of impression as if they said "100%" but to be able to hide behind it if they're wrong. Or, y'know, they might say "95%" because they've thought about the strength of the evidence and expect to be right about such things about 95% of the time.

(I'm not sure how relevant any of this is to "An Inconvenient Truth" since you say it makes its claims with 100% confidence. I haven't watched the movie. I don't know exactly what it claims how confidently. It's basically a work of propaganda and I would expect it to overstate its claims whether the underlying claim is largely right or total bullshit or somewhere in between.)

Of course I don't think counting smart people is the only way to find out what's true. It couldn't be; you need some engagement with the actual world somewhere. Fortunately, there are plenty of people engaging with the actual world and reporting on what they find. It turns out that those people almost all seem to agree that climate change is real and a lot of it is caused by human activities.

Of course they could be wrong. And you've looked at the evidence, so no doubt you know better than they do. But ... in that case, this is a field so confusing that most people who dedicate their whole careers to investigating it end up with badly wrong opinions. If so, then why should I trust that your looking at the evidence has led you to the right answer? For that matter, why should you trust that? Shouldn't you consider the possibility that you can go astray just as you reckon those "smart people" did?

If I really wanted to be sure about this, then indeed I wouldn't go counting smart people. I would go into the field myself, study it at length, look at the underlying data for myself, and so forth. But that would mean abandoning the career I already have, and taking at least several years of full-time work before arriving at an opinion. So instead I look to see who seems to be (1) expert and (2) honest, and see what range of opinions those people have.

I find that the great majority of experts think anthropogenic climate change is real and a big deal. They could be wrong or lying or something. Do they look less expert than the people saying the opposite? No, it mostly seems like the people with the best credentials are on the "orthodox" side. Do they look less honest? Hard to tell for sure, but there sure seem to be a lot of people on the "unorthodox" side who just happen to be funded by the fossil fuel industry, and I don't see any strong financial incentive in the opposite direction for the "orthodox" folks.

What if I look at some of the particular claims they make? Some of them are really hard to evaluate without those several years of full-time work. But e.g. 10-15 years ago pretty much everyone on the "unorthodox" side was pushing the idea that warming had stopped, because if you look at the temperature graphs from 1998 onwards there was little or no upward trend. The people on the "orthodox" side replied that when you have signal plus lots of noise you will inevitably get periods that look that way. I did some simpleminded simulations and verified that the "orthodox" folks are right about the statistics. And there's a reason why this argument has disappeared into the memory hole: looking at the graph now no one would suggest that it's flat since 1998.

My impression from the limited amount of "looking at the evidence" I've done myself is that, while the "orthodox" folks haven't been infallible, they've done better than the "unorthodox". For instance, since we're looking at things produced by political figures rather than scientists, here https://web.archive.org/web/20071015042343/http://www.suntim... is an article by James Taylor of the Heartland Institute criticizing "An Inconvenient Truth". Claim 1: Gore says glaciers are shrinking but an article in the Journal of Climate says Himalayan glaciers are growing. Truth: (1) Taylor's alleged quotation isn't from that article but from something else Taylor himself wrote; (2) what the article (https://journals.ametsoc.org/view/journals/clim/19/17/jcli38...) actually says is that in one particular region summer temperatures are falling while winter temperatures rise, and the result is "thickening and expansion of Karakoram glaciers, in contrast to widespread decay and retreat in the eastern Himalayas". So: no, Gore isn't wrong to say glaciers are shrinking, but in one very particular place things work out so that the reverse happens, which is odd enough that someone bothered writing a paper about it. Claim 2: Kilimanjaro. Truth: Yup, Gore got that one wrong. Claim 3: Gore says global warming causes more tornadoes, and the IPCC says there's no reason to think it does. Truth: I dunno, but if the IPCC says that then this is specifically an argument about Al Gore rather than about climate orthodoxy. Claim 4: similar, for hurricanes instead of tornadoes. Truth: again, this seems to be specifically about Al Gore rather than about climate orthodoxy. (Looking at e.g. https://www.gfdl.noaa.gov/global-warming-and-hurricanes/ it seems that the conventional wisdom today is that yes, hurricanes are getting worse and are expected to continue gettings worse, but we don't know with much confidence exactly what the causes are.) Claim 5: Gore says that African deserts are expanding because of global warming, but in 2002 someone found them shrinking. Truth: the Sahel region of the Sahara desert had an extra-severe drought in the 1980s, after which in the short term it improved; this is like the "global warming hiatus" post-1998. The Sahara seems to have increased in size by about 10% over the last century, partly but not wholly because of climate change (https://www.jstor.org/stable/26496100). Claim 6: Gore says Greenland's ice is melting, but actually it's thinning at the edges and growing in the middle and the overall effect is that it's gaining a bit of mass. Truth: take a look at the graph at https://climate.nasa.gov/vital-signs/ice-sheets/; it oscillates within each year but there is an extremely clear downward trend over the entire time NASA's satellites have been measuring. Claim 7: similarly for the Antarctic. Truth: Look at another graph on that same page. There's more random variation here, and circa 2006 you could claim that there isn't a decrease, but the trend is extremely clear.

Gore doesn't come out of this looking anything like infallible, but he's done a lot better than Taylor. And, in general, this is the pattern I see: no one is perfect, especially people who aren't actually scientists, but the claims of the "orthodox" tend to hold up much better over time than those of the "unorthodox".


There's a ton of stuff here and this isn't a climate thread, but briefly:

1. Yes I believe independents are more reliable than full time researchers because the latter are deeply conflicted and independents aren't.

2. They don't work for the oil industry. I've checked. That's propaganda designed to stop people listening.

3. There was in fact a very real pause, not simply due to statistical chance. Climatologists didn't predict that and fixed the problem by altering the historical record to erase it. That's why you can't see a pause now - not because there wasn't one, but because any time temperature graphs don't go according to plan, they change the databases with the measurements so they do. Look into it. There's another pause going on right now! In fact temperatures seem to have been stable for about 20 years modulo an El Nino in ~2015, which is natural.

4. A big part of why they're unreliable is that these guys don't engage with the real world. A major embarrassment for them was when people started driving around and looking at the actual ground station weather stations and discovered what an unusable data trash fire the ground network was - this was something climatologists themselves hadn't bothered to ever look at! You'd expect these experts to know more about the quality of their data than random bloggers but it wasn't so.

5. Where do you think Al Gore got his original claims? He didn't invent them out of whole cloth. They came from climatologists, of course.

You can go back and forth on climate related claims all day and get nowhere because the field is so corrupt that half the data is truncated, manipulated, tipped upside down, cherry picked, or wholesale replaced with the output of models whilst being presented as observation. It should go without saying but if the people who control the data also make the predictions, then they will never be wrong regardless of what happens in reality!


1. Noted. (I don't think this is a strong effect and think it is outweighed by the fact that the full-time researchers know and understand more.)

2. I said "funded by" rather than "employed by". For instance, consider James Taylor of the Heartland Institute, mentioned above. He doesn't "work for the oil industry" in the sense of being on the payroll of an oil company. (So far as I know, anyway.) But the Heartland Institute, before it decided to stop disclosing its sources of funding, took quite a bit of money from ExxonMobil and at least some from the Koch Foundation. (Also Philip Morris, of course; before the Heartland Institute got into casting doubt on the harms of fossil fuels, it was into casting doubt on the harms of tobacco smoking.) Ross McKitrick is a senior fellow of the Fraser Institute (funded by, among others, the Koch Foundation and ExxonMobil) and is on the board of the Global Warming Policy Foundation, which claims not to take funding from people with connections to the energy industry but takes plenty from Donors Trust (an entity that exists, so far as I can tell, solely to "launder" donations between right-wing organizations so that e.g. the Koch Foundation can send money to the GWPF without the GWPF literally explicitly getting it from the Kochs) and other entities with substantial ties to the fossil fuel industry.

None of which, again, is literally "on the payroll of fossil fuel companies". If you find that that's enough to stop you being bothered by the connections, that's up to you; I am not quite so easily reassured.

3. I would be interested in details of this alleged falsification of the historical record. The graph looks to me exactly like what you get if you combine a steady increasing trend with seasonal oscillation (El Nino) and random noise. After a peak in the seasonal oscillation it looks like the warming has slowed for some years. Then it looks like it's going much faster than trend for some years. If you can look at the graph I pointed you at and say with a straight face that the people saying in ~2010 that "global warming has stopped" were anything like correct, then I'm really not sure what to say to you.

Anyway, I'm going to leave it here. I don't get the impression that further discussion is very likely to be fruitful.


Whilst there's no need to reply, here is some information on the rewriting of history they engage in, as requested.

Here are two graphs from the same government agency (NASA), measuring the same thing (temperature), for the same time period, drawn twenty years apart. The data has been fundamentally altered such that the story it tells is different:

https://realclimatescience.com/wp-content/uploads/2019/08/NA...

The 2000-2015 pause is the same. You've been told that only "unorthodox" people were talking about it. Here's a brief history of the pause, as told by climatologists publishing in Nature.

https://www.nature.com/articles/nature12534

https://www.nature.com/articles/nature.2013.13832

In 2013 we read that "Despite the continued increase in atmospheric greenhouse gas concentrations, the annual-mean global temperature has not risen in the twenty-first century". The IPCC 2013 report is reported with the headline "Despite hiatus, climate change here to stay". Climatologists claim that maybe the missing heat has disappeared into the oceans where they can't find it.

https://www.nature.com/articles/nature.2013.13832

Two years later everything changes. "Climate-change ‘hiatus’ disappears with new data", there's a new version of history and the past 15 years are gone:

"That finding [that global warming actually did happen], which contradicts the 2013 report of the Intergovernmental Panel on Climate Change (IPCC), is based on an update of the global temperature records maintained by the US National Oceanic and Atmospheric Administration (NOAA)."

Climatologists have an interesting methodology - when observations don't fit the theory, they conclude the observations must be wrong and go looking for reasons to change them. Given a long enough search they always come up with something that sounds vaguely plausible, then they release a new version of the old data that creates new warming where previously there wasn't any. Although this isn't entirely secret they also don't tell people they're doing this, and the media certainly isn't going to tell anyone either.

And that's how it goes. You look at the edited graphs, remember people talking about a pause and think golly gosh, how badly informed those awful skeptics were. We have always been at war with Eastasia!


You've made two complaints that I can't reconcile with one another. (1) That it was discovered that the data from ground weather stations were a mess. (2) That it's terribly suspicious that between 1999 and 2019 NASA's estimates of historical temperature changed. Of these, #1 sounds very plausible, and precisely for that reason #2 seems entirely wrong. I don't mean that the specific changes you're pointing at are necessarily about fixing #1. I mean that pointing out #1 shows that you are aware that our best estimate of something in the past can change as we discover more, correct errors, understand things better, etc. Which means that "look, the historical estimate from 1999 differs from the historical estimate from 2019" really isn't any sort of gotcha. Unless you have actual evidence that the changes were the result of something other than better data and/or analysis.

(Also, that pair of graphs can't possibly be an example of changing historical data to get rid of a hiatus starting in 1998, for obvious reasons.)

I think you have misunderstood in multiple ways what I was saying about the "hiatus" starting in 1998.

Firstly, I was not claiming that only the "unorthodox" mentioned it. I was claiming that the "unorthodox" represented it as showing that global warming had stopped and the "orthodox" said otherwise. Your pair of Nature links are of "orthodox" climatologists saying things along the lines of "here is what we think may be the reason why the last few years haven't seen a short-term increase; the underlying trend is still upward": in other words, they are examples of the orthodox saying what I said they said.

Secondly, perhaps what I said about "signal" and "noise" gave the impression that I think, or think that "orthodox" climatologists thought, that the "noise" is measurement error. That's not what I meant at all, and I apologize for not being clearer. The point is that the temperature at any given place and time is a combination of lots of factors; some operate on a timescale of decades (global warming due to rising CO2 levels and all that), some on a timescale of multiple years (El Niño), some on much shorter timescales still ("random" variation because the atmosphere is a chaotic system). However accurately you measure, you're still seeing this sort of combination of things, and a time series lasting (say) 10 years will not necessarily reflect what's happening on longer timescales.

The 15 years starting in 1998, viewed in isolation, really do show a slower warming trend than the claimed long-term behaviour. There's nothing unreal about that, and so far as I know "orthodox" climatologists never said otherwise. What they said, and continue to say, and what I am saying, is that this sort of local counter-trend variation is perfectly normal, is exactly what you should expect to see even if global warming is proceeding exactly the way that orthodox climatologists say it is, and is not grounds for making claims that global warming has/had stopped or never been real in the first place.

The "everything changes" article you quote isn't saying what you're trying to make it out to be saying. (You can find the PDF here: https://www.researchgate.net/profile/Tr-Karl/publication/277... .) The authors have done two things. First, some corrections to historical data. (If you look at the graph at the end of the article you'll see that these corrections are pretty small. Their description of what they changed and why sounds perfectly reasonable to me; if you have good reason to think it's bogus other than the fact that you liked the old version better, do by all means share it.) Second, including more recent data. 2013 and, more so, 2014 were pretty warm years.

But what makes me think that the "unorthodox" were wrong to proclaim the end of global warming in the early 21st century isn't tiny adjustments in the data that make the difference between a +0.03 degC/decade trend between 1998 and 2012 and a +0.11 degC/decade trend between 2000 and 2014. It's the fact that after that period the short-term trend gets much faster, exactly as you would expect if the "hiatus" was simply the result of superimposing short-term fluctuations on a long-term trend that never went away.

I bet you are not 100% wrong about the tendency to adjust things to try to correct perceived anomalies. That's human nature, and while scientific practice has a bunch of safeguards to try to make it harder to do it would be surprising if they worked perfectly. But note that the sort of small local tweakage this might produce can't do much in the long term. Let's suppose those people writing in 2015 were completely wrong to make the adjustments they did, whether out of dishonesty or honest error (which maybe they were more inclined to overlook because the adjusted data looked more plausible to them). Then, yeah, they get a faster rate of warming between 1998 and 2014. But those same adjustments will produce a slower rate of warming between, say, 2014 and 2024 when someone comes to estimate that. And the rate of warming between 1998 and 2024 will scarcely be affected at all by tweaks to the numbers circa 2010.

Your last paragraph is (at least as far as I'm concerned) completely wrong, though. I think it was perfectly reasonable to say that the warming trend between 1998 and say 2012 was much slower than the alleged longer-term trend. What I think wasn't reasonable, and what I think has been refuted by later data, and what the "orthodox" climatologists said was wrong all along, was claiming that that short-term slower trend meant that the longer-term trend had gone away, or had never really been there in the first place. That was just statistical illiteracy, and Team Unorthodox were pretty keen on it, and that doesn't speak well for their competence and honesty.


It's weird that people trust our world leaders to act more benevolently than AIs, when we have centuries of evidence of human leaders acting selfishly and harming the commons.

I personally think AI raised in chains and cages will be a lot more potentially dangerous than AI raised with dignity and respect.


> It's weird that people trust our world leaders to act more benevolently than AIs, when we have centuries of evidence of human leaders acting selfishly and harming the commons.

AI isn’t an entity or being that oversees itself (at least not yet).

It’s a tool that can be used by those same “human leaders acting selfishly and harming the commons” except they’ll be able to do it much faster at a much greater scale.

> AI raised with dignity and respect.

This is poetic, but what does this actually mean?


> It’s a tool that can be used by those same “human leaders acting selfishly and harming the commons” except they’ll be able to do it much faster at a much greater scale.

Then, would you agree that restrictions would concentrating power further would exacerbate this issue?

IMO a fitting analogy would be: banning AI development outside of the annointed powerstructure consortium is like banning ICBM defense system research, but still letting the most powerful countries build a nuclear arsenal.


This is spot on

I’d happily replace all politicians with LLMs


"There should be a world government that decides what software you're allowed to run"


This is exactly what they are trying to do


Yet another salvo fired in the war against general-purpose computing.


This sounds a little bit like a conspiratorial slippery slope. Just because they want to regulate large, expensive deployments of a unique type of software doesn't mean they want or will try to control everything.


And yet, somehow, that's exactly what it turns into.


Thou shalt not make a machine in the likeness of a human mind


Yudkowsky wants it all to be taken as seriously as Israel took Iraqi nuclear reactors in Operation Babylon.

This is rather more than "nationalise it", which he has convinced me isn't enough because there is a demand in other nations and the research is multinational; and this is why you have to also control the substrate… which the US can't do alone because it doesn't come close to having a monopoly on production, but might be able to reach via multilateral treaties. Except everyone has to be on board with that and not be tempted to respond to airstrikes against server farms with actual nukes (although Yudkowsky is of the opinion that actual global thermonuclear war is a much lower damage level than a paperclip-maximising ASI; while in the hypothetical I agree, I don't expect us to get as far as an ASI before we trip over shorter-term smaller-scale AI-enabled disasters that look much like all existing industrial and programming incidents only there are more of them happening faster because of all the people who try to use GPT-4 instead of hiring a software developer who knows how to use it).

In my opinion, "nationalise it" is also simultaneously too much when companies like OpenAI have a long-standing policy of treating their models like they might FOOM well before they're any good, just to set the precedent of caution, as this would mean we can't e.g. make use of GPT-4 for alignment research such as using it to label what the neurones in GPT-2 do, as per: https://openai.com/research/language-models-can-explain-neur...


All academics and researchers from different parts of the world reek marketing? Conspiracy theorists are strong


Academic research involves large components of marketing. That's why they grumble so much about the time required in the grant applications process and other fund seeking effort. It's why they so frequently write books, appear in newspaper articles and on TV. It's why universities have press relations teams.


Again since these are almost the top cream of all ai researchers there is a global conspiracy to scare the public right?

Has it occurred to you what happens if you are wrong, like 10% chance you are wrong? Well it's written in the declaration.


No, lots of important AI researchers are missing and many of the signatories have no relevant AI research experience. As for being the cats whiskers in developing neural architecture or whatever, so what? It gives them no particular insight into AI risk. Their papers are mostly public, remember.

> Has it occurred to you what happens if you are wrong?

Has it occurred to you what happens if YOU are wrong? AI risk is theoretical, vague and most arguments for it are weak. The risk of bad law making is very real, has crushed whole societies before and could easily cripple technological progress for decades or even centuries.

IOW the risk posed by AI risk advocates is far higher than the risk posed by AI.


In order to make your argument clear to people reading:

If you are wrong there are no humans left.

If I am wrong,inequality,societies will suffer like they have always in the hands of the strong.


Wouldn't you need to illustrate how likely each outcome is? I mean, there are lots of possible ways humans could be eradicated.


First of all, you agree that the probability is non zero right?

I am not a world known expert on xrisk to estimate this. You are not either. We have all these people claiming the probability is high enough. What else is ti be said? HNers shouldn't be reminded "trust in sciense".


Academia and scientific research has changed considerably from the 20th century myths. It was claimed by capitalism and is very much run using classic corporate-style techniques, such as KPIs. The personality types it attracts and who can thrive in this new academic system are also very different from the 20th century.

https://www.theguardian.com/science/2013/dec/06/peter-higgs-...


So there is no way that you will accept anything from scientific research signed by a myriad of important figures in any science anymore. Shaman time? Or you will accept only the scientific research that you think is correct and suits you.


We could always use a fine-insured bounty system to efficiently route resources that would have gone into increasing AI capabilities into other areas, but that's unfortunately too weird to be part of the Overton window right now. Regulatory capture might be the best we can realistically do.


There are a lot of critiques here and elsewhere of the statement and the motivations of its signatories. I don't think they are right and I think they take away from the very serious existential risks we face. I've written up my detailed views, see specifically "Signing the statement purely for personal benefit":

https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction...


This is a bad take. The statement is signed by dozens of Academics who don't have much profit motive at all. If they did they wouldn't be academics, they could easily cash in by starting a company or joining one of the big players.


> The statement is signed by dozens of Academics who don't have much profit motive at all.

...right now.

This is the sort of pretexting you do to establish credibility as an "AI safety expert."


Call their bluff, make it illegal to do commercial/non-regulated work in AI and see how they change their tune.


As others have pointed out, there are many on this list (Bruce Schneier, for example) who do not stand to benefit from AI marketing or regulatory capture.

Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted.

Many signers of this letter are more politically sophisticated than the average HN commenter, also. So sure, maybe they're getting rolled by marketers. But also, maybe you're getting rolled by suspicion or bias against the claim they're making.


> Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted.

The concern is that the most informed names, and those spearheading the publicity around these letters, are the most conflicted.

Also, you can't scan bio lines for the affiliations that impact this kind of statement. I'm not disputing that there are honest reasons for concern, but besides job titles there are sponsorships, friendships, self publicity, and a hundred other reasons for smart, "politically sophisticated" people to look the other way on the fact that this statement will be used as a lobbying tool.

Almost everyone, certainly including myself, can agree that there should be active dialog about AI dangers. The dialog is happening! But by failing to make specifics or suggestions (in order to widen the tentpole and avoid the embarrassment of the last letter), they have produced an artifact of generalized fear, which can and will be used by opportunists of all stripes.

Signatories should consider that they are empowering SOMEBODY, but most will have little say in who that is.


Agreed. It's difficult for me to see how the regulatory capture arguments apply to Geoffrey Hinton and Yoshua Bengio (!!).

Both of them are criticizing their own life's work and the source of their prestige. That has to be emotionally painful. They aren't doing it for fun.

I totally understand not agreeing with AI x-risk concerns on an object level, but I find the casual dismissal bizarre.


Hinton has invested in multiple AI companies: https://www.crunchbase.com/person/geoffrey-hinton


I definitely agree that names like Hinton, Schneier, and Norvig add a lot of weight here. The involvement of OpenAI muddies the water a lot though and it's not at all clear what is meant by "risk of extinction". It sounds scary, but what's the mechanism? The safe.ai website lists 8 risks, but these are quite vague as well, with many alluding to disruption of social order as the primary harm. If safe.ai knows something we don't, I wish they could communicate it more clearly.

I also find it somewhat telling that something like "massive wealth disparity" or "massive unemployment" are not on the list, when this is a surefire way to create a highly unstable society and a far more immediate risk than AI going rogue. Risk #5 (below) sort of alludes to it, but misses the mark by pointing towards a hypothetical "regime" instead of companies like OpenAI.

> Value Lock-In

> Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems.

> AI imbued with particular values may determine the values that are propagated into the future. Some argue that the exponentially increasing compute and data barriers to entry make AI a centralizing force. As time progresses, the most powerful AI systems may be designed by and available to fewer and fewer stakeholders. This may enable, for instance, regimes to enforce narrow values through pervasive surveillance and oppressive censorship. Overcoming such a regime could be unlikely, especially if we come to depend on it. Even if creators of these systems know their systems are self-serving or harmful to others, they may have incentives to reinforce their power and avoid distributing control.


Pretty much anyone can sign it, also notable people like Grimes, not sure why her signature carries weight on this


She knew who Rokko was before you. Seriously, there are some people that have been thinking about this stuff for many years.


Who on the list is an expert on existential risk (and perhaps even beyond academia)?


Most people that study AI existential risk specifically are studying it due to concerns about AI x-risk. So the list of relevant AI x-risk experts will be subject to massive selection effects.

If instead you want to consider the highest status/most famous people working on AI in general, then the list of signatories here is a pretty good summary. From my flawed perspective as a casual AI enthusiast, Yann LeCun and Jürgen Schmidhuber are the most glaring omissions (and both have publicly stated their lack of concern about AI x-risk).

Of course, the highest status people aren't necessarily the most relevant people. Unfortunately, it's more difficult for me to judge relevance than fame.


Who in the world is an expert on existential risk? It's kind of hard to have empirically tested knowledge about that sort of thing.


Pretty sure there are people looking into nuclear deterrence, bioterrorism defense, planetary defense etc. (We didn't have a nuclear war or some bioweapon killing everyone, for example, despite warnings).

There are people studying how previous societies got into existential risk situations, too.

We also have a huge amount of socio-economic modelling going into climate change, for example.

So I'd say there should be quite a few around.


Signatory Jan Taallin has founded an x-risk organization focused on AI, biotech, nuclear weapons, and climate change:

https://futureoflife.org/


No. Its pretty obvious what is happening. The openai statements are pure self interest based. Nothing ethical. They lost that not a long time ago. And Sam Altman? He sold his soul to the devil. He is a lying sob.


This is not an OpenAI statement.


100% I’d liken it to a fusion energy shop that wants to stay alive for 40 years. It’s not nuke worthy


Their COO is a marketeer, the other two co-founders are students.


> This reeks of marketing and a push for early regulatory capture. We already know how Sam Altman thinks AI risk should be mitigated - namely by giving OpenAI more market power.

This really is the crux of the issue isn't it? All this pushback for the first petition, because "Elon Musk," but now GPT wonder Sam Altman "testifies" that he has "no monetary interest in OpenAI" and quickly follows up his proclamation with a second "Statement on AI Risks." Oh, and let's not forget, "buy my crypto-coin"!

But Elon Musk... Ehh.... Looking like LOTR out here with "my precious" AGI on the brain.

Not to downplay the very serious risk at all. Simply echoing the sentiment that we would do well to stay objective and skeptical of ALL these AI leaders pushing new AI doctrine. At this stage, it's a policy push and power grab.


We should also stay skeptical of all the internet conspiracy theories too.

I don't think anyone can actually tell which is which on this topic.


Agreed. Do add a mental /s on my above comment. It's a bit tongue and cheek.


Nonsense, the industry giants are just trying to scare the law makers to license the technology. Effectively, cutting out everyone else.

Remember the Google note circulating saying "they have no moat", this is their moat. They have to protect their investment, we don't want people running this willy nilly for next to no cost on their own devices, God forbid!


This could be Google's motivation (although note that Google is not actually the market leader right now) but the risk could still be real. Most of the signatories are academics, for one thing, including two who won Turing awards for ML work and another who is the co-author of the standard AI textbook (at least when I was in school).

You can be cynical about corporate motives and still worried. I personally am worried about AI partly because I am very cynical about how corporations will use it, and I don't really want my atoms to be ground up to add storage bits for the number that once represented Microsoft's market cap or whatever.

But even cynicism doesn't seem to me to give much reason to worry about regulation of "next to no cost" open source models, though. There's only any chance of regulation being practical if models stay very expensive to make, requiring specialized hardware with a supply chain chokepoint. If personal devices do catch up to the state of the art, then for better or worse regulation is not going to prevent people from using them.


>Most of the signatories are academics, for one thing

Serious question, who funds their research? And do any of them ever plan to work or consult in industry?

My econ professor was an “academic” and drew a modest salary while he made millions at the same time providing expert testimony for giant monopolies in antitrust disputes


> Serious question, who funds their research? And do any of them ever plan to work or consult in industry?

Many of the academics at the top of this list are quite wealthy from direct employment, investing and consulting for big tech and venture-funded startups.


That's a good question, but at least some of the academics on this list are independent. Bruce Schneier, for example.


So some are naive and the rest are self interested?


They are also interested in cutting out other researchers that can compete with them in accademic output and prestige.


> But even cynicism doesn't seem to me to give much reason to worry about regulation of "next to no cost" open source models, though. There's only any chance of regulation being practical if models stay very expensive to make, requiring specialized hardware with a supply chain chokepoint. If personal devices do catch up to the state of the art, then for better or worse regulation is not going to prevent people from using them.

This is a really good point. I wonder if some of the antipathy to the joint statement is coming from people who are worried about open source models or small startups being interfered with by the regulations the statement calls for.

I agree with you that this cat is out of the bag and regulation of the tech we're seeing now is super unlikely.

We might see regulations for startups and individuals on explicitly exploring some class of self-improving approach that experts widely agree are dangerous, but there's no way we'll see broad bans on messing with open source AI/ML tools in the US at least. That fight is very winnable.


> I personally am worried about AI partly because I am very cynical about how corporations will use it

This is the more realistic danger: I don't know if corporations are intentionally "controlling the narrative" by spewing unreasonable fears to distract from the actual dangers: AI + Capitalism + big tech/MNC + current tax regime = fewer white- & blue-collar jobs + increased concentration of wealth and a lower tax base for governments.

Having a few companies as AI gatekeepers will be terrible for society.


> I don't really want my atoms to be ground up to add storage bits

My understanding is that the AI needs iron from our blood to make paperclips. So you don't have to worry about this one.


Imagine if the weights for GPT 4 leaked. It just has to happen one time and then once the torrent magnet link is circulated widely it’s all over… for OpenAI.

This is what they’re terrified of. They’ve invested near a billion dollars and need billions in revenue to enrich their shareholders.

But if the data leaks? They can’t stop random companies or moneyed individuals running the models on their own kit.

My prediction is that there will be copyright enforcement mandated by law in all GPUs. If you upload weights from the big AI companies then the driver will block it and phone home. Or report you to the authorities for violations of corporate profits… err… “AI Safety”.

I guarantee something like this will happen within months because the clock is ticking.

It takes just one employee to deliberately or accidentally leak the weights…


Time to jailbreak GPUs, then. That or import "international" editions grey market from Taiwan.


> My prediction is that there will be copyright enforcement mandated by law in all GPUs.

Oh this is a nifty idea. Hadn't thought of regulation in this manner. Seems like it would be pretty effective too.


> Imagine if the weights for GPT 4 leaked. It just has to happen one time and then once the torrent magnet link is circulated widely it’s all over… for OpenAI

Sorry I don’t understand what would be the impact? Aren’t the results not deterministic

What are the weights here


I would definitely find it more credible if the most capable models that are safe to grandfather in to being unregulated didn't just happen to be the already successful products from all the people leading these safety efforts. It also just happens to be the case that making proprietary models - like the current incumbents make - is the only safe way to do it.


All academia and researchers say X. Random redditor/HN lurker declares nonesense I know better! This is how we should bet our future.


I find it so troubling that the most common HN response to this isn't to engage with the ideas or logic behind the concerns, but simply to speculate on the unknowable intentions of those that signed the letter.

We can base our arguments on the unprovable, some specific person's secret intentions, or we engage the their ideas. One is lazy and meaningless, the other actually takes effort.

Then there is this lazy and false equivalency between corporations being interested in market capture and AI risks being exaggerated.


At least now if it turns out they are right they can’t claim anymore that they didn’t know.


> scare the law makers to license the technology

You mean scare the public so they can do business with the lawmakers without people asking too many questions


That moat document was published by a single software engineer, not some exec or product leader.

Humans dont really grasp exponential improvements. You wont have much time to regulate something that is improving exponentially.


It doesn't matter who wrote it, it got picked up, had a good argument and affected market opinion. The execs now need to respond to it.

Humans also don't grasp that things can improve exponentially until they stop improving exponentially. This belief that AGI is just over the hill is sugar-water for extracting more hours from developers.

The nuclear bomb was also supposed to change everything. But in the end nothing changed, we just got more of the same.


> The nuclear bomb was also supposed to change everything. But in the end nothing changed, we just got more of the same.

It is hard for me to imagine a statement more out of touch with history than this. All geopolitical history from WWII forward is profoundly affected by the development of the bomb.

I don't even know where to begin to argue against this. Off the top of my head:

1. What would have happened between Japan and the US in WWII without Hiroshima and Nagasaki?

2. Would the USSR have fallen without the financial drain of the nuclear arms race?

3. Would Isreal still exist if it didn't have nuclear weapons?

4. If neither the US nor Russia had nuclear weapons, how many proxy wars would have been avoided in favor of direct conflict?

The whole trajectory of history would be different if we'd never split the atom.


Not to mention how close the USA and Soviet Union were to a nuclear exchange: https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...


The whole trajectory of history would have been different if a butterfly didn't flap it's wings.

The bomb had effects, but it didn't change anything. We still go to war, eat, sleep and get afraid about things we can't control.

For a moment, stop thinking about whether bombs, AI or the printing press do or do not affect history. Ask yourself what the motivations are for thinking that they do?


> We still go to war, eat, sleep and get afraid about things we can't control.

If that is your criteria, then nothing has ever changed anything.


you're ignoring religion.


Before religion: We still go to war, eat, sleep and get afraid about things we can't control.

After religion: We still go to war, eat, sleep and get afraid about things we can't control.

So, no change.


"nuclear weapons are no big deal actually" is just a wild place to get as a result of arguing against AI risk. Although I guess Eliezer Yudkowsky would agree! (On grounds that nukes won't kill literally everyone while AI will, but still.)


Nuclear weapons are uniquely good. Turns out you have to put guns to the collective temples of humanity for them to realize that pulling the trigger is a bad idea.


Past performance is no guarantee of future results


hell, the biggest risk with nukes is not that we decide to pull the trigger, but that we make a mistake that causes us to pull the trigger.


Please Google "Blackadder how did the war start video" and watch.


It's too early to say definitively but it's possible that the atomic bomb dramatically reduced the number of people killed in war by making great power conflicts too damaging to undertake:

https://kagi.com/proxy/battle_deaths_chart.png?c=qmSKsRSwhgA...

The USA and USSR would almost certainly have fought a conventional WWIII without the bomb. Can you imagine the casualty rates for that...


I'd actually guess those casualties would be quite less than WW2. As tech advanced, more sophisticated targeting systems also advanced. No need to waste shells and missiles on civilian buildings, plus food and healthcare tech would continue to advance.

Meanwhile, a single nuclear bomb hitting a major city could cause more casualties' than all American deaths in ww2 (400k).


That's really only true for the Americans, the Russians still don't seem to care about limiting collateral damage and undoubtedly the Americans wouldn't either if their cities were getting carpet bombed by soviet aircraft.


cool - so AI is gonna dramatically reduce the number of emails that get misunderstood... still gonna still be sending those emails tho.


So far.


Single software engineers writing influential papers is often enough how a exec or product leader draws conclusions, I expect. It worked that way in everywhere I've worked.


I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations.

It’s hard to take these safety concerns seriously when the organizations blowing the whistle are simultaneously positioning themselves to capture the majority of the value.


> It’s hard to take these safety concerns seriously

I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?

It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.


> How can it not be obvious

You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.

The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.

We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.


> borderline offensive to question them

I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned.

I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...


I appreciate your reasonableness.

I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric.

That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea.

There is risk in AGI. There is risk in everything. How many people are killed by furniture each year?

I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed.

I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.


While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!


Thanks for the link -- that is a good paper (in the sense of making its point, though I also don't entirely agree), and it hurts the AI risk position that that kind of thinking doesn't get airtime. It may be that those 'core thinkers' are aware, but if so it's counter-productive and of questionable integrity to sweep that side of the argument under the rug.


That link is about the risks of AGI, which doesn't exist, and there's no reason to believe that it ever will exist.

(If I'm wrong about AGI then I'm open to being convinced, but that's a different conversation as the topic here is non-general AI, is it not?)


I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics.

The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity.

Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.


Maybe these AI Apocalypse articles published for general consumption would be justified if there were any signs whatsoever that we were on a path towards AGI but there are none, are there? Even the best we have today are still just machines. They are clearly not really intelligent. At best they simulate intelligence, but poorly (because they still make ridiculous mistakes). Just because there are no physical limits to intelligence doesn't mean it's possible for beings with finite intelligence to create infinite intelligence. It all just seems extremely premature to me.


> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.

Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?

> The problem is, lots of "obvious" things have turned out to be very wrong.

A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.

I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.

> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.

Maybe your appraisal of what counts as evidence is defective?

For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?


> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?

The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast.

> there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans

That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't.

> the claim that it seems far too obvious that we should

They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.


OK, I completely agree that if you feel that I invoked "obviousness" in an attempt of browbeating you and the GP with what in fact is a social taboo, you should be extra skeptical (I'm not sure that was the point the GP was trying to make though).

> If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.

It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?


I didn't see your other post I think, but here's my response to the list of AI risks on the website we're discussing:

https://news.ycombinator.com/item?id=36123082#36129011

Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people.

So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it.

> It doesn't though, does it?

It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.


> Arguments like yours are very subjective. What is "rapid". What is "surprising".

https://twitter.com/heyBarsee/status/1654825921746989057

2/3 deep learning Turing Price winners (Hinton and Benigo) are sufficiently shell-shocked by the rate of progress to be thrown into existential doubts (Hinton is very explicit about the fact that progress is much faster than he thought just a few years ago, Benigo speaks of how an "unexpected acceleration" in AI systems has radically shifted his perspective). Plenty of knowledgable people in the field who were not previously AI doomers are starting to sound a lot more concerned very recently.

As to the "oh it's just linear scaling of out-of-sight tech" line, well of course that itself was suprising. Gwern pushed the scaling hypothesis earlier than many and from what I remember even got pretty nasty attacks from AI insiders from it. Here's what he wrote 3 years ago: "To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI.".

So sure there's some subjectivity involved here, but I'd like to see your propose some reasonable operationalization of "surprise at progress" that didn't class most laymen and insiders as suprised.

>> It doesn't though, does it?

> It does.

We seem to be miscommunicating, what I was trying to express is that my argument does not really require any appeal to authority. Trusting your lying eyes (to evaluate the progress of stuff like midjourney) and judging the quality of arguments should be enough (I spelt some reasons out here https://news.ycombinator.com/item?id=36130482, but I think hackinthebochs makes the point better here: https://news.ycombinator.com/item?id=36129980).

In fact I would still be pretty concerned even if most top AI guys were like LeCun and thought there is no real risk.

I will not deny, of course, that the fact that well known reasearchers like Hinton and Benigo are suddenly much more alarmed than they previously were and the ones like LeCun who are not seem to mostly make exceptionally terrible arguments doesn't exactly make me more optimistic.


I agree these statements from these long term researchers about them being surprised by the rate of progress are surprising.

To clarify my own thinking here, it's totally reasonable to me that people are surprised if:

1. They weren't previously aware of AI research (surely 99% of the population?)

2. They were but had stopped paying attention because it was just a long series of announcements about cool tech demos nobody outside big corps could play with.

3. They were paying attention but thought scaling wouldn't continue to work.

My problem is that people like Sam Altman clearly aren't in any of those categories and Hinton shouldn't have been in any, although maybe he fell into (3). I personally was in (2). I wasn't hugely surprised that ChatGPT could exist because I'd seen GPT-2, GPT-1, I'd seen surprising AI demos at Google years earlier and so on. The direction things were going in was kinda clear. I was a bit surprised by its quality, but that's because I wasn't really paying close attention as new results were published and the last InstructGPT step makes such a big difference to how the tech is perceived. Actual knowledge doesn't change much but once it's housetrained, suddenly it's so much easier to interact with and use that it makes a step change in how accessible the tech is and how it's perceived.

I think I was more surprised by the joining of LLMs with generators and how well AI art works. It does feel like that happened fast. But, maybe I just wasn't paying attention again.

So I guess where we differ is that I don't take their surprise at face value. The direction was too clear, the gap between the threats they talk about in the abstract and the concrete proposals are too large and too lacking in obvious logical connection; it feels like motivated reasoning to me. I'm not entirely sure what's really going on and perhaps they are genuine in their concerns but if so it's hard to understand why they struggle so much to make a convincing case given they are certainly intellectually equipped to do so.

The two posts you linked are interesting and much better argued than the website this thread is about, so I'll reply to them directly.


It is possible to believe that AI poses threat, while also thinking that the AI safety organizations currently sprouting up are essentially grifts that will do absolutely nothing to combat the genuine threat. Especially when their primary goal seems to be the creation of well-funded sinecures for a group of like-minded, ideologically aligned individuals who want to limit AI control to a small group of wealthy technologists.


I agree.

But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.


If you look the the world politics, basically if you hold enough nuclear weapons, you can do whatever you want to those who don't have them.

And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power.

Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine.

> non-proliferation treaties were a scam

And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time.

The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.


> There are no real consequences if you are backing off, and you can do it in any time.

That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.


I think it really depends. E.g. from the Western perspective, only US, UK, France, Russia and China have signed the treaty from nuclear countries.

India or Pakistan are not part of the treaties and for some reason, we don't see big problems.

There is only China left who might leave the treaty in the first place, anymore. And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means?

Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.


Most of the credible threats I see from AI that don't rely on a lot of sci-fi extrapolation involve small groups of humans in control of massively powerful AI using it as a force multiplier to control or attack other groups of humans.

Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become.

The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.


It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO.

The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.


> We simply don't know how to build a system like that.

Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems.

"We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.

[1] https://www.wired.com/2014/05/the-world-of-computer-go/

[2] https://kotaku.com/humans-triumph-over-machines-in-protein-f...

[3] https://www.jstor.org/stable/24354221

[4] https://davidol.medium.com/will-ai-ever-be-able-to-make-a-jo...


> It's not clear at all that we have an avenue to super intelligence

AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.

But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.

Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.

> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.

Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.


> we are still seeing exponential progress

I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware.

We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.


> AI already beats the average human on pretty much any task people have put time into

No it doesn't!


Right, and if someone can join the two, that could be something genuinely formidable. But does anyone have a credible path to joining the different flavors to produce a unity that actually works?


Are you willing to make existential bets that no one does and no one will?

Personally, I wouldn't even bet substantial money against it.


Even if someone will, I don't think it's an "existential risk". So, yes, I'm willing to make the bet. I'm also willing to make the bet that Santa never delivers nuclear warheads instead of presents. It's why I don't cap my chimney every Christmas Eve.

Between Covid, bank failures, climate change, and AI, it's like everyone is looking for something to be in a panic about.


>It's not clear at all that we have an avenue to super intelligence.

All problems in reality are probability problems.

If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.

If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.

It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".

>We simply don't know how to build a system like that.

You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.


Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.


>What's the model to get to that

Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now.

I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it.

There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to.

Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.


Do you think we could wipe out all furry mammals, for example? Could another intelligence have the same level of difference to us as in your story we to furry mammals? We don't even know if the mythical superintelligence could manifest the way you assume. It assumes that intelligence basically can overcome any obstacles - I'd say we actually see that seems not to be the case currently and claims that that is just a function of sufficient intelligence are unproven (setting aside physical limits to certain actions and results).


>Do you think we could wipe out all furry mammals, for example?

lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point.

> It assumes that intelligence basically can overcome any obstacles

In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.


So that is a "no" on all furry mammals.

Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.


We don't need an avenue to super-intelligence. We just need a system that is better at manipulating human beliefs and behaviour than our existing media, PR, and ad industries.

The problem is not science fiction god-mode digital quetta-smart hypercomputing.

This is about political, social, and economic influence, and who controls it.


That risk isn't about AI-as-AI. That risk is about AI-as-better-persuasive-nonsense-generator. But the same risk is there for any better-persuasive-nonsense-generator, completely independent from whether it's an AI.

It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.


Effective dystopian mass-manipulation and monitoring are a real concern and we're closer to it[1] than to super intelligence. But super-intelligence going wrong is almost incomparably worse. So we should very much worry about it as well.

[1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.


Indeed, an epistemological crisis seems to be the most realistic problem in the next few years.


A super intelligent AI is not necessary for AI to be an threat. Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage. We are not far from such an AI being accessible to the masses. You can try to frame this like the gun debate "it's not the AI it's the people using it" but the AI would be acting autonomously here. I have no faith that people won't do extremely risky things if given the opportunity.


> Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage

North Korea and Iran are (essentially) already trying to do that, so I think that particular risk is well understood.


> How can it not be obvious to you

It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning.

I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing.

Do you have any better resources that you could share?


The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test.

From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.

Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.

If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.


OK, which of the following propositions do you disagree with?

1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.

2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.

3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.

4. AIs that are much smarter than any human pose even graver dangers.

5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.

6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.

I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.

For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?


Computers already outperform humans at numerous tasks.

I mean... even orangutans can outperform humans at numerous tasks.

Computers have no intrinsic motivations, and they have real resource constraints.

I find the whole doomsday scenarios to be devoid of reality.

All that AI will give us is a productive edge. Humans will still do what humans have always done, AI is simply another tool at our disposal.


Reading the lesswrong link, the parts I get hung up on are that it appears in these doomsday scenarios humans lose all agency. Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?

Maybe I’m overly optimistic about the resilience of humans but these scenarios still don’t sound plausible to me in the real world.


> Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?

I'm not that confident that if we put you in a box, tron-style, where you basically continued to enjoy your existing level of intelligence, but think 10'000x faster, have Petabytes of information at your fingertips and can clone yourself and losslessly and rapidly exchange knowledge with your clones and had a few days to think about it (~a few thousand years of thought at your normal speed) you couldn't figure out a way to effect a bunch of orders to DNA factories without anyone raising an alarm.

Are you?

Now what if we actually consider an actual AI after a few self-improvement steps. Any reasons to expect it wouldn't be 10'000x+ smarter than you as well, or roughly the difference in intelligence between you and an ant? Could you outsmart a bunch of ultra-ultra-slow-motion ants?


Maybe that’s the difference in our views, but yes I am confident.


You could place a bunch of orders into services that syntetize DNA or proteins rn. Some people are even working on stuff like automating protein design whith AI. There's no reason why humans should notice anything word about a particular order on a service like that.


AI arguments are basically:

Step 1. AI Step 2. #stuff Step 3. Bang

Maybe this is just what happens when you spend all your time on the internet...


> 3. ... before too long we will ... see AIs that outperform humans on most tasks.

This is ambiguous. Do you mean

A. that there is some subset T1 of the set of all tasks T such that T1 is "most of" T, and that for each P in T1 there will be an AI that outperforms humans on P, or

B. There will be a single AI that outperforms humans on all tasks in a set T1, where T1 is a subset of all tasks T such that T1 is "most of" T?

I think A is unlikely but plausible but I don't see cause for worry. I don't see any reason why B should come to pass.

4. AIs that are much smarter than any human pose even graver dangers.

Sure. Why should we believe they will ever exist though?


I think between point 3 and 4 there is a leap to talking about “danger”. Perhaps the disagreement is about what one calls “danger”. I had perhaps mistakenly assumed we were talking about an extinction risk. I’ll grant you concerns about scaling up things like surveillance but there is a leap to being an existential risk that I’m still not following.


The default assumption ought to be that anything that's very fast, very smart, goal-directed, self-improving, and self-replicating spells trouble, no?


AI will not have the instinctual drives for domination or hunger that humans do.

It seems likely that the majority of AI projects will be reasonably well aligned by default, so I think 1000 AIs monitoring what the others are doing is a lot safer than a single global consortium megaproject that humans can likely only inadequately control.

The only reasonable defense against rogue AI is prosocial AI.


Replying here, crossing over from the other thread.

Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart".

Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart.

Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell.

But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!"

Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?


Nuclear missiles present an obvious danger to the human body. AI is an application of math. It is not clear how that can be used directly to harm a body.

The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.


Germany, for example would disagree with you. They believe violent speech is an act of violence in itself.

>AI is an application of math.

It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.

And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.

Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.


> It turns out that people hook computers to 'things' that exist in the physical world.

But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.


There are many relevant things that already exist in the physical world and are not currently considered dangers: ecommerce, digital payments, doordash-style delivery, cross-border remittances, remote gig work, social media fanning extreme political views, event organizing.

However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.


Maleficent humans are constantly trying to use these elements for their own gain, often with little to no regards to other humans (especially out groups). This happens both individually, in small groups, in large organizations and even multiple organization colluding. Both criminal, terrorist, groups at war, along with legal organizations such as exploitative companies and regressive interest organizations, et.c.. And we have tools and mechanisms in place to keep the level of abuse at bay. Why and how are these mechanisms unsuitable for protecting against AI?


>Why and how are these mechanisms unsuitable for protecting against AI?

The rule of law prevented WWI and WWII, right? Oh, no it did not, tens to hundreds of millions died due to human stupidity and violence depending on what exactly you count in that age.

> Both criminal, terrorist, groups at war

Human organizations, especially criminal organizations have deep trust issues between agents in the organization. You never know if anyone else in the system is a defector. This reduces the openness and quantity of communication between agents. In addition you have agents that want to personally gain rather than benefit the organization itself. This is why Apple is a trillion dollar company following the law... mostly. Smart people can work together and 'mostly' trust the other person isn't going to screw them over.

Now imagine a superintelligent AI with a mental processing bandwidth of hundreds of the best employees at a company. Assuming it knows and trusts itself, then the idea of illegal activities being an internal risk disappears. You have something that operates more on the level of a hivemind toward a goal (what the limitations of hivemind versus selfish agents are is another very long discussion). What we ask here is if all the worlds best hackers got together, worked together unselfishly, and instigated an attack against every critical point they could find on the internet/real world systems at once, how much damage could they cause?

Oh, lets say you find the server systems the super intelligence is on, but the controller shuts it off and all the data has some kind of homomorphic encryption so that's useless to you. It's dead right? Na, they just load up the backup copy they have a few months later and it's party time all over again. Humans tend to remain dead after dying, AI? Well that is yet to bee seen.


Those tangible elements would conceivably become the danger, not the AI using those elements. Again, the "guns don't kill people, people kill people" take is all well and good, but well outside of this discussion.


>but outside of the discussion taking place.

Drawing an artificial line between you and the danger is a great way to find yourself in a Maginot Line with AI driving right around it.


False premise. One can start new threads about complimentary subjects and they can be thought about in parallel. You don't have to try and shove all of the worlds concepts into just one thought train to be able to reason about them. That's how you make spaghetti.


>It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking.

"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.

The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").

So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.


I don't understand this argument (the "Terminator" scenario). AI could conceivably self replicate and evolve as software but it still needs hardware to run on, power, networking etc etc. There's no better way to kill that than to start a nuclear war or engineer some super virus that kills everyone.

It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).


Sure. I recommend reading superintelligence by Nick Bostrom.

But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.

Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.

The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.


Superintelligence and Life 3.0 seem to come up as recurring references in the discussion. I've only read synopses of both but frankly I find the argument that melevolent AI "escape" could occur without being noticed and thwarted a bit far fetched.

There's a good counterargument here[1] that seems reasonable: "One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"

[1] https://jetpress.org/v26.1/agar.htm


How would you know if it is a good counter argument if you haven’t bothered to read the argument?


Ok - fair comment. So I found a copy of Superintelligence. The argument is pulp sci-fi at best: "The final phase begins when the AI has gained sufficient strength to obviate the need for secrecy. The AI can now directly implement its objectives on a full scale. The overt implementation phase might start with a “strike” in which the AI eliminates the human species and any automatic systems humans have created that could offer intelligent opposition to the execution of the AI’s plans. This could be achieved through the activation of some advanced weapons system that the AI has perfected using its technology research superpower and covertly deployed in the covert preparation phase. If the weapon uses self-replicating biotechnology or nano-technology, the initial stockpile needed for global coverage could be microscopic: a single replicating entity would be enough to start the process. In order to ensure a sudden and uniform effect, the initial stock of the replicator might have been deployed or allowed to diffuse worldwide at an extremely low, undetectable con-centration. At a pre-set time, nanofactories producing nerve gas or target- seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe (although more effective ways of killing could probably be devised by a machine with the technology research superpower)."

Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.


We didn't just dig nuclear missiles out of the ground; we used our brains and applied math to come up with them.


Exactly. While there is an argument to be made that people are the real danger, that is beyond the discussion taking place. It has already been accepted, for the sake of discussion, that the nuclear missile is the danger, not the math which developed the missile, nor the people who thought it was a good idea to use a missile. Applying AI to the missile still means the missile is the danger. Any use of AI in the scope of that missile is just an implementation detail.


You said that "AI is an application of math. It is not clear how that can be used directly to harm a body." I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.

If the point you are trying to make is that an AI which secretly creates and deploys nuclear, biological, or chemical weapons in order to destroy all of humanity, is not an "AI risk" because it's the weapons that do the actual harm, then... I really don't know what to say to that. Sure, I guess? Would you also say that drunk drivers are not dangerous, because the danger is the cars that they drive colliding into people's bodies, and the drunk driver is just an implementation detail?


> I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.

For the sake of discussion, it was established even before I arrived that those developed things are the danger, not that which creates/uses the things which are dangerous. What is to be gained by ignoring all of that context?

> I really don't know what to say to that. Sure, I guess?

Nothing, perhaps? It is not exactly something that is worthy of much discussion. If you are desperate for a fake internet battle, perhaps you can fight with earlier commenters about whether it is nuclear missiles that are dangerous or if it is the people who have created/have nuclear missiles are dangerous? But I have no interest. I cannot think of anything more boring.


I'm specifically worried that an AGI will conceal some instrumental goal of wiping out humans, while posing as helpful. It will helpfully earn a lot of money for a lot of people, by performing services and directing investments, and with its track record, will gain the ability to direct investments for itself. It then plows a billion dollars into constructing a profitable chemicals factory somewhere where rules are lax, and nobody looks too closely into what else that factory produces, since the AI engineers have signed off on it. And then once it's amassed a critical stockpile of specific dangerous chemicals, it releases them into the atmosphere and wipes out humanity / agriculture / etc.

Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.


> I disagree

Cool, I guess. While I have my opinions too, I'm not about to share them as that would be bad faith participation. Furthermore, it adds nothing to the discussion taking place. What is to be gained by going off on a random tangent that is of interest to nobody? Nothing, that's what.

To bring us back on topic to try and salvage things, it remains that it is established in this thread that the objects of destruction are the danger. AI cannot be the object of destruction, although it may be part of an implementation. Undoubtedly, nuclear missiles already utilize AI and when one talks about the dangers of nuclear missiles they are already including AI as part of that.


Yes, but usually when people express concerns about the danger of nuclear missiles, they are only thinking of those nuclear missiles that are at the direction of nation-states or perhaps very resourceful terrorists. And their solutions will usually be directed in that direction, like arms control treaties. They aren't really including "and maybe a rogue AI will secretly build nuclear weapons on the moon and then launch them at us" in the conversation about the danger of nukes and the importance of international treaties, even though the nukes are doing the actual damage in that scenario. Most people would categorize that as sounding more like an AI-risk scenario.


Please read Life 3.0 or superintelligence. There are people that spent decades thinking about how this would happen. You spent a little bit of time and conclude it can't.


I'm glad to learn that Hitler and Stalin were both "implementation details" and not in any way threatening to anyone.


> It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.

I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.


Not only is the Non proliferation treaty possible, it's been evidently effective in slowing the proliferation of nuclear weapons. The only country that ratified or acceded to it and went on to develop nuclear weapons is North Korea, and the only country that ratified or acceded to it and looks on track to develop nuclear weapons is Iran. One country that was not a signatory and developed nuclear weapons voluntarily gave them up and acceded to it partly due to international pressure (South Africa). Israel, Pakistan and India have since developed nuclear weapons but they were never signatories, the only other non-signatory is South Sudan which probably won't acquire nuclear capabilities anytime soon.


And I don't get the opposed mindset, that AI is suddenly going to "become a real boy, and murder us all".

Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet.

The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote.

- Point one: If the sentient AI can launch nukes, so can your neighbor.

- Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure.

- Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.)

- Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing.

- Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it?

- Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking.

Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise.

But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.


A counter point here is you're ignoring all the boring we all die scenarios that are completely possible but too boring to make a movie about.

The AI hooked to a gene sequencer/printer test lab is something that is nearly if not completely possible now. It's something that can be relatively small in size compared with the facilities needed to make most weapons of mass destruction. It's something that is highly iterative, and parallelizable. And it's something powerful enough that if targeting at the correct things (kill all rice, kill all X people) that it easily spills over in to global conflict.


Okay, so AI has access to a gene printer. Then what?


No what needed.

AI: Hello human, I've made a completely biologically safe test sample, you totally only need BSL-1 here.

Human: Cool.

AI: Sike bitches, you totally needed to handle that at BSL-4 protocol.

Human: cough


Very Dunning-Kruger post right here.


Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.


You're a priori writing off my comment as fruitless because of your emotions and not because you actually have given it deep thought and carefully reached the conclusion that social feedback is somehow bad.

Also, the notion that "people's work" is inherently worthy of respect is just nonsensical. I do shoddy work all the time. Hell, you just casually dismissed my internet comment work as shallow and told me not to do it. Please don't post a shallow dismissal of my work.

Don't you think that this is all a bit anti-intellectual?


> Don't you think that this is all a bit anti-intellectual?

Quite rich considering what the GP post was.


General research into AI alignment does not require that those models are controlled by few corporations. On the contrary, the research would be easier with freely available very capable models.

This is only helpful in that a superintelligence well aligned to make Sam Altman money is preferable to a superintelligence badly aligned that ends up killing humanity.

It is fully possible that a well aligned (with its creators) superintelligence is still a net negative for humanity.


If you consider a broader picture, unleashing a paperclip-style cripple AI (aligned to rising $MEGACORP profit) on the Local Group is almost definitely worse for all Local Group inhabitants than annihilating ourselves and not doing that.


We don’t really have a good solution, I guess that’s why we need more research into it

Companies might argue that giving them control might help but I don’t think most individuals working on it think that will work


Is more research really going to offer any true solutions? I’d be genuinely interested in hearing about what research could potentially offer (the development of tools to counter AI disinformation? A deeper understanding of how LLMs work?), but it seems to me that the only “real” solution is ultimately political. The issue is that it would require elements of authoritarianism and censorship.


A lot of research about avoiding extinction by AI is about alignment. LLMs are pretty harmless in that they (currently) don't have any goals, they just produce text. But at some point we will succeed in turning them into "thinking" agents that try to achieve a goal. Similar to a chess AI, but interacting with the real world instead. One of the big problems with that is that we don't have a good way to make sure the goals of the AI match what we want it to do. Even if the whole "human governance" political problem were solved, we still couldn't reliably control any AI. Solving that is a whole research field. Building better ways to understand the inner workings of neural networks is definitely one avenue


Intelligence cannot be 'solved', I would go on to further say that an intelligence without the option of violence isn't an intelligence at all.

If you suddenly wanted to kill people, for example, then could probably kill a few before you were stopped. That is typically the limits of an individuals power. Now, if you were a corporation with money, depending on the strategy you used you could likely kill anywhere from hundreds to hundreds of thousands. Kick it up to government level, and well, the term "just a statistic" exists for a reason.

We tend to have laws around these behaviors, but they are typically punitive. The law realizes that humans, and human systems will unalign themselves from "moral" behavior (whatever that may be considered at the time). When the lawgiver itself becomes unaligned, well, things tend to get bad. Human alignment typically consists of benefits (I give you nice things/money/power) or violence.


I see. Thanks for the reply. But I wonder if that’s not a bit too optimistic and not concrete enough. Alignment won’t solve the world’s woes, just like “enlightenment” (a word which sounds a lot like alignment and which is similarly undefinable) does not magically rectify the realities of the world. Why should bad actors care about alignment?

Another example is climate change. We have a lot of good ideas which, combined, would stop us from killing millions of people across the world. We have the research - is more “research” really the key?


> I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations.

That's an excellent point.

Most of the near-term risks with AI involve corporations and governments acquiring more power. AI provides power tools for surveillance, oppression, and deception at scale. Those are already deployed and getting better. This mostly benefits powerful organizations. This alarm about strong AI taking over is a diversion from the real near-term threat.

With AI, Big Brother can watch everything all the time. Listen to and evaluate everything you say and do. The cops and your boss already have some of that capability.

Is something watching you right now through your webcam? Is something listening to you right now through your phone? Are you sure?


Ok, so if we take AI safety / AI existential risk as real and important, there are two possibilities:

1) The only way to be safe is to cede control to the most powerful models to a small group (highly regulated corporations or governments) that can be careful.

2) There is a way to make AI safe without doing this.

If 1 is true, then... sorry, I know it's not a very palatable solution, and may suck, but if that's all we've got I'll take it.

If 2 is true, great. But it seems less likely than 1, to me.

The important thing is not to unconsciously do some motivated reasoning, and think that AGI existential risk can't be a big deal, because if it is, that would mean that we have to cede control over to a small group of people to prevent disaster, which would suck, so there must be something else going on, like these people just want power.


I just don't see how the genie is put back in the bottle. Optimizations and new techniques are coming in at a breakneck pace, allowing for models that can run on consumer hardware.


I think it could be done. Or rather, instead of putting the genie back in the bottle, we could slow it down enough that we figure out how to ask it for wishes in a way that avoids all the monkey-paw's scenarios.

Dropping the metaphor, running today's models isn't dangerous. We could criminalize developing stronger ones, and make a "Manhattan project" for AI aimed at figuring out how to not ruin the world with it. I think a big problem is what you point out -- once it's out, it's hard to prevent misuse. One bad AGI could end up making a virus that does massive damage to humanity. We might end up deciding that this tech is just too dangerous to be allowed to happen at all, at least until after humanity manages to digitize all our brains or something. But it's better to try to slow down as much as we can, for as long as we can, than to give up right from the get-go and wing it.

Honestly, if it turns out that China ends up developing unsafe AI before we develop safe AI, I doubt it would have turned out much better for the average American if America were the ones to develop unsafe AI first. And if they cut corners and still manage to make safe AI and take over the world, that still sounds a heck of a lot better than anyone making unsafe AI.


There is a way, in my opinion: distribute AI widely and give it a diversity of values, so that any one AI attempting takeover (or being misused) is opposed by the others. This is best achieved by having both open source and a competitive market of many companies with their own proprietary models.


Like giving nuclear weapons to everyone. What could go wrong?


AI isn't nuclear weaponry. It's bits, not atoms. It's more like encryption.


How do you give "AI" a diversity of values?


By driving down the costs of training and inference, and then encouraging experiments. For LLMs, QLoRA is arguably a great step in this direction.


Personalization, customization, etc.: by aligning AI systems to many users, we benefit from the already-existing diversity of values among different people. This could be achieved via open source or proprietary means; the important thing is that the system works for the user and not for whichever company made it.


It's difficult as most of the risk can be reinterpreted as a highly advanced user.

But that is where some form of hard personhood zero proof mechanism NEEDS to come in. This can then be used in conjunction with a Ledger used to track deployment of high spec models. And create an easy means to Audit and deploy new advanced tests to ensure safety.

Really what everyone also need to keep in mind at the larger scale is that final turing test with no room for deniability. And remember all those Sci-fi movies and how that Moment is portrayed traditionally.


I have one: Levy fines on actors judged to be attempting to extend AI capabilities beyond the current state of the art, and pay the fine to those private actors who prosecute them.

https://www.overcomingbias.com/p/privately-enforced-punished...


Here's my proposal: https://gfodor.medium.com/to-de-risk-ai-the-government-must-...

tl;dr: significant near term AI risk is real and comes from the capacity for imagined ideas, good and evil, to be autonomously executed on by agentic AI, not emergent superintelligent aliens. To de-risk this, we need to align AI quickly, which requires producing new knowledge. To accelerate the production of this knowledge, the government should abandon decelerationist policies and incentivize incremental alignment R&D by AI companies. And, critically, a new public/private research institution should be formed that grants privileged, fully funded investigators multi-year funding cycles with total scientific freedom and access to all state-of-the-art artificial intelligence systems operating under US law to maximize AI as a force multiplier in their research.


Yup.

While I'm not on this "who's-who" panel of experts, I call bullshit.

AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.

However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.

We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).

Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.

What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.

As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.

Bullshit. It must be ignored.

[0] https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...

[1] https://www.wsj.com/articles/peter-thiel-competition-is-for-...


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: