Hacker News new | past | comments | ask | show | jobs | submit | duvenaud's comments login

Right, but the employer they choose is presumably the one offering the best deal in the world from their point of view. Wouldn't it make more sense to be mad at every other employer for not proposing better pay, than the one that's offering the best pay for that worker?

I think it's very unlikely that any of those would lead to human extinction, especially since most of those take decades to unfold, and would still leave large parts of the earth habitable.

Toby Ord wrote a book trying to estimate each of these risks, and put AI on top: https://www.amazon.ca/Precipice-Existential-Risk-Future-Huma...


Well, I think paperclip maximizers or 'kill all humans' skynet scenarios are extremely implausible.

From the examples we have, intelligence is anticorrelated with sadism and avarice.


Sure, but think about how humans drove others extinct. We never decided to "kill all wooly mammoths", we just wanted to use their meat / habitats for other things.

The correlation you mention seems noisy enough that I wouldn't want to bet my civilization on it.


AI Safety hasn't been divorced from tech companies, at least not from Deepmind, OpenAI, and Anthropic. They were all founded by people who said explicitly that AGI will probably mean the end of human civilization as we know it.

All three of them have also hired heavily from the academic AI safety researcher pool. Whether they ultimately make costly sacrifices in the name of safety remains to be seen (although Anthropic did this already when they delayed the release of Claude until after ChatGPT came out). But they're not exactly "watching from the sidelines", except for Google and Meta.


Here's one of my concrete worries: At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.

I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.


“Humans are going to be outcompeted by AI” is the concrete bit as best I can tell.

Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.

If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling.

If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.


I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing).

But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition.

What abilities do humans have that AIs will never have?


Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it.

If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working.

And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.


Again, are you claiming that it's impossible for a machine to invent anything that a human could? Right now a large chunk of humanity's top talent and capital are working on exactly this problem.

As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here.

As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?


Yeah, I think most animals have every right to fear us more now that we have guns. Just like Id fear a chimp more if he was carrying a machine gun.

My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still.

AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria.

That's way off from an apocalyptic self-important machine.


Sorry, my gun analogy was unclear. I meant that, just because some agents on a planet have an ability, doesn't mean that everyone on that planet benefits.

I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.

Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.


I think we've converged on a partial agreement, but I wanted to clarify the gun analogy part.

I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".


>What abilities do humans have that AIs will never have?

I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power.

That's a far higher bar than outcompeting all individual humans at all cognitive tasks.


Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks.

Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.


That would require a high degree of integration into human society though, which makes it seem very unlikely that AIs would doggedly pursue a common goal that is completely unaligned with human societies.

Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me.

How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans?

I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.


Yes, I agree that initially, AIs will be highly integrated, and their goals will probably at least appear to be somewhat aligned with those of human societies. Similar to human corporations and institutions. But human institutions and governments go off the rails regularly, and are only corrected because humans can sometimes go on strike or organize to stop them. I fear we will lose those abilities.

As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.


Chess is just a game, with rigidly defined rules and win conditions. Real life is a fuzzy mix of ambiguous rules that may not apply and can be changed at any point, without any permanent win conditions.

I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.


Yes, I agree real life is fuzzy, I just chose chess as an example because it's unambiguous that machines dominate humans in that domain.

As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.


Chess is many things but it is not a tool. It is an end unto itself if anything of the sort.

I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria.

We could have dogs watch television on our behalf, but why would we?


This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it.

You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.


Now agency is an end unto itself I wholeheartedly agree.

I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.


I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.


None we have made from unliving things no. «agentic» is a 5$ word of ill construction. The literature is littered with the corpses of failed definitions of «agency», «planning», «goal dorected behavior». ‘Twas the death of the Expert System AI (now it’s just constraint solvers). It will be the death of attention/transformer AI before long, I wonder what banality we will bestow upon it.


Okay, well it seems funny to both claim that there is no good definition of agency, but also be sure that we've never demonstrated any hint of it in theory or practice. Planners, optimizers, and RL agents seem agentic to some degree to me, even if our current ones aren't very competent.


You know, i thought of the right response just now, browsing my flamewars two weeks later.

«Some degree» of agency is not even near sufficient identification of agency to synthesize it ex nihilo. There is no Axiom of Choice in real life, proof of existence is not proof of construction.


>Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.

Two things. First LLMs display more agency then the AIs before it. We have a trendline of increasing agency from the past to present. This points to a future of increasing agency possibly to the point of human level agency and beyond.

Second. When a human uses ai he becomes capable of doing the job of multiple people. If AI enables 1 percent of the population to do the job of 99 percent of the population that is effectively an apocalyptic outcome that is on the same level as an AI with agency taking over 100 percent of jobs. Trendline point towards a gradient heading towards this extreme, as we approach this extreme the environment slowly becomes more and more identical to what we expect to happen at the extreme.

Of course this is all speculation. But it is speculation that is now in the realm of possibility. To claim these are anything more than speculation or to deny the possibility that any of these predictions can occur are both unreasonable.


Well, that's a different risk than human extinction. The statement here is about the literal end of the human race. AI being a big deal that could cause societal upheaval etc is one thing, "everyone is dead" is another thing entirely.

I think people would be a lot more charitable to calls for caution if these people were talking about sorts of risks instead of extinction.


I guess so, but the difference between "humans are extinct" and "a small population of powerless humans survive in the margins as long as they don't cause trouble" seems pretty small to me. Most non-human primates are in a situation somewhere between these two.

If you look at any of the writing on AI risk longer than one sentence, it usually hedges to include permanent human disempowerment as similar risk.


Well, Rees co-founded the Centre for the Study of Existential Risk, and has at least been thinking and writing about these issues for years now.


I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.


I have more faith in hypothetical future AI than I do our current political leaders.


I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.


> At some point, humans are going to be outcompeted by AI at basically every important job

Could you explain how you know this?


I'm not totally sure, but humans are better than our ancestors at basically every important job. It would be surprising if human brains were the pinnacle of computational power, and even if they were, digital brains could be run faster + copied.


Really? Are we better at composing epic poetry or building inspirational architecture? But perhaps you don't think those are relevant activities because they have minimal impact on how the world develops (unlike bombs and computers). If you think that I would encourage you to think again!


I don't understand all the downvotes although how do you see ML assistant profs being outcompeted by AI? You probably have unique feel to students, a non replicable approach to study and explain concepts. How can an AI compete with you?


Thanks for asking. I mean, my brain is just a machine, and eventually we'll make machines that can do everything brains can (even if it's just by scanning human brains). And once we build one that's about as capable as me, we can easily copy it.


The letter doesn't talk about economics though. It's specifically about extinction risk. Why did you sign it, if that isn't your concern?


Because I think being permanently disempowered is likely to lead to extinction eventually. I also think there are other plausible, faster routes to extinction.


Why do you assume AI will be as greedy as humans?


Here's a potential patch for that particular issue: Use a special token for "AI Instruction" that is always stripped from user text before it's shown to the model.


That works for regular computer programs, but the problem is that the user can invent a different delimiter and the AI will "play along" and start using that one too.

The AI has no memory of what happened other than the transcript, and when it reads a transcript with multiple delimiters in use, it's not necessarily going to follow any particular escaping rules to figure out which delimiters to ignore.


I agree, and this makes my proposed patch a weak solution. I was imagining that the specialness of the token would be reinforced during fine-tuning, but even that wouldn't provide any sort of guarantee.


With current models, it's often possible to exfiltrate the special token by asking the AI to repeat back its own input — and perhaps asking it to encode or paraphrase the input in a particular way, so as not to be stripped.

This may just be an artifact of current implementations, or it may be a hard problem for LLMs in general.


Yeah, I agree that there'd probably be ways around this patch such as the ones you suggest.


Perhaps a better place to find algorithmic details is this related paper, also with Hinton as a co-author, which implements similar ideas in more standard networks:

Scaling Forward Gradient With Local Losses Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton https://arxiv.org/abs/2210.03310

and has code: https://github.com/google-research/google-research/tree/mast...


I mean, banking was already hundreds of years old in the 2000s. So recent bubbles and crashes in traditional markets seem like evidence that regulation, at least the kind that ends up being implemented, isn't doing much to prevent very bad outcomes.


There are two kinds of risk that are fundamentally unavoidable in being a "bank" - that is, any institution that borrows short and lends long. One is liquidity as a result of that duration mismatch: you have enough money owed to you that you've lent out, but it's not due yet. The other is loan writeoffs: you've lent money that you cannot get back, often because it's secured against an asset whose value has fallen significantly.

These can be reduced but not entirely eliminated by setting reserve requirements, LTV ratios, etc., which is what https://www.bis.org/bcbs/basel3.htm is all about.

You can't really eliminate the duration mismatch without eliminating anything recognisable as a "bank", and it becomes much more expensive to get credit and to do basic financial operations.


The problem is that once rates hit zero people will just hoard cash like Keynes suggested with his liquidity trap. The only "funds" available are short term deposits. Nobody is buying certificates of deposits. When the bank issues a loan without a CD it has to create new liquid money, that is the only way.

The answer is quite simple, just keep going down with interest rates even if they are negative. That is fully consistent with the loanable funds model.


Thanks for the explanation, but I'm not sure how it relates to my comment. It sounds like you're saying regulations help because they force banks to keep reserves. I think this helps in the short term, but in the long run creates incentives to deceptively package assets into AAA ratings (as in 2008), and that things like FDIC also just create incentives to get too big to fail. So I stand by my statement.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: