I’m just re-reading Blindsight by Peter Watts, a novel that is amazingly prescient in — amongst other things — its predictions of this coming economic upheaval.
Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying.
This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there.
Back to the economic aspect:
Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless.
PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!
>This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s
I'm a huge fan of Peter Watts' work, but to be fair that is at the very least a close-cousin to the Chinese Room thought experiment that's been around for a while.
Yeah that’s the thing; LLMs like all software are just the latest codification of existing theories and memes, that’s exactly what they’re trained on. That’s exactly the path we envisioned to create them decades ago.
The concept wasn’t new in 2006 either.
Not really seeing the “now that’s sci fi!” emotion here.
Most of our historical compute problems were coupled to network reliability, performance. Those bottlenecks are largely solved; oh wow explosion in AI progress.
This isn’t new it’s more like the Higgs; just needed to wait for hardware.
Yes and no: good science fiction takes a trend and extrapolates. Extrapolation should be based on logic (to be good - but you can also just aim for entertaining and go wild).
The details might differ, but it's worth thinking about whether the implications make sense. Watt's certainly asks a lot of good questions which other orders have also asked - i.e. Accelerando[1] is very interested in the idea of "what happens to mankind when economic systems can no longer be participated in by recognizably human minds?" - taken out to the limits of known physics.
Agreed. The concepts were floating around before, but nothing presented the idea of an LLM quite as coherently and presciently like Rorschach did. (I hasten to add that this is in my own limited experience - I am an SFF fan but I have not read every book in existence.)
As a layman with respect to LLMs, I feel like Blindsight and its perspective of the chinese room puzzle is required reading. I wonder if people working in AI feel the same and I'd be really curious to know how they feel about the book.
I don't think it is described as non-sentient in the book, but only as lacking consciousness.
Consciousness not being required for sentience is one of the two major themes of the book (the other being that consciousness is an evolutionary dead end).
Still your example is very relevant. I very much believe that ChatGPT, for example, is not conscious. The question is if it has some amount of sentience.
Sentience without consciousness is just cartesian dualism again: society without people; forests without trees; etc.
Sentience is a power of consciousness which is a power of the body which is a power of the cells of the body.
There is no dualism other than in the imagination: one could imagine some fuzzy distant "forest" without thereby imagining trees -- so much for the imagination.
I think sapience would be a better word than sentience. But with regards to the process talking to the crew I think the book explicitly said that it didn't have any sort of comprehension of what it was saying. I belive the Chinese room was an attack from rorschach similar to the perceived attack from humanity. basically it wanted to waste their time and energy on something pointless.
I just finished reading this book after someone in Hacker News plugged the book last week. I agree with your assessment. The book explores different types of intelligence.
Similar to the book Solaris (there is a new English translation, and I recommend both the US and Russian movies).
I rate Iain M Banks as my favourite author of all time, but the AIs in his book have a very different "flavour" compared to the intelligent-but-not-sentient aliens and AIs in Blindsight.
An interesting aspect of the Culture Minds is that they're described as hybrids of human-like / neural-net / organic intelligence and also direct numerical compute and storage capacity. They can both "think" and "compute", something few if any current AI models attempt.
A well-known limitation of current LLMs like ChatGPT is that they are atrocious at even basic arithmetic, whereas a $2 pocket calculator from the 1980s outperforms them in this respect!
I'd love to see an attempt at a hybrid architecture where the neurons are also connected to a numeric compute component of the architecture.
Think of parsing an expression such as:
let foo = 5 + 10
ChatGPT parses this as:
"let", " foo", " =", " 5", " +", " 10"
Essentially, its lexer produces a series of token IDs that don't contain the numeric values of "5" and "10":
[1309, 22944, 796, 642, 1343, 838]
Programming language lexers would parse that as:
[LET, ID("foo"), EQ, NUM(5), PLUS, NUM(10)]
In other words, lexer tokens in programming languages can carry baggage such as numeric values.
What if we lifted this restriction and made an AI embedding where every token had both an ID and a value? For literal text just set the ID to NaN or whatever.
E.g.: assume that "integer constant" is ID 1000, then an LLM might be fed this input instead:
Then the model could be wired up such that the additional tensor dimension can be sent to both the neural net or a direct compute block that does various kinds of maths, such as adding all inputs, multiplying all inputs, etc...
Love the culture series. Though I would say it’s on a different tangent, in that AIs will absolutely eclipse us but will hopefully self align on being morally good because we are cute pets ;)
What’s interesting in the culture series in light of the AI alignment debates is that the culture aligns every mind to its task at “birth”. A picket ship, falling outside the normal moral constraints, is a very different set of personalities and morals than a GSV such as wisdom like silence
> Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”.
Decoding animal communication using machine learning is a popular research topic. See the current issue of Science, for example: Using machine learning to decode animal communication (https://www.science.org/doi/10.1126/science.adg7314)
Yes, but we don't heed the warnings their stories foretell, because it's just science fiction, and obviously if it happened in a fictional book it could never happen IRL.
I don't like Blindsight or Echopraxia, I just get caustic misantrophic vibe from Watts' writing. The whole materialist/hyperintelligent/nonsentient thing tastes very Marxist.
I don’t understand why economic disruption and human obsolescence isn’t considered an existential risk. We could end up in a world where 99.99% of people are redundant and useless cost centers for our GDP maximizing economic paradigm. In that case, you don’t need killer robots in order to push humanity towards near-extinction. The invisible hand will smite them.
But don’t worry, TED Talk attendee. The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian. So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.
It's a bizarre and blinkered article. The most immediate dangers are economic. We already have a hopelessly unstable economic system, with increasing swathes of the population economically disenfranchised. AI is more likely to accelerate that than prevent it.
The other immediate dangers are social and political. When one person with resources can run an AI-enhanced troll farm and social media PR engine - not even remotely science fiction - we have a serious problem.
Those threats are already politically and culturally existential.
And that's before anyone has even fitted a gun to an AI-powered autonomous drone. Or gunship.
AI is inherently conservative because it reinforces hierarchy. The AI-poor will have far less political, cultural, and personal leverage than the AI-rich.
Essentially it will have the same effects as money as a cultural and political practice - but much more so.
To me, the hype around the economic consequences feels like cover for the collapse that is already in the cards. So I agree, when the house collapses, it won't be because AI took our jobs, but the politicians will have their scapegoat.
They don’t need to be consumer goods. $1000 spent on clothes is the same as $1000 spent on GPUs as far as GDP is concerned. Headless businesses swapping dollars around creates plenty of GDP with few to no people involved.
A common fallacy is thinking that transactions exist without a human at either end of it (usually due to many layers of complexity in between).
You can talk all you want about companies selling to companies, etc., but at the end of the day everything on earth is owned by a human eventually.
Even high-frequency trades between two hedge funds are done with capital that was supplied by Limited Partners who probably are acting on behalf of pensioners.
The scenario you're describing is one of extreme wealth inequality, which is a real problem, but one that's supposed to be solvable through democratic means. Stopping technological progress isn't going to solve it.
Sure, everything is owned by a human. But does it have to be 8 billion humans? Why not 8 million or even 8 thousand? If humans provide literally zero economic value and their costs are significant, what can our economic systems say about whether they should even exist? It’s Macroeconomic Changes Have Made it Impossible for Me to Want to Pay You [1] but on a global level.
Yes, but at our current scale? At first glance it really seems that our simple economical metrics are a good way to have billions of humans living largely peacefully together – while ruining their environment of course.
The scenario you're describing is one of extreme wealth inequality, which is a real problem, but one that's supposed to be solvable through democratic means. Stopping technological progress isn't going to solve it.
The broken window fallacy is a controversial axiom that implies demand generation isn't a meaningful way to increase GDP (it comes from the camp that believes output capacity is the only true measure of GDP).
This concept became particularly controversial during the Cambridge Capital Controversy [0].
If you believe that GDP is an exogenous measure of what society can produce, you run into oddities like global GDP changing trajectory in 2008 [1] (i.e. we never 'caught up' but did the US housing crisis really cause the human race to lose our ability to create more goods and services?).
On the other hand, if you believe demand drives GDP, then why don't we just demand ourselves into more wealth? Shouldn't natural constraints like resource scarcity then drive GDP (i.e. the original underpinning of the broken window fallacy)?
In either case, the broken window fallacy is far from an agreed-upon axiom.
If I understand your point, the broken window fallacy is to say that a child who breaks a window has actually done something good for the economy by keeping glass makers and installers employed. Obviously this isn’t true because those people could be building new things instead of fixing the old.
Well, a huge portion of our economy is in fixing the old. Why waste so much money on healthcare for humans when we could use it to build bigger and better things. We have whole portions of the economy like food production whose entire purpose is to keep people alive. What a waste. The far more efficient option is to eliminate those people and let machines do the work. Even if GDP goes down, ultimate productivity goes up.
The same can be said for entertainment, hospitality, most education (we spend 20+ years educating people before they can meaningfully contribute) and marketing. A broken window theory might say these industries contribute to the economy and give people jobs, but actually they accomplish little.
>I don’t understand why economic disruption and human obsolescence isn’t considered an existential risk.
Because that's the goal of our economic system! If everything could be made with no human labor requirement, then the price of these goods is going to trend towards zero. The disruption is going to cause pain, but ultimately that's what we've always been after.
Well, no. The goal is to maximise profit. One way to do that is to minimise labour costs. But at the extreme, if there are then no consumers who can buy your goods there is no profit...
No, it isn't. Profit is meaningless, money is inherently worthless.
Capitalism exists to encourage the production of (new) goods and services. The system is set up to encourage people to take their capital and invest it for this purpose.
Investments help business ideas that would otherwise never have been made, because the person with the idea couldn't afford it. In return the investor gets a slice of the potential future profits. But the goal of the system is the products the company creates.
Money is worthless. It only carries value based on what you can trade that money for. If I have $10, but a cup of coffee costs $1000, then my $10 are meaningless. But if a house cost $1 then my $10 is very valuable. And what you can trade your money for are the goods that are produced by the economic system.
You are talking past each other; You're discussing very different things - capitalism's terminal utility to humanity versus capitalism's direct operational mechanisms - without actually disagreeing substantively.
The goal of the investor isn't to provide goods & services, it's to get profits in whatever way is possible. He doesn't give a shit about goods and services. This tends to diverge from the interests of non-investors.
We have an investor class that is presently largely in control of our government and our society, whose interests as a part of the capitalist system are currently at a few percent remove from our interests as people who enjoy goods & services. That divergence, as well as the accumulated privilege opposed to our interests that this class derives from control of our government and our society, appears to grow over time with uninterrupted peacetime improvements in automation, and AI is promising almost unimaginably large improvements over the next decade.
The status quo socially liberal, economically neoliberal society is not sustainable even as a polite fiction in the face of that kind of shift.
Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world
As we no longer weep for serfdom-slavery, you don't have to weep for wage-slavery, but you do have to replace it with something else more functional if our society is not to completely collapse. And it's looking like we will be empowering our de facto leaders with all the blood and treasure in the world to stop us from replacing it with anything more functional. Every civil war in the world up to this point has relied upon human soldiers with their own interests and allegiances, faced with a brutal choice of risking their lives to kill their countrymen - people who have the option of not pulling the trigger. This is not going to be the case for much longer.
And that's all assuming we don't get paperclipped, assuming that Ayn Rand is the penultimate cosmic horror and there is nothing darker in the possibility space, no pressure on the right side of the Drake Equation. I don't know about that.
The more salient term isn't "capitalism", but rather "capitalization".
Most investment is done by converting anticipated future value (modulo some degree of risk), and encapsulating it into an asset price, which is relative to the opportunity cost of alternative investments, and the current discount rate.
And capitalization doesn't have a goal beyond honoring those future returns (or "embedded growth obligations"). It doesn't care whether those returns come from innovation, production, rent-seeking, or even a "greater sucker" Ponzi scheme. Investors (and their financial agents) operate almost entirely on Make Number Go Up ("buy low sell high"), and any incentives that result in win-win increases to human flourishing are entirely secondary effects.
the purpose of an economy is to support and ensure the existence of its host organism, its country. there is no rule that says any specific amount of human labour has to be involved.
> If everything could be made with no human labor requirement, then the price of these goods is going to trend towards zero. The disruption is going to cause pain, but ultimately that's what we've always been after.
I would argue that our economic system is built on the accumulation of power, not an ultimate goal of zero marginal cost production. That's why cheap drugs keep getting expensive, among a litany of other examples.
Everything already costs basically nothing. There's a Moore's Law for every industry, and if you go into mature industries you'll see how easily they ignore it. There's open collaboration to keep prices "stable" in most of them.
I think that's mostly due to consumers rather than the industry. The industry keeps inventing new stuff that they can charge more money for because that's what the people want.
You can go buy decent t-shirts with no logos for €2.50 right now. And yet most people walk around with t-shirts that cost €10-50 with some logo on them.
This becomes apparent when you look at common food like potatoes, milk, cheese, bread etc. All the competition more or less provides the same product for about the same price. If they could go cheaper while still being profitable they would.
There was a time when 90% of the labour force worked in agriculture. If I lived in the 1600s I'd be terribly concerned that mass unemployment would coincide with large increases in agricultural productivity.
However, when the agricultural sector became more productive the 'winners' wanted bigger houses, travel, education, healthcare etc and this provided enough opportunity for the displaced workers to find jobs. These new sectors, over time, even offered more wealth and better working conditions to their workers than the old agricultural sector.
This is a common trope, but the reality is that the winners bought the farms from folks who couldn't compete. The ex-farmers did not go on to find better compensation, they just became poor. Their children perhaps found a larger salary in the information economy, but ask them how many acres they can pass on to their grandchildren.
It’s important to distinguish between the farmers pushed out of agriculture and those pulled towards the industrial/service sector.
Some of the farmers pushed out of agriculture were in a truly terrible position. Some effectively exiled into other countries (e.g highland clearances in Scotland) or left with no choice beyond military service in poor conditions.
Other ex-farmers were pulled into the high wage new economy sectors. I imagine this was a comparatively happier experience, although not without its difficulties. For example, the shipbuilding or railway workers. In a more modern example, the Chinese factory worker.
Yeah, I've been trying to figure out how to feel about china "lifting a billion people out of poverty" for a long while, because if I squint it kinda looks like land owners getting displaced by a government that would rather work then to death in cities than allow them a meek existence in the countryside. For young people, having a wage and the agency to spend it how they choose is worth a lot, but I still wonder how much choice they really have. I suppose it's a matter of whether you see a future for yourself down on the farm vs where all the economic growth and excitement lies.
But even given that the kids will be alright, my point was that the breadwinners displaced by progress do not find an alternate income - they rely on the next generation to adapt.
Outside of North America, it was relatively rare for farmers to own their land. Tenant farming and serfdom were more common in Europe/Japan. In China, as a general principle, the government itself owned all land during the Mao era.
Farmer-landowners are much more financially resilient to technological change because (generally) the same technologies which reduced the value of their labour increased the value of their property.
I suspect this difference in historic land ownership had a big impact on subsequent politics and it's something I'm looking to read more about.
now that I think of it, if farmers had already owned their farms, Mao & co wouldn't have gone through the trouble of redistributing it via Land Reform.
I might suggest that drawing boundaries around a plot of land and calling it "owned" was the beginning of politics, as it represents collective agreements and mechanisms to arrive at them. Have you dug into Elinor Ostrom's work around the tragedy of the commons being an unrealistic model given that people don't just let other people's cows overgraze? We create rules and sanctions before the resource collapses (or rather, we try to; and that attempt to prevent collapse could be called government).
Given that the number of people in agriculture declined from 90% to 10% over the past 200 years or so, I'm less worried about disruption like this. People are remarkably retrainable, and just because current jobs no longer exist doesn't mean we won't create new ones out of thin air (like marketing, or Youtubers, or film lighting specialists).
How much of that would be due to the fact that we increased productivity on essential goods which inevitably benefitted everyone to some extent? It may be very different when a large portion of the jobs being destroyed are mostly useless to the productivity to start with.
This is just repeating the fallacy that the person being replaced is going to be the blue-collar worker.
But it isn't...it's the office knowledge worker. Anyone cruising in the middle bucket where you don't do creative research, but you do sit in front of a computer. AI is coming for those groups.
The blue collar worker? They'll be fine for decades. Anyone who gets out there and manipulates the physical environment, won't be replaced any time soon.
Maybe this is why billionaires and hundred-millionaires are the ones so concerned about existential risk. A collapse of consensus in our current and historically not-that-old system of property rights is an existential risk -- to them.
> The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian.
Anyone doing physical labor is going to be just as necessary in a world of artificial mind, it's those knowledge workers of the service economy that will find their skills automated.
As soon as a significant share of the population is „irrelevant“ and treated as such (read: no good welfare, …), capitalism will end quickly. „Who owns the machines that generate value“ will shift from individuals/corporations towards a public good, and Marx ultimately was right.
Yeah, but capitalists have all the guns. Not directly, but the hired guns (police, military), work for the capitalists.
In fact, if you watch The Rules for Rulers, there’s the possibility of turning the entire world into a 3rd world corrupted structure: a few people at the top enjoying the riches of full automation, a few more people with guns to protect them from the unwashed masses, and everyone else.
If democracy really comes from the fact that we need everyone’s labour to make a country, we may actually lose it as automation makes most of us economically irrelevant. We need an alternative before the window closes. If we’re lucky it won’t ever close, but to be honest I won’t bet on it.
The tipping point is when some „unwashed masses“ decide to go to a billionaire mansion and explain them (forcefully) that they aren’t billionaires any longer, and neither the security staff nor the called police will step in, since they are part of the masses (family ties, defunding, …). Why should they do cruel work for a salary if they can take their part of the wealth directly?
That’s possible as long as the police aren’t robot, and the mansion isn’t orbiting in space. Now Elysium doesn’t have to happen (and good luck with the giant space station) but a sufficiently walled garden guarded by sufficiently automated defences supplied by a sufficiently automated workforce could in fact repel anyone else trying to get in with nothing more than small arms.
Though I reckon an organised unwashed mass could most likely do something.
I see the reasoning but in practice it’s just sci-fi I believe. Even if you have the _perfect_ walled garden… there you are with your wealth. What will you do with it if everyone else refuses to deal with you? What are shares of corporations worth if no one cares that you own something? Why would the military listen to you, when they could obliterate your escape room and take over your wealth directly?
All this stuff you mention relies on the the „fact“ that capitalism and guaranteed private property rights are absolute truths - when in fact they aren’t, actually more like a temporal fashion wave in some parts of the world, and faith in this destructive obsession in „the economy“ is on the decline finally. :-)
>So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.
Amusing, usually rationalists are made fun of because the concerns about Roko's Basilisk are not performative.
Well, it’s a problem with any economic system which uses labor as a proxy for value/rights. If my political power is predicated on my ability to withdraw labor or move to a new country, it’s not going to be good for me when my services are no longer required.
Reminds me of the point made by Ted Chiang that (roughly, from memory) when people express fear of technology, quite often what they really fear is capitalism and how it will use the technology.
This goes back all the way to the Luddites, who weren't actually anti-technology/progress. They were opposed to how its benefits were captured and by whom.
and EvP - it will take something away from our human experience
None of those really capture the weight of tradition though. Loss of control is a hard pill to swallow. Feels better to keep on doing the old thing that kinda works, instead of the new unknown.
We're seeing this happen a lot with "AI" - now that it's a super popular topic people use it as the lens though which they project their general political grievances. It gets more attention.
This is a truly awful article. I'm mildly positive towards AI because I think people tend to react with fear and examine the risks without equivalently examining the rewards, and I think that's happening here and the rewards seem amazing. But the arguments of risk are serious and plausible and this article is just lazily dismissive of them.
> AI is not competing for resources with human beings.
A superintelligence would absolutely be competing for resources (mainly electricity and cooling) with human beings.
> Rather, we provide AI systems with their resources, from energy and raw materials to computer chips and network infrastructure.
A superintelligence will be able to convince anyone it wants to do anything for it. Even the previous President did a pretty good job at causing adulation and democracy-threatening personal loyalty from people he had not actually met in person. Imagine how amplified and personal the effect could be.
> For now, AI depends on us, and a superintelligence would presumably recognize that fact and seek to preserve humanity since we are as fundamental to AI’s existence as oxygen-producing plants are to ours. This makes the evolution of mutualism between AI and humans a far more likely outcome than competition. Moreover, the path to a fully automated economy — if that is the goal — will be long, with each major step serving as a natural checkpoint for human intervention.
The author is literally relying on us outsmarting the superintelligence (collectively, with coordination!). It doesn't sound like they've come to terms with the concept of superintelligence at all. Or with the state of human global coordination problems, come to think of it.
> AI cannot physically hunt us.
This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process.
> AI’s impact on the climate is up to us.
Our own impact on the climate is up to us. Our collective decision-making is.. suboptimal.
> If we really think that superintelligent AI presents a plausible existential risk, shouldn’t we simply stop all AI research right now? Why not preemptively bomb data centers and outlaw GPUs?
You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.
>This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process.
I'm stunned at the number of people who try to make this argument. The operators of Radio Télévision Libre des Mille Collines never had to leave their broadcast studio. Goebbels never had to do anything but get in front of a microphone or a typewriter. Still, millions were violently killed. We bicker about LLMs' ability to write C code that will compile, but their best abilities are to gaslight, emotionally manipulate, lie and create FUD.
I thought "Ex Machina" portrayed this expertly, that a superintelligence with the motive to escape would develop psychopathic manipulation well before anything resembling empathy.
> A superintelligence will be able to convince anyone it wants to do anything for it.
This "just so" claim has always struck me as quite a reach. Humans are manipulable, yes, especially by (perceived) group pressure, but there are still limits. Especially if the human can easily just mute the AI's output. It's like when people believe that a sufficiently clever hacker can get a given program to do anything. There are millions of security vulnerabilities, yes, but you're unlikely to find a RCE in the program `true`. But instead of the 80s wizard hacker trope we're going back to the older B-movie hypnotist who can take full control of any person by talking
> You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.
There's a few links in the article - which person are you referring to? Being the most prominent voice calling something unsafe is not an inherently valuable qualification. The most vocal opponents of nuclear power tend to have a poor understanding of it
> This "just so" claim has always struck me as quite a reach. Humans are manipulable, yes, especially by (perceived) group pressure, but there are still limits.
You could reimagine it as "convince most people to do anything, including killing people who are unconvinced" if you prefer.
> There's a few links in the article - which person are you referring to?
> Being the most prominent voice calling something unsafe is not an inherently valuable qualification.
That's true. But the article does something odd instead of making that critique -- it presents bombing GPU data centers as a reductio ad absurdum, suggesting if you really believed in AI risk then you would have to be willing to consider doing that. The people who really believe in AI risk already are willing to consider it, though.
> A superintelligence will be able to convince anyone it wants to do anything for it.
What is the evidence for such an extravagant claim? Is there a correlation between IQ delta and persuadability that has no degradation as the lower IQ rises? Especially given that there are plenty of people far more intelligent than the President in your example who are much less persuasive.
Just define intelligence as "ability to do a task". The human brain isn't magical, it can simulated, so we know for certain that a powerful enough AI can be at least as skilled at any given task as the most skilled human who has ever lived.
So for any given person, consider how well the most effective persuader out of 8 billion people would be at persuading them. Maybe they can't always persuade them of anything, but pretty close. This is the lower bound for an AI's maximum theoretical abilities, and it's already enough to rule the world.
It's very hard to transport humans to other planets, but a bunch of robots are already exploring Mars. I see no reason why a superintelligence would want to confine itself to Earth.
Why compete with humans on Earth if there's the whole Solar system to exploit?
you completely fail to understand it. when you understand it you wont use the word compete — thats like saying the AI will have to compete with an inanimate object. there will be no competition, no struggle and no choice in the matter. if the singularity kicks off, humans will be rendered completely helpless. you clearly dont realize this because of the ridiculous notion that you harbor of humans competing with machines. trying to predict their motives is totally useless — you cant do it.
The foundation of this article is that there is a zero sum game for resources for AI safety. I think we should reject this notion and improve AI safety both in terms of existential and non-extensional risk. If more resources are needed and can be used effectively for dealing with safety we should instead take resources away from propelling AI forward and apply them to safety.
I see this argument against existential risk alliance – that AI needs humans to provide it with resources- this we can hold it hostage. I like to point out that power plants are increasingly since the pandemic remotely operable- that AI will increasingly be able to hold us hostage by hacking the infrastructure we need to survive. Throw in the ability to use some robots and bio weapons to keep humans from taking them back easily.
The other aspect is that AI doesn’t have to properly conclude ahead of time that such an attempt has a high probability of success. Its program just has to decide that this will help it achieve some objective.
Arguments in the first part are really bad. I haven't read about ai safety that much, but still see it.
For example, ASI may be able to automate manufacturing really quickly, maybe be able to convince humans easily to do what it needs, etc
Moreover, why ai would not be competing with us for resources? Materials on earth are finite.
Also, most likely scenario is not evil AGI, as it outlined in the article, but indifferent. And basing reasoning about AI on Darwinian evolution is barely useful.
Also, talk about this tribal people AI is ironically even worse then silicon valley narrow viewpoint: it is Californian liberal virtue signaling.
Is there a difference between evil and indifferent if you are on the receiving end? Humanity is largely indifferent to animal welfare and animals would qualify it as evil how we treat them.
I am currently a bit of a doomer and if I had a magic wand, I'd pause AI research for the foreseeable future because I don't think it's creating a better world for children, at least from what I've seen so far however...to your point.
I think we're only like this to animals because we are animals (we forgot), and we compete for the same type of resources they need, water, protein, calories, we're not intelligent enough to get ourselves out of this situation, but if we could, we could.
If we could we'd be able to grow tasty meat in a lab, reverse climate change, restore forrest, synthesize water cheaply, live in floating cities above beautiful habitats,,,etc etc. We're just not that smart so we do dumb shit, like factory farm.
I do agree that we might live peacefully with AIs because they need different things. Two points:
We still compete for space and energy at the very least.
Humans are sympathetic to animals because we are alike. We don’t want animals to suffer. However, will an AI be sympathetic to humans because we „think alike“? We don’t know.
How could something survive with other entities without things like fear and empathy though? It would just be constant chaos and nothing could get done. With no fear, you could just wipe yourself out and not thing twice about it. Without empathy, same, you'd just be killing things permanently.
I do wonder if the quest to build AI will teach us a lot about why the thing we take for granted exist. Why have fear, limits, empathy, anger etc.
I suppose bacteria is a bit like this but it’s limited in its capabilities to evolve too quickly and wipe everything out?
No, but I think that was exactly the point being made: that the article suggests evil would be required for existential risk, when indifferent would suffice.
Although I disagree with the conclusion, this looks like a pretty decent article in terms of detailed thinking.
I just think that we not only need to work on all of the more immediate problems but also worry about less immediate problems that are existential risks.
Also people always say "AGI" and they mean different things but I am guessing most now mean something like a digital living human/simulation of human when they say that. Mixed in with some blurry connotation of it automatically becoming godlike in power.
That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating.
Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology.
If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk.
Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous.
Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.
imo GTP-X is going to need to be much much “smarter” and consistently accurate (which may be fundamentally impossible) before it’s going to be “widely deployed for military and industrial decision-making” anywhere.
I don't think a piece of (AI? no, software? Meh?) thinking material needs to be above average human in its capabilities, or in any way consistently correct in its thinking, or in any way all that broadly learned, or in any way accountable - to cause significant mayhem if given half the chance.
There is this image of super-human, much faster, much broader thinking in the collective. When a bog standard human, drunk and in a foreign country, can do plenty. By this measure, chatGPTs are plenty intelligent enough - just not yet connected enough to tools, a wallet and vague orders or intentions.
Sure, a llm might not be general enough/trustworthy enough yet, but transformer based analysis of drone sensory data surely is already being tested/developed? I'm no engineer but I think it is fairly obvious they'd train their own models not trust an off the shelf general LLM
The general discussion is definitely prematurely focused on some sort of end-boss fight, while AI and large scale data collection are causing serious harm and privacy concerns to real humans already. It’s only reasonable to expect that this increases in the future.
I would like to see more discussion focussing on these pressing issues.
Also a pause on AI development is not an option, not a solution and is digressing from the issue at hand. Big capital with their hands on big valuable data will support anything that distracts here.
Finally, I do think that AI also brings many positives, I am not against the technology itself at all- we shouldn’t be.
name a single reason why a pause is not an option. dont say why it would be hard or unlikely or inconvenient or whatever, to exclude it as a possibility means it has to be physically impossible.
of course it isnt physically impossible. AI requires large compute, large resources. you cant advance the state of the art inconspicuously. the only reason you say it isnt an option is because you dont consider AI an existential threat, really, and therefore extreme measures arent justified. but they actually are.
We are already facing numerous existential threats at the moment, even ones that are deliberately made to be an existential threat, take nuclear weapons.
“Hey fellow nation states, let’s not create nuclear bombs, but let’s pause developments and keep the current power dynamics fixed please” is completely unrealistic. Mind that creating nuclear weapons actually requires technical operations that are factors more complex than “large compute” in many aspects… like… enriching uranium and working with supercritical masses.
> you cant advance the state of the art inconspicuously.
That's not even a valid request. There is obviously no valid reason to require such a pause in the first place. You've been watching too many scifi movies and don't understand the technology.
It can't be repeated enough apparently, because people still whine on about skymesh and "but IFF the robots wake up" when the real risk is what people believe machine derived "facts" are (I quote facts because they are highly conjectural inferences from models, and often diverge from fact pretty rapidly)
aussie here, former dole bludger, still know people who live off the welfare system, hide their side income, and are proud of it. There was sooooo much warning for robodebt that they all made sure their taxes were clean before the first demands went out, but a few still got through. Mostly accurate, but a few dodgy ones too. Of course, as we know, it was a dragnet and the human suffering produced was excessive. Considering the depth of the corruption, it was clear the system was incapable of correcting itself, so what else is there to do? How do you get help to the people with real need when there's so many people cheating? Robodebt was big hammer approach.
The distant future. Like five years.
Self-aware (or a simulation thereof) AGI with self-directed motivation and the ability to self modify is barely a half step from ASI. And AGI could happen in the coming year. Five if you wanna be pessimistic. Never if you’re extraordinarily pessimistic but that’s looking unlikely. If it’s possible it’s possible now.
This isn’t the Yellowstone super volcano that might blow in the next billion years.
People who talk like this are idiotic at best.
Granted, the other concerns are real too but the accusation that the concern of existential risk is being used to hide the other known problems is dangerous in the extreme.
Why can't a (very optimized to the point of not really being a) markov chain be smarter than a person? What are your requirements for outputs, abilities, and internal methods?
99% of what people do day to day is repetitive/unoriginal so it is not too surprising that this is happening, but I wonder if we are really any closer to an AI that can develop a novel and important scientific theory or mathematical result for instance? When can we expect AI to resolve the quantum gravity issue? If it can't do that, but eventually humans do, what does that say about AI I wonder?
We're no closer. Like you say, we're showing that more and more of what we do involves no actual intelligence, but it's in the margins that real intelligence exists. We're no closer to simulating (or even describing) the unique aspects intelligence than we were in the 50s. We just have Eliza with 200 billion parameters.
"Mathematicians at the University of Oxford used the AI to discover a surprising connection between algebraic and geometric invariant knots, establishing a completely new theorem in the field."[0]
The goalposts will continue moving until morale impairs.
We live in a very confusing, transitory time. Right now there's a large chunk of people who are convinced that we are a few years away from extinction if we do not ration out GPUs and bomb illegal clusters of compute, while at the same time there's a large chunk that believes that a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company). Absolutely wild.
"Convinced that we are a few years from extinction," is an exaggeration of the pessimistic/alarmed end of the spectrum of opinion.
All the forecasts I've seen have the period of danger spread out over at least 20 or 30 years, and no one claims to understand the topic well enough to be able to tell whether a particular model is dangerous even if given complete freedom to examine the source code and interview the creators.
Our inability to predict is in large part the basis of the pessimism/alarm because it means that the leading labs won't know when to stop.
Exactly. And here is an example of that inability in the much clearer case:
"How many years after 1945 ... would it take for the Soviet Union to get the bomb? Framed this way, it was a much easier question than forecasting the advent of AGI: The Americans were concerned with only one adversary and the development of a single technology which they already knew all the details of. American predictions of Soviet proliferation offer a highly constrained case study for those who would undertake technological forecasting today."
No, even the more-panicked wing is not betting the proverbial house on AI killing us within the next few years, although the possibility is there. The "don't expect to live much longer" sentiment is based on a probability horizon that extends over decades.
Can you give a name so I don't have to read a long article?
It is easy to find people who believe that it is possible (with some low probability) that AI will end the human race in the next 12 mo. (I'm one of them.) I haven't seen anyone however put most of their probability mass in the next 12 or 24 mo, and I'd like to find out what their reasoning is.
Sure, first name in the article I posted, Joep Meindertsma. They have an open discord you can join, but fair warning that it's entirely panicky people lamenting their inevitable doom.
> a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company).
I think the criticisms of the Boring Company are more "small diameter tunnels are a bad way to build high-throughput underground infrastructure, and autonomous cars will not solve that"
Sure, but have you ever been on the London Tube and noticed it's notoriously small and uncomfortable in parts? Have you also noticed how windy some stations get because the small trains fill most of the tube and act like an air piston?
Those tunnels are 12 feet diameter. Boring Co's tunnels are 12 feet diameter.
A modern subway system needs considerably larger tunnels.
I've only seen grifters, hysterics, or the ignorant talking about existential AI risk. A lot of people just nods their heads and go along, but I don't think there are many that have really thought about it (and understand what modern AI is) that really belive the risk.
It's more like political polarization where people dwell on something they don't agree with to the point where they treat it as if the world is going to end because of it, instead of in context. AI is more like the political party they don't like winning, potentially undesirable from a certain point of view but they see it as "existential".
I think it's important to see it in this light, instead of trying to actually debate the points raised, which are absurd and will only give credibility where none is due.
“Appeal to the stone, also known as argumentum ad lapidem, is a logical fallacy that dismisses an argument as untrue or absurd. The dismissal is made by stating or reiterating that the argument is absurd, without providing further argumentation. This theory is closely tied to proof by assertion due to the lack of evidence behind the statement and its attempt to persuade without providing any evidence.
So because he made one(+) bad prediction, all future predictions are inherently poor? It seems that may be a tad of a leap? Where is the line between "poor prediction" and "hysterics"?
"Deep learning will replace radiologists within 5 years" isn't as inherently crazy as it probably seems in hindsight. Sure, it didn't pan out, but I know a vet who works for an Ai company classifying scans for training and he says first hand the models have gotten a lot better. That said, he's still not worried about being replaced.
Probably things would become clearer if what we currently describe as 'AI' we instead term as 'Statistical Pattern Regurgitator' or similar.
Any serious attempt at AI would: need to be trained on reality - like people are trained; need to overcome the cardinality barrier - digital systems can only approximate analogue systems; and need to demonstrate spontaneous language emergence driven by survival in a social environment.
Why wouldn’t you call a human brain “statistical pattern regurgitator” as well, if it comes to this? Matching learned patterns is a big part of how brains work.
I don’t see why a strong AI _needs_ to be trained specifically on reality. Imagine an alien race that sees world in X-Rays and neutrinos. An alien AI researcher claims that a “true intelligences” must train on the same inputs, and all those systems trained on the part of the spectrum between infrared and ultraviolet only fake intelligence.
I don’t see why intelligence has to be analogue in any way. Analogue systems are inherently noisy and contain limited amount of useful information, so you can always faithfully approximate them using digital representations.
If I am not mistaken Deepmind has shown spontaneous emergence of communication around 5 years ago when training agents in multi-player team shooters.
you absolutely understand. such a key point. GTP discovers deep structural truth, and characterized this with a multidimentional heat map, about the system that produced the data its trained on. that comes from Stephen wolfram. if you trained it on clouds, it would have an autistic knowledge of clouds. what GTP is doing is exactly the same thing as human intuition. its not statistically driven, this is proven mathematically, thats why we havent seen it before. but its still just patterns and concepts, no intellect or logic.
the key observations: imagine a human with intuition so good and so powerful that you could compose an essay without thinking, as a reflex. people dont understand how insanely powerful this AI would be if it had an intellect to match its intuition. its coming. people dont understand that intuition is a necessary part of an intellect, that were already half way to true AI. these LLMs are a bundle of kindling.
One day, someone came up with the Transformer architecture for neural networks. The ideas didn't quite come out of nowhere, but their impact did - I don't think that anyone predicted the degree to which ChatGPT and its ilk are superior to LSTMs and the like. More importantly, nobody predicted the details, the way that the GPTs behave, the way they're willing to invent facts like they're spooling out plausible thoughts without being constrained to observed reality.
From a history-of-technology standpoint, the state of the art for generative AI went from LSTMs to ChatGPT overnight. There was no warning. Nobody could have looked at the Attention Is All You Need paper and predicted how much it would improve the state of the art.
And worst of all, we already know that AIs can be hostile. What else is a public corporation's corporate bureaucracy but a Chinese Room running an AI whose goal is to maximize quarterly dividends? What else is a government but another Chinese Room trying to survive? We talk about corporations being faceless or friendly, we ascribe them wants and needs and behaviors and habits, we give them names, and they suffer natural selection and mutation and reproduce and evolve over time so as to select from those bureaucracies that are most fit for survival. And this class of AI is already responsible for global climate change and the Cold War and so many other things that might already be on the path to push us into extinction.
LLMs are nowhere near as humanlike as a corporation. Corporations are bound by regulations and capitalism and basic human decency. It is absolutely possible to build an LLM whose fundamental goal is to kill people. All you have to do is tell ChatGPT to pretend it's playing a game of global thermonuclear war, and some joker on 4chan would do that in a heartbeat.
So what? Why does this matter?
We have no way to predict when the next revolution in AI capability will occur.
We have no way to predict what the next revolution will be.
We have no way to predict the capabilities of the next generation of generative AIs.
We have no way to predict how the next generation of AIs will "think", insofar as they do.
And because we have no predictions, we have no way to prepare.
So the real question is, how much risk are you willing to tolerate? Are you wiling to take a coin flip on it? A spin of the cylinder? A roll of the dice? Because that's what we're looking at here - the next generation of AIs has no reason to be any less inimical to human existence than the examples we already have, and AI will _only_ get better.
exactly. where were all these self proclaimed experts, who are now making confident predictions about AI, before attention is all you need? literally 100% confidently predicted AI like chatGTP was hundreds or thousands of years away, or would have if asked. these people are the real bots.
and i never saw a single person on HN or anywhere else raise concerns or even appreciate the true impact GTP would have today in 2018 when GTP was released and samples available! people coundnt get it right even when it was right in front of them. how can they think for a single second theyre getting it right now?
to my credit i recognized immediately what was going on in 2018 and to my knowledge i was the only person leaving comments warning people and begging people to support a pause of research. so by the only material metric, i am more qualified to make predictions than anyone here or even the experts who all confidently predicted that what is happening now would never happen in our childrens lifetimes.
i will never forget when hinton saw the light. when he started warning of danger and using the key words that are at the heart of the issue. and i realized i had been ahead of him. just because youre an expert in one thing, even the implementation of ML or AI, doesnt make you an expert on the broader consequences of AI.
Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying.
This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there.
Back to the economic aspect: Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless.
PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!