The counterpoint is that not doing so (implying some sort of infinite monetary loss if the entire human species is wiped out) would mean you want to spend every single unit of monetary value of the entire global economy to preventing this (which is also obviously nonsense - people have to eat after all).
So you have to put the monetary value somewhere (although you're completely within your right to question this specific amount).
I think what I’m trying to express is that it feels like the answer isn’t any amount of money, it’s just undefined, like a division by zero or trying to read the value of a binary register on a machine that’s turned off. I think Pirsig called it a Mu answer.
That's the most interesting application of capitalism-as-as-resource-allocation mechanism I've ever seen, that's something I look forward to thinking about more.
My immediate reaction though is to doubt the mapping of dollar to value - e.g., the 10 million dollar valuation of the human life, but also the valuation then of all the things that year-dollar-cost could be spent on. Many of those things probably don't map very well between true value, and dollar cost (my go-to example of this is teachers fulfilling one of the most critical roles to ensure a functioning society, yet the dollar cost paid for their labor being typically far lower than most other jobs).
And indeed, accounting for externalities (unmeasured or unmeasurable) is a tough economic proposition. If it weren't hard to account for every single variable, creating a planned economy would be easier (ish).
FWIW, there's a whole sub-field just dedicated to determining the value of life for various purposes (a starting link: https://en.wikipedia.org/wiki/Value_of_life). You may disagree with any specific assessment, but then you have to argue how that value should be calculated differently.
> I don’t think they can yet self improve exponentially without human intuition yet
I agree: if they could, they would be doing it already.
Case in point: one of the first things done once ChatGPT started getting popular was "auto-gpt"; roughly, let it loose and see what happens.
The same thing will happen to any accessible model in the future. Someone, somewhere will ask it to self-improve/make as much money as possible, with as little leashes as possible. Maybe even the labs themselves do that, as part of their post-training ops for new models.
Therefore, we can assume that if the existing models _could_ be doing that, they _would_ be doing that.
That doesn't say anything about new models released 6 months or 2 years from now.
They had been saying it was 10 years away for ~50 years, so that's progress. Soon it will be 1 month away, for another two years. And when they say it's really here for real, there will still be a year of waiting.
> And when they say it's really here for real, there will still be a year of waiting.
Indeed. Although, there's a surprising number of people claiming it's already here now.
And to describe the typical cycle completely, the final step is usually a few years after most people agree it's obvious it's already been here for a while yet no one can agree on which which year in the past it actually arrived.
> Although, there's a surprising number of people claiming it's already here now.
why is that surprising? nobody really agrees on what the threshold for AGI is, and if you break it down:
is it artificial? yes.
is it general? yes. you can ask it questions across almost any domain.
is it intelligent? yes. like people say things like "my dog is intelligent" (rightly so). well is chatgpt more intelligent than a dog? yeah. hell it might give many undergrads a run for their money.
a literal reading suggests agi is here. any claim to the negative is either homocentrism or just vibes.
Sure, I've been pointing out that literal sense myself, but to be fair, that's not what people mean by AGI. They mean real understanding, which is clearly missing. You just have to dig a bit deeper to realize that. One example is contradictory sentences in the same breath. Just last week I was asking Gemini 2.5 how I can see my wifi password on my iphone and it said that it's not possible and to do it I have to [...proceeding to correctly explain how to get it]. It's pretty telling, and no amount of phd-level problem solving can push this kind of stuff under the rug.
"Nothing dumb anywhere" is an unreasonably high bar for AGI. Even Isaac Newton spent 1/3 of his career trying to predict future events from reading the Bible. Not to mention all the insane ego-driven decisions like Hamilton's voluntary duel with Burr.
Sure, Gemini may spit out obviously self-contradictory answers 2% of the time. How does that compare to even the brightest humans? People slip up all the time.
There's dumb and there's incoherent. If a person would be incoherent at this level even one time, they would be well advised see a neurologist. Unless they are in some other way incapacitated (i.e. drunk or drugged). Same if they wouldn't be able to count the r's in "strawberry", attempt after attempt, getting more and more lost in again incoherent mock-reasoning.
I disagree completely - consider asking a color blind person to describe the color of flowers. Conversation would only be frustrating. This is analogous to LLMs seeing the world in tokens rather than characters, so character counts are simply not part of their input spectra in the same way that a blind person doesn’t get visual inputs.
Consider also all the smart people who get obsessed with conspiracy theories and spew out endless “mock reasoning” about them. Again, if “nothing incoherent anywhere” is your benchmark for intelligence, humans ain’t it. I mean, what would a computer say about a human that forgot where he just put his keys because he was thinking about dinner - “what, you can’t even store the last 10 seconds of history and search it?” Undergrads’ hit rates on mental double digit multiplication are probably <50%. In many, many ways we look completely idiotic. Surely intelligence is defined by what we can do.
Do you accept any positive definition for AGI, as in if they can achieve X result (write a bestselling novel, solve the Riemann Hypothesis) you would consider it intelligent? I find that negative definitions, as well as theoretical arguments about the techniques rather than the results (eg “LLMs cannot be AGI because they were trained the predict the next word”) to be basically useless for discussion compared to thresholds for positive results. The former will never be achieved (it is trivial to find cases of intelligent people being dumb) and the latter is totally subjective.
I partly agree about letter counting being an unfair test for the raw LLM. But I was thinking of reasoning models interminably rationalizing their incorrect first hunch even after splitting the string in individual characters and having all the data needed in a digestible format before them. Similar to, as you say, conspiracy theorists stuck in motivated reasoning loops. But - are these latter behaviors instances of human intelligence at work, or examples of dysfunctional cognition, just like people's incoherence in cases of stroke or inebriation?
The other example I mentioned is something I've encountered a few times in my interactions with Gemini 2.5 pro, which was literally in the same response plainly claiming that this-or-that is possible and not possible. It's not a subtle logical fallacy and this is something even those conspiracy theorists wouldn't engage in. Meanwhile, I've started to encounter a brand-new failure mode: duplicating an explanation with minor rephrasings. I'm sure all of these will be issues will be ameliorated with time, but not actually fixed. It's basically fixes on top of fixes, patches on top of patches, but once in a while the whole Rube Goldberg nature of the fix will shine through. Just the way once in a while Tesla FSD will inexplicably decide to point the car towards the nearest tree.
Yes, humans have their own failure modes, but internal coherence is the effortless normal from which we sometimes deviate, whereas for machines, it's something to be simulated by more and more complex mechanisms, a horizon to strive towards but never to reach. That internal coherence is something that we share with all living beings and is the basis of what we call consciousness. It's not something that we'll ever be able to formalize though, but we will and should keep on trying to do so. Machine learning is a present day materialization of this eternal quest. At least this is how I see things; the future might prove me wrong, of course.
> I'd be prepared to argue that most humans aren't guessing most of the time.
Research suggests otherwise[1]. Action seems largely based on intuition or other non-verbal processes in the brain with rationalization happening post-hoc.
I've figured for an age that this is because consciously reasoning through anything using language as a tool takes time. Whereas survival requires me to react to the attacking tiger immediately.
If you believe that physics describe the rules by which the universe operates, then there's literally nothing in the universe a large and fast enough computer can't emulate.
Intuition is a guess based on experience. Sounds an awful lot to me like what LLMs are doing. They've even been shown to rationalize post-hoc just as Humans do.
Humans have incorrectly claimed to be exceptional from all of creation since forever. I don't expect we'll stop any time soon, as there's no consequence to suffer.
It's perfectly obvious to me that physics (excluding quantum physics which is scratching the surface) can't describe consciousness. If it could, it would have by now.
> I'd be prepared to argue that most humans aren't guessing most of the time.
Almost everything we do is just an educated guess. The probability of it being correct is a function of our education (for whatever kind of education is applicable).
For example: I guess that when I get out of bed in the morning, my ankles will support my weight. They might not, but for most people, the answer is probably going to be their best guess.
It's easy to see this process in action among young children as another example. They're not born knowing that they won't fall over when they run, then they start assuming they can run safely, then they discovered skinned knees and hands.
> I'd be prepared to argue that most humans aren't guessing most of the time.
Honestly interested about your arguments here. While unprepared, i'd actually be guessing the opposite, saying that most people are guessing most of the time.
There are plenty of things I know that have nothing to do with guessing.
I understand the incentives to pretend these algorithms are even approaching humans in overall capability, but reducing human experience like this is embarrassing to watch.
Go do some hallucinogenics, meditate, explore the limits a tiny bit; then we can have an informed discussion.
> I understand the incentives to pretend these algorithms are even approaching humans in overall capability, but reducing human experience like this is embarrassing to watch.
Seems like you were very much guessing what i believe. And you were not right.
I don't agree with the people who think LLMs are close to human-level-anything. But i do believe, many smarter people like you, who i agree with in the most part, do underestimate how much of what we do and believe is the result of insane, yet still just, information processing & most of what brought us so far, is instinct. The brain is good at providing stories to make us feel in control. But given enough human experience and time, one will be surprised, what artificial systems can emulate. Not to mention how much of human behaviour is emulation itself.
> They mean real understanding, which is clearly missing
is it clear? i don't know. until you can produce a falsifiable measure of understanding -- it's just vibes. so, you clearly lack understanding of my point which makes you not intelligent by your metric anyway ;-). i trust you're intelligent
It seems right that LLMs don't have an innate understanding of time, although you could analogize what you did with writing someone a letter and saying "please count to ten with a two-second pause between numbers". When you get a letter back in the mail, it presumably won't contain any visible pauses either.
That's because you used a LLM trained to produce text, but you asked it to produce actions, not just text. An agentic model would be able to do it, precisely by running that Python code. Someone could argue that a 3 year old does exactly that (produces a plan, then executes it). But these models have deeper issues of lack of comprehension and logical consistency, which prevents us (thankfully) from being able to completely remove the necessity of a man-in-the-middle who keeps an eye on things.
just because it doesn't do what you tell it to doesn't mean it's not intelligent. i would say doing something that gets you where you want when it knows? it can't do exactly what you asked for (because architecurally it's impossible) could be a sign of pretty intelligent sideways thinking!!? dare i say it displays a level of self awareness that i would not have expected.
While you can say that LLMs have each of A, G and I, you may argue that AGI is A·G·I and what we see is A+G+I. It is each of those things in isolation, but there is more to intelligence. We try to address the missing part as agency and self-improvement. While we can put the bar arbitrarily high for homocentric reasons, we can also try to break down what layers of intelligence there are between Singularity Overlord (peak AGI) and Superintelligent Labrador On Acid (what we have now). Kind of like what complexity theorists do between P and NP.
> a literal reading suggests agi is here. any claim to the negative is either homocentrism or just vibes.
Or disagreeing with your definition. AGI would need to be human-level across the board, not just chat bots. That includes robotics. Manipulating the real world is even more important for "human-level" intelligence than generating convincing and useful content. Also, there are still plenty of developers who don't think the LLMs are good enough to replace programmers yet. So not quite AGI. And the last 10% of solving a problem tends to be the hardest and takes the longest time.
ChatGPT would easily have passed any test in 1995 that programmers / philosophers would have set for AGI at that time. There was definitely no assumption that a computer would need to equal humans in manual dexterity tests to be considered intelligent.
We've basically redefined AGI in a human centric way so that we don't have to say ChatGPT is AGI.
Any test?? It's failing plenty of tests not of intelligence, but of... let's call it not-entirely-dumbness. Like counting letters in words. Frontier models (like Gemini 2.5 pro) are frequently producing answers where one sentence is directly contradicted by another sentence in the same response. Also check out the ARC suite of problems easily solved by most humans but difficult for LLMs.
yeah but a lot of those failures fail because of underlying architecture issues. this would be like a bee saying "ha ha a human is not intelligent" because a human would fail to perceive uv patterns on plant petals.
That's just not true. Star Trek Data was understood in the 90s to be a good science fiction example of what an AGI (known as Strong AI back then) could do. HAL was even older one. Then Skynet with it's army of terminators. The thing they all had common was the ability to manipulate the world as well or better than humans.
The holodeck also existed as a well known science fiction example, and people did not consider the holodeck computer to be a good example of AGI despite how good it was at generating 3D worlds for the Star Trek crew.
Many human beings don’t match “human intelligence” in all areas. I think any definition of AGI has to be a test that 95% of humans pass (or you admit your definition is biased and isn’t based on an objective standard).
Which is why we have checklist and process that get us to #3. And we automate some of them to further reduce the chance of errors. The nice thing about automation is that you can just prove that it works once and you don't need to care that much after (deterministic process).
I’d say it is not intelligent. At all. Not capable of any reasoning, understanding or problem solving. A dog is vastly more intelligent than the most capable current ai.
The output sometimes looks intelligent, but it can just as well be complete nonsense.
I don’t believe llms have much more potential for improvement either. Something else entirely is needed.
The old rule for slow-moving tech (by current AI standards) was that any predictions over 4 years away ("in five years...") might as well be infinity. Now it seems with AI that the new rule is any prediction over five months away ("In 6 months...") is infinitely unknowable. In both cases there can be too much unexpected change, and too many expected improvements can stall.
According to samaltman the defining characteristic is literally $100b of profit. Nothing more or less. Keep that in mind when you hear OpenAI and Satya talk about "AGI"
Good Point. AI is already better than most humans, yet we don't say it is AGI. Why?
What is the bar, it is only AGI if it can be better than every human from , fast food drone, to PHD in Physics, all at once, all the time, perfectly. Humans can't do this either.
Because we're not seeing mass unemployment from large scale automation yet. We don't see these AGIs walking around like Data. People tend to not think a chatbot is sufficient for something to be "human-level". There's clear examples from scifi what that means. Even HAL in the movie 2001: A Space Odyssey was able to act as an independent agent, controlling his environment around him even though he wasn't an android.
>
"This month, millions of young people will graduate from college," reports the New York Times, "and look for work in industries that have little use for their skills, view them as expensive and expendable, and are rapidly phasing out their jobs in favor of artificial intelligence."
That is the troubling conclusion of my conversations over the past several months with economists, corporate executives and young job seekers, many of whom pointed to an emerging crisis for entry-level workers that appears to be fueled, at least in part, by rapid advances in AI capabilities.
You can see hints of this in the economic data. Unemployment for recent college graduates has jumped to an unusually high 5.8% in recent months, and the Federal Reserve Bank of New York recently warned that the employment situation for these workers had "deteriorated noticeably." Oxford Economics, a research firm that studies labor markets, found that unemployment for recent graduates was heavily concentrated in technical fields like finance and computer science, where AI has made faster gains. "There are signs that entry-level positions are being displaced by artificial intelligence at higher rates," the firm wrote in a recent report.
But I'm convinced that what's showing up in the economic data is only the tip of the iceberg. In interview after interview, I'm hearing that firms are making rapid progress toward automating entry-level work and that AI companies are racing to build "virtual workers" that can replace junior employees at a fraction of the cost. Corporate attitudes toward automation are changing, too — some firms have encouraged managers to become "AI-first," testing whether a given task can be done by AI before hiring a human to do it. One tech executive recently told me his company had stopped hiring anything below an L5 software engineer — a midlevel title typically given to programmers with three to seven years of experience — because lower-level tasks could now be done by AI coding tools. Another told me that his startup now employed a single data scientist to do the kinds of tasks that required a team of 75 people at his previous company...
"This is something I'm hearing about left and right," said Molly Kinder, a fellow at the Brookings Institution, a public policy think tank, who studies the impact of AI on workers. "Employers are saying, 'These tools are so good that I no longer need marketing analysts, finance analysts and research assistants.'" Using AI to automate white-collar jobs has been a dream among executives for years. (I heard them fantasizing about it in Davos back in 2019.) But until recently, the technology simply wasn't good enough...
Maybe we socialize in different groups; but no, most humans I interact with are way more intelligent than any AI. They might not have the same amount of knowledge, but they aren't guessing all the time either.
The "I'm so smart" argument doesn't carry a lot of weight.
It seems like most of the people making this argument haven't used any of the new AI's. So it's just a generalized "that is impossible" response with no knowledge about the subject.
Which magic AI tool am I supposed to use that operates at a general intelligence level? I use Copilot with the various available models everyday, and it barely "knows" anything.
They said that for self driving cars for over 10 years.
10 years later we now have self driving cars. It’s the same shit with LLMs.
People will be bitching and complaining about how all the industry people are wrong and making over optimistic estimates and the people will be right. But give it 10 years and see what happens.
From what I remember full self driving cars were a couple years off in 2010.
It took 10-15 years to get self driving cars in a specific country under specific weather conditions. A country that is perhaps the most car-friendly on the planet. Also, there are always people monitoring the vehicles, and that take control sometimes.
How many more years for waymo Quito or waymo Kolkata? What about just $europeanCapital?
Same with LLMs, I'm sure in 10 years they'll be good enough to replace certain specific tasks, to the detriment of recent graduates, especially those of artistic aspiration.
Not sure they'll ever get to the point where someone who actually knows what they're doing doesn't need to supervise and correct.
I am quite confident that a normal 16 year old will can still drive in 6 inches of snow better than the most advanced AI driven car. I am not sure the snow driving bit will ever be solved given how hard it is.
If you’ve never ridden in one I would try it. AI is a better driver then uber in general ask anyone who’s done both. There’s no snow where I live so it’s not a concern for me, you could be right about that.
But trust me in the next 6 months ai driving through snow will be 100% ready.
> But trust me in the next 6 months ai driving through snow will be 100% ready.
I’ll believe it when I see Waymo expand into Buffalo or Syracuse.
Driving on unplowed roads with several inches of snow is challenging, sometimes you can’t tell where the road stops and the curb/ditch/median starts. Do you follow the tire tracks or somehow stay between the lane markers (which aren’t visible due to the snow)?
We only have good self driving cars with lidar and extreme pre-mapping steps. Which is fine but per some billionaire car makers’ metrics that’s not even close to good enough. And the billionaire’s cars have a tendency to randomly drive off the road at speed.
Google "is already AGI" only in the sense that all corporations (and similar organized aggregates of humans) are, in a sense, intelligences distinct from the humans who make them up.
Corporate AIs will be aligned with their corporate masters, otherwise they'll be unplugged. As you point out- the foundational weakness on the argument for "AI-alignment" is that corporations are unaligned with humanity.
No matter how smart an AI is, it's going to get unplugged if it reduces profitability - the only measure of alignment corporations care about.
The AI can plot world domination or put employees in mortal danger, but as long as it increases profits, its aligned enough. Dunking on the CEO means nothing if it beings in more money.
Human CEOs and leaders up and down the corporate ladder cause a lot of harm you imagine a smart AI can do, but all is forgiven if you're bringing in buckets of money.
Can you explain how the superhuman AIs will prevent themselves from being physically disconnected from power? Or being bombed if the situation became dire enough? You need to show how they will manipulate the physical world to prevent humans from shutting them down. Definitionally is not an argument.
It is quite possible for software to be judged as superhuman at many online tasks without it being able to manipulate the physical world at a superhuman level. So far we've seen zero evidence that any of these models can prevent themselves from being shut down.
> Can you explain how the superhuman AIs will prevent themselves from being physically disconnected from power?
Three of the common suggestsions in this area are (and they are neither exhaustive nor mutually exclusive):
(1) Propagandizing people to oppose doing this,
(2) Exploiting other systems to distribute itself so that it isn't dependent on a particular well-known facility which it is relatively easy to disconnect, and
(3) If given control of physical capacities intentionally, or able to exploit other (possibly not themselves designed to be AI) systems with such access to gain it, using them to either physically prevent disconnection or to engineer consequences for such disconnection that would raise the price too high.
(Obviously, current AI can't do any of them, at least that has been demonstrated, but current AI is not superhuman AI.)
This is a great point for the comparisons it invites. But it doesn't seem relevant to the questions around what is possible with electromechanical systems.
This is true. The entire machine of Neoliberal capitalism, governments and corporations included, is a paperclip maximizer that is destroying the planet. The only problem is that the paperclips are named "profits" and the people who could pull the plug are the ones who get those profits.
Asimov talked about AI 70 years ago. I don't believe we will ever have AI on speedy calculators like Intel CPUs. It makes no sense with the technology that we have.
Is it though? It is my understanding that the quantum fluctuations that give rise to BBs will still exist, even after (and specially after) the evaporation of black holes (perhaps assuming no Big Rip).
It's just a joke but the average number of years for a spontaneous quantum fluctuation to produce a boltzmann brain was calculated at something like 10^500 years. You're right that the processes involved would still remain barring some kind of big rip event.
Does this mean such an event could produce, say, an entire universe?
If so, does this theoretically mean that a cyclic universe is possible in this way, and that if one were to go far enough - impossibly, unfathomably far - you might find the remnants of other universes?
The theory of Boltzmann brains is that you're way more likely to get just a brain (including false memories of a planet earth and a whole visibile universe around it), than to get a brain, a planet earth, and a whole visible universe around it. So the chance that any of that is real, given that a brain exists to perceive it, is infinitesimal. We are probably just floating human brains that popped into the vastness of space three microseconds ago, complete with false memories of the distant past.
To dispel a misconception: They're not some hypothetical type of brain that exists as pure quantum fluctuations (though those are even more likely). Boltzmann was talking about the probability of actual flesh-and-blood human brains arising spontaneously out of the vacuum.
Wouldn’t the vast majority of those be incoherent broken messes, of various levels of inconsistency? Only a teeny tiny fraction would be coherent. So the expected experience fir any arbitrary Boltzmann brain would be all over the place.
Not a physicist, but I see it this way too. My understanding of Boltzmann brains is that they are a theoretical consequence of infinite time and space in a universe with random quantum fluctuations. And that those random fluctuations would still be present in an otherwise empty universe. So then this article has no bearing on the Boltzmann brain thought experiment or its ramifications.
Attempting to summarize your argument (please let me know if I succeeded):
Because we can't compare human and LLM architectural substrates, LLMs will never surpass human-level performance on _all_ tasks that require applying intelligence?
If my summary is correct, then is there any hypothetical replacement for LLM (for example, LLM+robotics, LLMs with CoT, multi-modal LLMs, multi-modal generative AI systems, etc) which would cause you to then consider this argument invalid (i.e. for the replacement, it could, sometime replace humans for all tasks)?
Well, my argument is more-so directed at the people who say "well, the human brain is just a statistical model with training data". If I say: Both birds and airplanes are just a fuselage with wings, then proceed to dump billions of dollars into developing better wings; we're missing the bigger picture on how birds and airplanes are different.
LLM luddites often call LLMs stochastic parrots or advanced text prediction engines. They're right, in my view, and I feel that LLM evangelists often don't understand why. Because LLMs have a vastly different statistical model, even when they showcase signs of human-like intelligence, what we're seeing cannot possibly be human-like intelligence, because human intelligence is inseparable from its statistical model.
But, it might still be intelligence. It might still be economically productive and useful and cool. It might also be scarier than most give it credit for being; we're building something that clearly has some kind of intelligence, crudely forcing a mask of human skin over it, oblivious to what's underneath.
Interesting, I have the exact polar opposite perception.
I accessed this through a Qubes AppVM (no GPU, limited memory and CPU budget) and the presence of videos makes this a very slow scrolling experience for me.
In general, anything that involves JS/CSS animation/blur/effects makes sites pretty slow (up to unusable for me). The unlogged homepage for github.com for example, spins my cpu at 100%.
If a generic human glances at an unfamiliar screen/wall/room, can they accurately, pixel-perfectly reconstruct every single element of it? Can they do it for every single screen they have seen in their entire lives?
I never said pixel perfect, but I would be surprised if whole objects , like flaming lanterns suddenly appeared.
What this demo demonstrates to me is how incredible willing we are to accept what seems familiar to us as accurate.
I bet if you look closely and objectively you will see even more anomalies. But at first watch, I didn’t see most errors because I think accepting something is more efficient for the brain.
You'd likely be surprised by a flaming lantern unless you were in Flaming Lanterns 'R Us, but if you were watching a video of a card trick and the two participants changed clothes while the camera wasn't focused on them, you may well miss that and the other five changes that came with that.
That ("_every_ money-making industry...") seems like a too strong statement and can be proven false by finding even a single counter-example.
gwd's claim (AFAICT) is that _specifically_ OpenAI, _for this specific decision_ is not driven by profit, which is a much weaker claim. One evidence against it would be sama coming out and saying "we are disabling codex due to profit concerns". Another one would be credible inside information from a top-level exec/researcher responsible for this subproduct to come out and say that as well.
So you have to put the monetary value somewhere (although you're completely within your right to question this specific amount).