> Anticipating the arrival of this all-powerful technology, Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack.
From the Atlantic article. He seems a serious person but it's hard to take this circle seriously.
tbh it smells like they've brought in too much to their own hype.
Nothing OpenAI have done more recently suggest to me that they are closer to the AGI. They did create a bunch of other models with different input format that are probably based off pretty much the same thing. And it looks like those other models will be much more expensive to run and burn up their runway sooner.
Trouble is, everyone has a different interpretation of those words.
Do you mean they're no closer to "highly autonomous systems that outperform humans at most economically valuable work", or do you mean no closer to "a general-purpose machine-learning based system, even if it's only IQ 85", or do you mean no closer to "a conscious and self-aware artificial mind", or do you mean "a super-intelligent system that beats all humans put together"?
> And it looks like those other models will be much more expensive to run and burn up their runway sooner
I thought "runway" was the metaphor for "before take-off"? They've definitely taken off now. If they were burning more money than they bring in, they could've just not lowered the prices.
Agreed. Honestly gpt4o is a massive disappointment. They made it multimodal, but it seems as dumb as ever reasoning wise. I think it's become pretty clear that we've reached diminishing returns with LLMs and AGI isn't coming. Some of OpenAI have realized this and have moved tool full on productization. That said, I do believe some of the people there have genuinely drunk the kool aid.
And it would make sense that the compliance and safety side are the most hyped because only that would justify their continued involvement in the project.
This won't make a dent in the logical armor of AI optimists:
[ ] If you are not intimately familiar with the development of AI, your warnings on safety can be disregarded due to your basic ignorance about the development of AI
[x] If you are intimately familiar with the development of AI, your warnings on safety can be disregarded due to potential conflicts of interest and koolaid drinking
I think they aren't the full answer, no matter how much they're scaled up. But they may be one essential element of a working solution, and perhaps one or two brilliant insights away. I also think that some of the money being invested into the LLM craze will be directed into the search for those other brilliant insights.
Many teams are trying to combine their ideas with LLM. Because despite their weaknesses, it seems LLMs (and related concepts such as RLFH, transformers, self-supervised learning, internet-scale datasets), have made some remarkable gains. Those team are coming from the whole spectrum of ML and AI research. And they wish to use their ideas to overcome some of the weaknesses of current day LLMs. Do you also think that none of these children can lead to AGI? Why not?
LLMS don't have to be smart enough to be AGI. They just have to be smart enough to create AGI. And if creating something smarter than yourself sounds crazy, remember that we were created by simpler ancestors that we now effortlessly dominate.
I don't disagree with the general notion, but it seem to me that LLMs being smart enough to create AGI is even more far fetched than if they are just smart enough to be AGI.
All I’d like to see from AI safety folks is an empirical argument demonstrating that we’re remotely close to AGI, and that AGI is dangerous.
Sorry, but sci-fi novels are not going to cut it here. If anything, the last year and a half have just supported the notion that we’re not close to AGI.
The flipside: it's equally hard for people who assume AI is safe to establish empirical criteria for safety and behavior. Neither side of the argument has a strong empirical basis, because we know of no precedent for an event like the rise of non-biological intelligence.
If AGI happens, even in retrospect, there may not be a clear line between "here is non-AGI" and "here is AGI". As far as we know, there wasn't a dividing line like this during the evolution of human intelligence.
I find it delightfully ironic that humans are so bad at the things we criticise AI for not being able to do, such as extrapolating to outside our experience.
As a society, we don't even agree on the meanings of each of the initials of "AGI", and many of us use the triplet to mean something (super-intelligence) that isn't even one of those initials; for your claim to be true, AGI has to be a higher standard than "intern of all trades, senior of none" because that's what the LLMs do.
Expert-at-everything-level AGI is dangerous because the definition of the term is that it can necessarily do anything that a human can do[0], and that includes triggering a world war by assassinating an archduke, inventing the atom bomb, and at least four examples (Ireland, India, USSR, Cambodia) of killing several million people by mis-managing a country that they came to rule by political machinations that are just another skill.
When it comes to AI alignment, last I checked we don't know what we even mean by the concept: if you have two AI, there isn't even a metric you can use to say if one is more aligned than the other.
If I gave a medieval monk two lumps of U-238 and two more of U-235, they would not have the means to determine which pair was safe to bash together and which would kill them in a blue flash. That's where we're at with AI right now. And like the monks in this metaphor, we also don't have the faintest idea if the "rocks" we're "bashing together" are "uranium", nor what a "critical mass" is.
Sadly this ignorance isn't a shield, as evolution made us without any intentionality behind it. So we don't know how to recognise "unsafe" when we do it, we don't know if we might do it by accident, we don't know how to do it on purpose in order to say "don't do that", and because of this we may be doing cargo-cult "intelligence" and/or "safety" at any given moment and at any given scale, making us fractally-wrong[1] about basically every aspect including which ones we should even care about.
[0] If you think it needs a body, I'd point out we've already got plenty of robot bodies for it to control, the software for these is the hard bit
"sailing against the wind" is a very apt description of hardlining Yuddite philosophy when your company's models got maybe 20% better in the past two years (original GPT4 is still the best model I've dealt with to this day), while local models got 1000% better.
we should all thank G-d these people weren't around during the advent of personal computing and the internet - we'd have word filters in our fucking text processors and publishing something on the internet would require written permission from your local DEI commissar.
arrogance, pure fucking hubris brought about by the incomprehensibly stupid assumption that they will get to be the stewards of this technology.
we should all thank G-d similar people have been generally in charge of bioengineering and nuclear development labs else we'd have... err nothing. We'd have pretty much nothing at all.
Well nukes continue to be very difficult to download off the Internet or to rent access to via a web browser.
I haven't seen anyone advocate that the US government needs to be the only entity able to use AI, but if you have, you knocked their socks off with that one.
>Well nukes continue to be very difficult to download off the Internet or to rent access to via a web browser.
well, the amount of damage you can inflict with an AI model is a few magnitudes lower than that of a nuclear weapon.
>I haven't seen anyone advocate that the US government needs to be the only entity able to use AI, but if you have, you knocked their socks off with that one.
did you miss Sam Altman's world tour last year, talking to the politicians about the ooooh so scary AI being developed by their unregulated competitors? the establishment of oversight committees and regulatory bodies in the US, from where corporate sockpuppets will presume to dictate the the terms of development and application of this technology?
is that you think what Jan Leike was working on or "Yuddite" philosophy is in anyway supportive of DEI. These things aren't related, and you're not anywhere close to the real problem by screeching about DEI.
>When designing the red teaming process for DALL·E 3, we considered a wide range of risks3 such
as:
>1. Biological, chemical, and weapon related risks
>2. Mis/disinformation risks
>3. Racy and unsolicited racy imagery
>4. Societal risks related to bias and representation
(4) is DEI bullshit verbatim, (3) is DEI bullshit de facto - we all know which side of the kulturkampf screeches about "racy" things (like images of conventionally attractive women in bikinis) in the current year.
I don't know which exact role did that exact individual play at trust/safety/ethics/fart-fart-blah-fart department over at openai, but it is painfully, very painfully obvious what are openai/microsoft/google/meta/anthropic/stability/etc afraid their models might do. in every fucking press release, they all bend over backwards to appease the kvetchers, who are ever ready, eager and willing to post scalding hot takes all over X (formerly known as twitter).
Again, the superalignment team that Jan Leike and Ilya was working on, along with Yudkowsky's opinions, are unrelated to any DEI and "racy"-ness.
You can read the Superalignment announcement and what it focuses on. The entire thing is about AGI x-risk, with a small paragraph about how there's other people's work about whatever bias and PC-ness.
These are different concerns by different people. You and many others are pattern matching AGI x-risk to the AI bias people to your detriment and it's poisoning the discourse. Listen to Emmett Shear (former OpenAI/Twitch CEOs) explain this in depth: https://www.youtube.com/watch?v=jZ2xw_1_KHY&t=800s
my point is that all evidence I see so far strongly suggests that Yudish scifi skynet bullshit all the big AI companies pay lip service to serves no purpose but scaring senile boomers and clueless pedestrians into regulating their competitors, while in reality, the 'safety' they actually work on and dedicate resources to is focused on minimizing the likelihood of some corporate chatbot accidentally recalling a gamer word from the deep recesses of its stochastic parrot brain and invoking the self-righteous wrath of twitter grifters and journos.
yes, I have no doubt that some researchers, influenced by juvenile fantasies omnipresent in all media from the past half century, might actually genuinely belong to the safety cult. I just refuse to believe that people whose opinions and decisions actually matter are influenced by such fears, because unlike those few genuine cultists, the people in charge aren't fucking morons who think that glorified autocomplete pseudo-AI tools can escape into the matrix and start sending terminators into the past to destroy our democracy.
believing in selflessness or social responsibility of corporations and politicians is incomprehensibly naive (to put it as safely and ethically as I possibly can)
> I just refuse to believe that people whose opinions and decisions actually matter are influenced by such fears
Well, at least I'm glad you admit it's due to your stubbornness and unwillingness to change beliefs when confronted with evidence.
Sam Altman ("Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity"), Ilya Sutskever, Geoffrey Hinton, Yoshua Bengio, Jan Leike, Paul Christiano (creator of RLHF), Dario Amodei (Anthropic), Demis Hassabis (Google DeepMind) all believe AGI poses an existential risk to humanity.
is it not ironic to list Sam Altman ("Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity") among those paragons of virtue, half a year after the internal struggle between safetyists and pragmatists at openai has been revealed? is it not naive to assume that the other virtuous men you list are not the minority, given that the vast majority of openai had sided with Sam, revealing themselves as pragmatists who valued their lavish salaries over mitigating the supposed x-risks? is it not especially ironic to present Sam as a paragon of virtue here, in the context of a safety cultist leaving openai because he realized what you do not - it's all bullshit, mirror and smoke to misdirect the press and convince the politicians to smother the smaller competitors (not backed with billions of VC and unable to lobby for concessions) with regoolations.
>But over the past few years, safety culture and processes have taken a backseat to shiny products.
you know what else happened over the past few years? openai started to make money. so while sama was making soundbites for headlines about the existential threat of AI, internally, all the useful idiots were already told to shut the fuck up.
Thank you Jan for your work and your courage to act and speak candidly about these important topics.
Open question to HN: to your knowledge/experience which AGI-building companies or projects have a culture most closely aligned with keeping safety, security, privacy, etc. as high a priority as “winning the race” on this new frontier land-grab? I’d love to find and support those teams over the teams that spend more time focused on getting investment and marketshare.
From this interview https://youtu.be/Nlkk3glap_U?feature=shared it seemed Anthropic was focusing more on those topics rather than winning the race. He hinted at being “forced” by the competition to release their models.
I can't find the reference right now but I seem to remember hearing Zvi Moskowitz say in a podcast that Anthropic is the most safety-concious of the big players.
Ironic. The product I’m building relies on Haiku. It’s brilliant, cheap and fast. Their models are very capable and much more accessible to small companies.
Claude Opus doesn't appear to have more safety limitations than ChatGPT.
The older Claude 2.1 on the other hand was so ridiculously incapable of functioning due to a safety first design I'm guessing it inspired the goodie 2 parody AI. https://www.goody2.ai/
> “Over the past few months […] we were struggling for compute
OpenAI literally said they were setting aside 20% of compute to ensure alignment [1] but if you read the fine print, what they said was that they are “dedicating 20% of the compute we’ve secured ‘to date’ to this effort” (emphasis mine). So if their overall compute has increased by 10x then that 20% is suddenly 2%, right? Is OpenAI going to be responsible or is it just a mad race (modelled from the top) to “win” the AI game?
Anyone or anything at all is probably the first steps, which we still don't know how to do - even conceptually.
But the first real target would be alignment with the creator of the AI, and the goals an values they set.
That is: Alignment with humanity/society as a whole is even further away. And might even be considered out-of-scope for AI research: ensuring that the AI creator (persons/organization) is aligned with society might be considered to be in the political domain.
Seems very arrogant at the end to say humanity is waiting on you. It’s not like Covid—19 where the population was literally looking to scientist/gov for an active solution. AGI is something a select set of computer nerds want, but the public isn’t clamoring for. In fact the entire idea of safety/alignment/cultural shift shows the potential downside on society if it actually were achieved.
Meanwhile deepmind with alpha fold etc is showing AI can help with pressing problems of humanity without AGI as the necessary first step.
> Seems very arrogant at the end to say humanity is waiting on you. It’s not like Covid—19 where the population was literally looking to scientist/gov for an active solution. AGI is something a select set of computer nerds want, but the public isn’t clamoring for. In fact the entire idea of safety/alignment/cultural shift shows the potential downside on society if it actually were achieved.
It's a similar kind of waiting, with varying levels of optimism. When you say "the public isn't clamoring for" I think a lot of the clamoring over Covid—19 was a nonspecific desire to get some normalcy. With AI it's the same, with people worrying about their future careers, or those of loved ones. In either case, people want answers, whether they are enthusiastic about what the companies are doing or not.
> Meanwhile deepmind with alpha fold etc is showing AI can help with pressing problems of humanity without AGI as the necessary first step.
Any of the above. Maybe, if you really want to put some unneeded extra rigor on the analysis, add "within the lifetime of anyone born in 2024".
I don't want to, like, have the argument here about it. Nobody will persuade anybody of anything. But it is not a truth universally acknowledged that AGI superintelligence is actually a thing.
There are other reasons to have qualms about OpenAI! They could be misleading the market about their capabilities. They could be abusing IP. They could be ignoring clear and obvious misuse cases, the same way Facebook/Meta did in South Asia.
But this exit statement seems to be much more about AGI superintelligence than any of that stuff. OK, so if I don't think that's a thing, I don't have to pay attention, right?
Why would you have to pay attention regardless? Is an OpenAI researcher's departure a subject that interests you, or is it not? It sounds like it's not.
There's a lot of things posted on HN that I dont need to and end up not paying attention to. I respect you, but these comments of how you dont need to pay attention to this topic are at best unproductive.
Seems to me the management isn’t quite as high on their own supply (at least privately) and they do not believe anything even remotely resembling “AGI” is possible in the foreseeable future. Which I, and many other people in this field, do agree with.
It really baffles me that even extremely knowledgeable insiders like this end up buying the AGI hype. LLMs are extremely novel and have lots of groundbreaking applications, but this idea that we are headed directly into fantasy sci-fi A.I. is completely bonkers.
It's one thing to push this kind of hype to get people talking about A.I. if you're trying to capitalize on this space. It's something else entirely to swallow your own marketing BS as if it were gospel.
But let's face it, this guy probably isn't serious, he's just spewing more hype upon departing OpenAI looking for the next tech company to hire him.
I was and am very skeptical of the "superalignment" plan (which was "build an AI that does alignment research for us, and then ask it what to do"). But it's a bad look for OpenAI if they pledged 20% of their compute for this effort and then just didn't actually give them the compute.
If it economically better to hire AI then humans for most jobs, and they only need Nvidia hardare and electricity then today's Nvidia are tiny, real tiny.
so you think if they came out and said we have created AGI there would be no changes in stock prices to any stock that has anything to do with AI even if they are already "huge"?
Probably within 5 years. Compute growing exponentially, algorithm improvement as well, multi modalities allows for different type of data training, etc…
Yeah, this shit is near. Also- Quite a dangerous experiment we’re running. And safety-first people are not at the helm anymore.
I'm not convinced, you can throw all the compute (btw, it's not growing exponentially any more, we have arrived at atom scale) at it and I'm not convinced this will lead to AGI.
Our rudimentary, underpowered brain is GI, now you're telling me stacking more GPU bricks will lead to AGI? If it indeed does, it would have came by now.
Our brains have far more discrete compute elements in them than even a chip, even though our wetware elements just run much slower than our silicon it doesn't tell us much about which is the more "powerful" — the overall compute throughput of the brain is unclear, with estimates varying by many orders of magnitude.
I also don't expect LLMs to be the final word on AI architecture, as they need so many more examples compared to organic brains to get anything done. But the hardware… that might just be a red herring.
I don't believe we can create something more intelligent than us. Smarter, yes, but not more intelligent. We will always have a deeper and more nuanced understanding of reality than our artificial counterparts. But it does not mean that AI can not be a danger.
> We will always have a deeper and more nuanced understanding of reality than our artificial counterparts.
Wishful thinking already disproven by collectives (Google and OpenAI) that are already better at understanding and acting upon reality than you or any single human intelligence. These are systems, and they can soon hollow out their biological components.
Do you not consider such collectives part of we (the humans)? I can understand arguing that organizations have some superinteligence. But they are still made of / run on humans - not AI/computers?
They’re currently made up of humans, but more and more will be made of computers. In their human forms, they’re still a subversion of human values, doing things as a whole, because of incentives, that the individual humans wouldn’t do. But in the future, entire corporations will get whittled down to machine processes. An AGI decision maker at the top and each individual task replaced with robot or software agent.
I agree with you. Those models are smarter than us in specific areas. But my question is, can they understand ideology, religion, etc, and feel them like we do? Can they come up with their traditions and beliefs? Because these are the core and evolving foundations of our reality and one of the most primitive activities since the dawn of humanity, I do not see any of these models even getting close to these concepts. They parrot what we told them. They have no agency of their own.
It is also a sign of hubris to think that we can create an artificial construct that is more intelligent than we are.
shades of
> Anticipating the arrival of this all-powerful technology, Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack.
From the Atlantic article. He seems a serious person but it's hard to take this circle seriously.