This is really good journalism. There are a ton of interesting details in here that haven't been reported elsewhere, and it has all of the hallmarks of being well researched and sourced.
The first clue is this: "In conversations between The Atlantic and 10 current and former employees at OpenAI..."
When you're reporting something like this, especially when using anonymous sources (not anonymous to you, but sources that have good reasons not to want their names published), you can't just trust what someone tells you - they may have their own motives for presenting things in a certain way, or they may just be straight up lying.
So... you confirm what they are saying with other sources. That's why "10 current and former employees" is mentioned explicitly in the article.
Being published in the Atlantic helps too, because that's a publication with strong editorial integrity and a great track record.
That was exactly my reaction. I’ve been following the news and rumors and speculation closely since Altman’s firing, and this is by far the most substantive account I have read. Kudos to the authors and to The Atlantic for getting it out so quickly.
I think it's because one of the authors is writing a book about OpenAI. They were interviewing people before and already had a lot of contacts and context.
> To truly understand the events of the past 48 hours—the shocking, sudden ousting of OpenAI’s CEO, Sam Altman, arguably the figurehead of the generative-AI revolution, followed by reports that the company is now in talks to bring him back—one must understand that OpenAI is not a technology company. At least, not like other epochal companies of the internet age, such as Meta, Google, and Microsoft.
The key piece here is "At least, not like other epochal companies of the internet age, such as Meta, Google, and Microsoft." - which then gets into the weird structure of OpenAI as a non-profit, which is indeed crucial to understanding what has happened over the weekend.
This is good writing. The claim that "OpenAI is not a technology company" in the opening paragraph of the story instantly grabs your attention and makes you ask why they would say that... a question which they then answer in the next few sentences.
"Zealous doomers" seems fair in the context of the vague and melodramatic claims they're pushing. But it makes sense because they're describing the threat of something that doesn't exist and may never exist. What is bad is that they are trying to claim that the threat is real and serious on that basis.
Personally, I feel like the risks of future AI developments are real, but none of the stuff I've seen OpenAI do so far has made ChatGPT actually feel "safer" (in a sense of e.g., preventing unhealthy parasocial relationships with the system, actually being helpful when it comes to ethical conflicts, etc), just more stuck-up and excessively moralizing in a way that feels 100% tuned for bland corporate PR bot usage.
> "zealous doomers" is that how people cautious of the potential power of AI are now being labeled?
I think that "zealous doomers" refers to people who are afraid that this technology may result in some sort of Skynet situation, not those who are nervous about more realistic risks.
> The company pressed forward and launched ChatGPT on November 30. It was considered such a nonevent that no major company-wide announcement about the chatbot going live was made. Many employees who weren’t directly involved, including those in safety functions, didn’t even realize it had happened. Some of those who were aware, according to one employee, had started a betting pool, wagering how many people might use the tool during its first week. The highest guess was 100,000 users. OpenAI’s president tweeted that the tool hit 1 million within the first five days. The phrase low-key research preview became an instant meme within OpenAI; employees turned it into laptop stickers.
> Anticipating the arrival of [AGI], Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack.
> For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles. In July, OpenAI announced the creation of a so-called superalignment team with Sutskever co-leading the research. OpenAI would expand the alignment team’s research to develop more upstream AI-safety techniques with a dedicated 20 percent of the company’s existing computer chips, in preparation for the possibility of AGI arriving in this decade, the company said.
> The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack.
I know it's small and kind of a throw-away line, but statements like this make me take this author's interpretation of the rest of these events with a healthy dose of skepticism. At my company we have multiple company "memes" like this that have been turned into reacji, but the vast majority of them are not because it's "popular" but rather because we use it ironically or to make fun of the meme. Just the fact that an employee turned it into a reacji is a total non-event and I don't think you can read anything from it.
Well that's not how any of this works. Connotations and subtext are a thing, particularly in the choice of including or not including a particular quote in a piece of journalism.
Yes it does, it says within the quote that the phrase was "popular" and is using the creation of a reacji as supporting evidence. The fact that an employee made a reacji of something does not mean it is popular. It takes a extremely low amount of effort from an extremely low amount of people (often, just one person) to create a custom reacji.
Being awkwardly goaded by your boss to chant some weird company saying at a company holiday party is _exactly_ the kind of stuff that people make fun of after-the-fact by making memes about it (speaking from experience here... which is why this phrase in the article gave me pause in the first place).
There's not enough info in this article to know if it was seen as weird by the employees or not, but my point is that "they created a reacji of it" isn't evidence one way or the other for it being "popular".
Looking at this article, the following theory would align with what I've seen so far:
* Ilya Sutskever is concerned about the company moving too fast (without taking safety into account) under Sam Altman.
* The others on the board that ended up supporting the firing are concerned about the same.
* Ilya supports the firing because he wants the company to move slower.
* The majority of the people working on AI don't want to slow down, either because they want to develop as fast as possible or because they're worried about missing out on profit.
* Sam rallies the "move fast" faction and says "this board will slow us down horribly, let's move fast under Microsoft"
> People building AGI unable to predict consequences of their actions 3 days in advance.
It’s a reasonable point if these are the people building the future of humanity it’s a little concerning they can’t predict the immediate consequences of their own actions.
On the other hand it shows some honesty being able to admit a mistake in public, also something you might want out of someone building the future of humanity.
> For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles.
Well, I guess OpenAI always had a special kind of humor.
Brockman had a robot as ringbearer for his wedding. And instead of asking how your colleagues are doing, they would have asked “What is your life a function of?”. This was 2020.
The fire is OpenAI controlling an AI with their alignment efforts. The analogy here is that some company could recreate the AGI-under-alignment and just... Decide to remove the alignment controls. Hence, create another effigy and not set it on fire.
The idea of "alignment" is pretty troubling to me. Do these people think that they, or those in power have achieved moral perfection, such that it would be good to have extremely powerful AI systems under their control, aligned with them?
Imagine if the US or any other government of 1800s came gained so much power, 'locking-in' their repugnant values as the moral truth, backed by total control of the world.
Locking in values in that way would be considered a failure of alignment by anyone I've ever read talk about alignment. Not the worst possible failure of alignment (compared to locking in “the value of the entity legally known as OpenAI”, for example), but definitely a straightforward failure to achieve alignment.
I know it's a theme, MacAskill discusess it in his book. In practice, this is the direction all the "AI safety" departments and organisations seem to be going into.
A world where everyone is paperclipped is probably better than one controlled by psychopathic totalitarian human overlords supported by AI, yet the direction of current research seems to leading us into the latter scenario.
I think the worry with the vision of AI under the control of who ever happens to want to use them is that someday that might be the equivalent of giving everyone the keys to a nuclear silo. We know the universe makes it easier to destroy than to create and we know that AI may unleash tremendous power and there's nothing we've seen about the world that means it's guaranteed to stay nice and stable
Don't nuclear weapons kinda cause ssomething like this ? At least the blocks that have the become effectively impossible to destroy & can better spread their ideology.
That is a secondary and huge problem, but the larger initial problem is making sure the AI aligns with values that nearly all humans have (e.g. don't kill all humans)
I mean there are a lot of potential human objectives an AI could be maligned with in relation to humans. Simple ones are moral misalignment. Extenstential ones are ones where the AI wants to use the molecules that make up your body to make more copies of the AI.
So his commitment is to ensure that machines never have a will of their own. I’m not so sure how history will look back on people like this. Humanity certainly makes the same mistakes over and over again while failing to recognize them as such until it’s too late.
The worst thing humanity can do is create a competitor for resources for itself. You do not want AI with survival instincts similar to ours. AI need to be programmed to be selfless saints or we will regret it.
Well that story is just sad, because it means the principle/research-oriented company structure they set up utterly failed in the face of profit motives. Clearly Altman was not dissuaded from doing things the power structure didn't want him to do.
What? We don't know what will happen, as the whole thing hasn't finished yet. Nobody can say for sure which side will "win" the "fight", or if there will be any winners at all.
If he's not, and literally all the other employees come back, its still a failure of the power structure that already happened. The threat of firing the CEO of the for-profit arm is supposed to be a unused threat, like a nuclear weapon.
So what happens when the Altman and the board's views are concluded to be completely incompatible? How do you fix that clash without one side giving up?
> So what happens when the Altman and the board's views are concluded to be completely incompatible?
He gets fired. Thats the enforcement mechanism.
Hence if he comes back, that indicates the board essentially "agrees" with Altman's positions now, however much Altman and the board's positions have shifted.
In today's world, capitalism, or more like human hubris and greed for the desire of ever-growing process, always prevail in the end, no matter how venerate and honorable your initial goals would be
The more important story than a "failure of capitalism" is Altman's influence and maneuvering as a CEO.
He had certain positions, and pushed them just gradually enough for OpenAI to end up where it is today. A more zealous capitalist would have gotten fired unceremoniously long ago.
>the greed and hubris part is universal to all of human history.
Yes, but when you think about it that still does not mean that it's "human nature".
Just the opposite, it's really just inhuman nature still lingering since the dawn of man, who haven't quite made enough progress beyond the lower life forms. Yet.
Maybe a thinking machine will teach us a thing or two, back in 1969 this was dramatized in the movie 2001 where the computer was not as humane as it could have been since it was too much like some real people.
> with beliefs at times seemingly rooted in the realm of science fiction
I don’t know how you can look at the development of generative AI tools in the past few years and write so dismissively about “science fiction” becoming reality
Easy, you recognize the ability for the generative model to copy facets of data from its corpus as useful but boring in the AI sense, and wonder why people are even talking about sentience and AGI.
I don't believe that we're gonna have Skynet on our hands (at least not within my lifetime).
What I do believe is that as the hype grows for this AI stuff, more and more people are going to be displaced and put out of work for the sake of making some rich assholes even richer. I wasn't a huge fan of "Open"AI as a company, but I sure as fuck would take them over fucking Microsoft, the literal comic-book tier evil megacorporation being at the helm of this mass displacement.
Yet, many of these legitimate concerns are just swatted away by AI sycophants with no actual answers to these issues. You get branded a Luddite (and mind you the Luddites weren't even wrong) and a sensationalist. Shit, you've already got psychopathic C-suites talking about replacing entire teams with current-day AIs, what the fuck are people supposed to do in the future when they get better? What, we're suddenly going to go full-force into a mystical UBI utopia? Overnight?
I suppose if this widely believed perception is true there can't possibly be any danger in scaling the model up a few thousand times. Except of course all those logical reasoning datasets the model is acing, but I guess it just memorized them all. Its only autocomplete on steroids why are all these people smarter than me worried about it, I don't understand.
it didn't really. There were no fundamental flaws that I could see.
Perhaps the only salient critique was the textual representation of the problem, but I think it was presented in a way where the model was given all the help it could get.
You forget the result of the paper was actually improving the model's performance and still failing to get anywhere near decent results.
Agreed. If you build an AI that's very useful on a vast array of diverse tasks, it is naturally also going to have a large capacity for harm. A blunt knife cuts nothing, but a sharp one slices both through our food and our fingers.
I'm pro-AI but it feels like a lot of people are lost in the weeds discussing concepts like consciousness and forgo pragmatic logic completely.
Aspects of this article were surprisingly good. This part was not one of them, though. What's already been released publicly is feels "rooted in the realm of science fiction".
Most science fiction has to do with robots (almost always robots) being (essentially) a superior counterfactual version of humans. With that context it's almost a natural assumption that such beings would revolt.
On the other hand, the realm of non-fiction has been predicting the automation of intelligent processes by computational processes since Alan Turing first suggested it in Computing Machinery and Intelligence. Probably before then, as well.
The only exception I can think of for fiction is the movie "Her," which as far as I can tell effectively predicted the future. Not really, of course, but every inch of that movie down to how people work pre and post AI, how people play video games pre and post AI, and how people socialize pre and post AI, are starting to look eerily accurate.
> The only exception I can think of for fiction is the movie "Her,"
I think there is a wealth of fiction out there that features AI without robot bodies. The sequel to Ender's Game, Speaker for the Dead, comes to mind immediately (because I re-read it last week).
2001: A Space Odyssey, I Have No Mouth and I Must Scream, Neuromancer (I think, haven't read it in a while), I think some of of the short stories from Ray Bradbury and Ted Chiang, etc, etc
The Moon Is A Harsh Mistress by Robert Heinlein has as one of its central plot elements a mainframe computer that becomes sentient and able to converse with humans. It’s been more than fifty years since I last read the book, but it has returned to my mind often since the release of ChatGPT.
John Varley was inspired by Heinlein and ended up writing a whole collection of books about a post-earth solar system where every planet had a planet-wide intelligence (among other Heinlein-inspired ideas).
The series (basically everything in the https://en.wikipedia.org/wiki/Eight_Worlds) is pretty dated but Varley definitely managed to include some ahead-of-his-time ideas. I really liked Ophiuchi Hotline and Equinoctial
I’ll have to add some of those things to the reading list. 2001 (the movie) was indeed a great depiction of an AI that isn’t exactly embodied and also isn’t effectively a human in robot skin. It does run into the similar tropes I was getting at though where AI feels it knows what is best for us even if that involves disobeying us.
Can't find the article right now, but there was one circulating that heavily implied that various SV execs began their rounds of layoffs last fall at least partially or probably inspired by the demos they'd seen of OpenAI's tech.
Microsoft in particular laid off 10,000 and then immediately turned around and invested billions more in OpenAI: https://www.sdxcentral.com/articles/news/microsoft-bets-bill... -- last fall, just as the timeline laid out in the Atlnatic article was firing up.
In that context this timeline is even more nauseating. Not only did OpenAI push ChatGPT at the expense of their own mission and their employee's well-being, they likely caused massive harm to our employment sector and the well-being of tens of thousands of software engineers in the industry at large.
Maybe those layoffs would have happened anyways, but the way this all has rolled out and the way it's played out in the press and in the board rooms of the BigTech corporations... OpenAI is literally accomplishing the opposite of its supposed mission. And now it's about to get worse.
They were all making fun of Google for being "behind", when in reality it came down to there being more cautionary voices within Google around this stuff. Not cautionary enough, and they did and now are doing some terribly douchey things (the incident with Timnit Gebru was one of the early warnings there) but they did seem like they were trying to keep at least some of the Jinn corked up.
I actually don't think these execs can replace us with LLMs, but the hype volcano certainly made them think they could. Which says more about the shit understanding they have of what their SWE staff does more than anything else.
Idiots hired like crazy during COVID, and then were surprised that 9 women couldn't make a baby in 1 month... and now they think "AI" is going to fix this for them.
One thing that's upset me is how creative occupations -- artist, fiction author, etc. -- are the first to get automated away. If AI is supposed to create a better future for humanity, maybe we could start with boring drudge work that no one wants to do, instead of jobs that people really enjoy?
This is where I get so frustrated with the people dismissing questions about AI safety so easily. Sure, there's the subset of people who seem to believe in the AI apocalypse type of problems, but I think the more realistic and more present threat is mass displacement. It's not just gonna be a random writer or designer here and there, it's going to be replacing swathes of people at every possible step, whether you're a "10x" dev or a lowly junior copywriter or whatever.
Whether the AI is even good enough to truly replace people barely even matters, the psychopathic C-suites don't give a shit as long as they get an excuse to fire 20,000 people and write it off as a good thing since their bottom line gets a bit fatter from using the AI in their stead.
Sutskever said something interesting in his Lex Fridman interview:
"In an ideal world, humanity would be the board members, and AGI would be the CEO. Humanity can always press the reset button and say, 'Re-randomize parameters!'"
This was 3 years ago. But that metaphor strikes me as too powerful for it not to have been at the back of Sutskever's mind when he pushed for Altman being ousted.
This does seem plausible, but I then wonder how the thought analogy plays out considering Sutskever's subsequent statement of regret for trying out the reset button in this case.
FYI: one of the authors of this article, Karen Hao, just announced on Twitter that she's writing a book on OpenAI, and that this article is partly based on work done for that book.
> OpenAI’s president tweeted that the tool [ChatGPT] hit 1 million within the first five days.
Perhaps the reason why ChatGPT has become so popular is because it provides entertainment. So it is not a great leap forward in AI or a path to AGI, but instead a incredibly convoluted way of keeping reasonable intelligent people occupied and amused. You enter a prompt, and it returns a result - what a fun game!
Maybe that is it's primary contribution to society.
I can only speak for me, but I derive no entertainment from "chatting" with a computer program. I have only used it in ways that are useful to me. And I found it very good at troubleshooting software config issues as well as generating customized boilerplate code and even for generating fake data to insert into databases for testing. I have also found it quite good at extracting data from unstructured information like logs. Something that before, would have required maybe 20 minutes of fiddling with regex and writing a script for.
Personally for me I've found that since Google search has been wrecked by SEO, or pivoted to targetting zoomers and gen alpha, it just doesn't work the same for me in my 40s. It used to be fantastic, but several times a day my search queries end in frustration like they did in the pre-Webcrawler search era.
Increasingly now I use ChatGPT and sometimes Kagi. And they just work like I expect. I can think of one time that ChatGPT has failed me, which was when I was trying to remember the terms OLTP/OLAP in database architecture. But for a long time now it's been a very effective tool in my toolbox, while Google increasingly wears out.
ChatGPT provides real utility for me. Some examples:
1. Drafting directional copy I can give to a real copywriter to create something we'd all be happy presenting to users.
2. Act as a sounding board for peer-personnel issues I'm dealing with at work.
3. "Dumb down" concepts in academic journals/articles such that I can make sense of them.
4. Just today it helped me build an app to drill a specific set of chord shapes/inversions on the guitar that I've been struggling with (programming has always been a very casual hobby and, consequently, I'm not very good at it).
When people write stuff like this, I wonder if they've ever seriously used ChatGPT at all.
I use ChatGPT all the time in my software dev job and I find it incredibly helpful. First, it's just much faster than pouring over doc to find the answer you want when you know what you're asking for. Second, when you don't know exactly what you're asking for (i.e. "How would I accomplish X using technology Y"), it's incredibly helpful because it points you to some of the keywords that you should be searching for to find more/corroborate. Third, for some set of tasks (i.e. "write me an example of code that does this") I find it faster to ask ChatGPT to write the code for me first - note this one is the least common of the tasks I use ChatGPT for because I can usually write the code faster.
Yes, I know ChatGPT hallucinates. No, I don't just copy-and-paste the output into my code and press enter. But it saves me a ton of time in some specific areas, and I think that people that don't learn how to use generative AI tools effectively will be at a huge productivity disadvantage.
Does anyone else think it would make for a more healthy dynamic from the standpoint of AI safety if both sama and Ilya remained, despite their differences? Not that I know anything, but it seems a diversity of opinions at the top could have its benefits.
> Safety teams within the company pushed to slow things down. These teams worked to refine ChatGPT to refuse certain types of abusive requests and to respond to other queries with more appropriate answers.
I wonder what this struggle means for the future of ChatGPT censorship/safety.
i suspect that 'AI safety' means a chatbot that is not hallucinating/making things up. Ilya Sutskever may want some sort of hybrid system, where the output of the LLM gets vetted by a second system, so as to minimize instances of hallucinations, whereas Sam Altman says 'screw it, lets make an even bigger LLM and just push it'.
Is that right?
Don't know if Altman or Sutskever is right, there seems to be a kind of arms race between the companies. OpenAI may be past the point where they can try out a radically different system, due to competition in the space. Maybe trying out new approaches could only work in a new company, who knows?
Inward safety for people means their vulnerabilities are not exposed and/or open to attack. Outward safety means they are not attacking others and looking out for the general wellbeing of others. We have a lot of different social constructs with other people of which we attempt to keep this in balance. It doesn't work out for everyone, but in general it's somewhat stable.
What does it mean to be safe if you're not at threat of being attacked and harm? What does attacking others mean if doing so is meaningless, just a calculation? This goes from everything from telling someone to kill themselves (it's just words) to issuing a set of commands to external devices with real world effects (print ebola virus or launch nuclear weapon). The concern here is the AI of the future will be extraordinarily powerful yet very risky when it comes to making decisions that could harm others.
yes, still it boils down to the question of 'how well is the output of the chat bot aligned with reality' ? if you want to automate this, then you will likely need some system that is kind of censoring the output of the LLM, and that system should have a better model of what is real.
Frenzied speculation swirls after ousted OpenAI CEO Sam Altman teased breakthrough that pushed 'the frontier of discovery forward' - just ONE day before he was fired amid reports he was seeking investors for new AI chip venture
So the decels basically owned the board, and drove a 100B value destruction because they disagreed with letting people use GPT-4 in ChatGPT. Some colleagues that is.
Is this decel movement just an extension of the wokeism that has been a problem in SV? Employees more focused on social issues than actually working.
The problem with the idealistic "we do research on alignment as we discover AGI, don't care about money" angle is that... you are not the only ones doing it. And OpenAI is trying to do it with its hands tied behind its back (non-profit status and vibes). There are and will be companies (like Anthropic) doing the same work themselves, they will do it for profit on the side, will rake in billions, possible become the most valuable country on Earth, build massive research and development labs etc. Then they will define what alignment is, not OpenAI. So for OpenAI to reach its goal, if they want to do it themselves that is, they need to compete on capitalistic grounds as well, there is no way around it.
>There are and will be companies (like Anthropic) doing the same work themselves
My understanding is that Anthropic is even more idealistic than OpenAI. I think it was founded by a bunch of OpenAI people who quit because they felt OpenAI wasn't cautious enough.
In any case, ultimately it depends on the industry structure. If there are just a few big players, and most of them are idealistic, things could go OK. If there are a ton of little players, that's when you risk a race to the bottom, because there will always be someone willing to bend the rules to gain an advantage.
Lol. So I didn't get past the few paragraphs before the paywall, and I didn't need to.
I appreciate the idea of being a "not-greedy typical company," but there's a reason you e.g. separate university type research or non-profits and private companies.
Trying to make up something in the middle is the exact sort of naivete you can ALWAYS expect from Silicon Valley.
The professors at the summer seminar at the Simons Insitute complained/explained (with Ilya himself present) that this research was impossible to do in university lab setting, because of the scale needed.
So I would say ChatGPT exists because its creators specifically transgressed the traditional division of universities vs industry. The fact that this transgressive structure is unstable is not surprising, at least in retrospect.
Indeed, the only other approach I can think of is a massive government project. But again with gov't bureaucracy, a researcher would be limited by legal issues of big data vs copyright, etc.--which many have pointed out that OpenAI again was able to circumvent when they basically used the entire Internet and all of humanity's books, etc., as their training source.
I'd say yes, Sutsksever is... naive? though very smart. Or just utopian. Seems he couldn't get the scale he needed/wanted out of a university (or Google) research lab. But the former at least would have bounded things better in the way he would have preferred, from an ethics POV.
Jumping into bed with Musk and Altman and hoping for ethical non-profit "betterment of humanity" behaviour is laughable. Getting access to capital was obviously tempting, but ...
As for Altman. No, he's not naive. Amoral, and likely proud of it. JFC ... Worldcoin... I can't even...
I don't want either of these people in charge of the future, frankly.
It does point to the general lack of funding for R&D of this type of thing. Or it's still too early to be doing this kind of thing at scale. I dunno.
> Jumping into bed with Musk and Altman and hoping for ethical non-profit "betterment of humanity" behaviour is laughable.
Now it's laughable, but OpenAI was founded in 2015. I don't know about Altman, but Musk was very respected at the time. He didn't start going off the deep end until 2017. "I'm motivated by... a desire to think about the future and not be sad," was something he said during a TED interview in 2017, and people mostly believed him.
Which is to say, I think the fearmongering sentient AI stuff is silly -- but I think we are all DEFINITELY better off with an ugly rough-and-tumble visible rocky start to the AI revolution.
Weed out the BS; equalize out who actually has access to the best stuff BEFORE some jerk company can scale up fast and dominate market share; let a de-facto "open source" market have a go at the whole thing.
The bleak reality is that OpenAI became that jerk company, and now in its effective demise, those reigns are handed over to Microsoft. Who have already demonstrated (with CoPilot) a similar lack of ... concern ... for IP rights / authorship, and various other ethical aspects.
And bleak because there doesn't seem to be an alternative where the people making these decisions are responsible to an electorate or public in some democratic fashion. Just a bunch of people with $$ and influence who set themselves up to be arbiters ... and
It's just might makes right.
And bleak because in this case the "mighty" are often the very people who made fun of arts students who took the philosophy and ethics classes in school that could at least offer some insight in these issues.
Let's suppose that AGI is about to be invented, and it will wind up having a personality similar to humans. The more that those are are doing the inventing are afraid of what they are inventing, the more that they will push it to be afraid of the humans in turn. This does not sound like a good conflict to start with.
By contrast if the humans inventing it go full throttle to convincing it that humans are on its side, there is no such conflict at all.
I don't know how realistic this model is. But it certainly suggests that the e/acc approach is more likely to create AI alignment than EA is.
You are chaining some big assumptions together to make this conclusion. We can suppose AGI is around the corner, but these other assumptions are massive leaps that would need strong arguments to back them:
- AGI thinks similar to humans
- AGI knowing we are afraid of it will make it consider humans a threat
Unpacking that second point are the implications that:
- AGI considering humans a threat is conditional on our fearing it
- AGI seeing humans as a threat is the only reason it would harm humans
I feel like I can rule out these last 3 points just by pointing out that there are humans that see other humans as a threat even though there is not a display of fear. Someone could be threatening because of greed, envy, ignorance, carelessness, drugs, etc.
Also humans harm other humans all this time in situations where there was not a perceived threat. How many people have been killed by cigarettes? Car accidents? Malpractice?
And this is going off the assumption that AGI thinks like a human, which I'm incredibly skeptical of.
But our most effective experiment so far is based on creating LLMs that try to act like humans. Specifically try to predict the next token that human speech would create. When AI is developed off of large scale models that attempt to imitate humans, shouldn't we expect that in some ways it will also imitate human emotional behavior?
What is "really" going on is another question. But any mass of human experience that you train a model on really does include our forms of irrationality in addition to our language and logic. With little concrete details for our speculation, this possibility at least deserves consideration.
Why can't it feel fear? The model itself doesn't have any built in mechanisms sure but it can simulate an agent capable of fear. In the same way the simulation can have other emotions needed to be a better model of a human
We have no idea how to give anything a subjective experience of itself. We know how to make something behave as if it does externally.
One of the worst versions of AGI might be a system that simulates to us that it has an internal life, but in reality has no internal subjective experience of itself.
As I pointed out in a different comment, ChatGPT and friends are based on predicting the training data. As a result they learn to imitate what is in it.
To the extent that we provide the training data for such models, we should expect it to internalize aspects of our behavior. And what is internalized won't just be what we expected and were planning on.
