Here's what gets me about this: Mark Zuckerberg bought like 1500 acres of land in Hawaii and is apparently building an underground apocalypse bunker on it. He's doing so, presumably, because he's worried about society collapsing. He's doing it with money that he's made from Facebook, which is the _reason_ society is collapsing.
Mark Zuckerberg has worked his ass off for like 20 years to go live in an underground bunker. Granted, it's a large and luxurious underground bunker, but this does not seem like an improvement.
Mark Zuckerberg controls the largest mass-persuasion device on earth - literally controls, he's got more than 50% of the shareholder votes - and is using the proceeds from owning the globe-spanning apocalypse machine that he's turned Facebook into to build himself a bunker underground to survive the apocalypse his machine is creating.
Just turn off the goddamn machine and you can actually enjoy Hawaii, Mark. All the good stuff is above ground!
The mental model precludes the self awareness you describe. That’s why laws and regulators are needed: you cannot entirely take the lizard brain out of the human.
You’re not going to convince people like Mark, but government can grind slowly, forever. No one else is going to protect you from these people at scale.
Oh I know that. The whole process is so obviously self-defeating that there’s clearly no actual rationality to it. It’s just a shame we’re all stuck on this boat with him.
Mega prepper bunker is a bigger signal of selfishness and not giving a fuck what happens to everyone else. I hope it becomes a Winchester Mystery House-type tourist attraction in 100 years.
If only it were possible to just turn it off and have it not exist! Facebook is a faucet of money, and were that Zuckerberg turned his off, one of his competitors or someone new would fill the void. Maybe they’d be better than he is, maybe not. Either way we can’t go back to 2000.
I dont agree with your psychoanalysis, but I do think it is interesting. IF I accept everything you say is true, there are still several questions.
IF Zuckerberg turns off facebook, will Armageddon be averted? will it be hastened?
As long as 1 human in 8 billion is willing to make Facebook 2.0, apocalypse is still on. The only durable solution is for the majority of humans to moderate and control themselves. IF this seems farfetched, consider that self control and moderation has always been a prerequisite for human life and society, albeit with different subjects.
I'm being moderately facetious with "turn off the machine", but - there isn't another Facebook. Facebook has a billion active users, and the Facebook algorithm, Facebook groups, and Facebook interaction design patterns are phenomenally powerful in affecting social discourse. As structured, they nudge towards discord, they reward clannishness, they propagate outrage, they generally push the worst versions of ourselves, and they do so while pushing every dopamine button we've managed to find in the human brain to make people stay within that environment. All of that is Facebook: all of that is what Zuckerberg built, and all of it follows his command. We can lump Instagram and all its various properties in there, too.
I don't know what a version of Facebook looks like that doesn't promote the apocalypse, but I'm also not the guy with his hands on the wheel, and that guy also seems scared of what it's doing. I'm also not particularly sympathetic to the "_someone_ would've sold those kids drugs!" argument.
This is an excellent question which deserves a response. I would argue that there are many factors other than Facebook which are leading the race towards a collapse of civilization.
Some may point to being sued for 150 billion for its alleged responsibility in the Myanmar genocide which was viewed by many as a standin case for countless micro variations of similar incidents that rippled throughout the network daily. And scandals like Cambridge Analytica. The thesis being that it has been transformed into a weapon that powerful entities use to manipulate, control, and harass the weak.
This is going to sound like whataboutism, but while I deleted my FB 10 years ago, it is worth noting that there are also legacy factors like CNN & Fox News (>200k dead in Iraq,) the petroleum cartels, proudly neo-nazi friendly X.com, the greed is good C-suite circle jerk, and many more a-holes in addition to FB.
FB deserves derision, but it is far from the sole reason that civilization is declining.
Tech CEOs and "tech culture" has a fundamentally different understanding of causality and human agency than the author.
Cybertrucks arent what make the world of bladerunner a dystopia.
AI isnt what makes the world of "Her" a dystopia.
Techies see dystopian Sci-fi not as cautionary tale which suggests technology should be avoided, but rather a cautionary tale about how humans should interact with technology, and human behavior to be avoided.
I think this can be argued to be an optimistic view of human capacity, choice, individual agency. Naturally, it conflicts with pessimistic views where humans have low-agency and are at the mercy of their environment.
One side thinks that when presented unlimited ice cream, people can learn to control themselves. The other side thinks that when presented unlimited ice cream, people will eat themselves to death.
That among humans rich with agency, there's inevitably someone who can see the opportunity in restricting access to the unlimited supply of ice cream as a way to control others and reduce their agency.
There are billions of people with "agency" navigating an unfathomably complex and dynamic environment that gains shape by the collective results of their individual actions. It's really not so binary as pessimism vs optimism, agency vs passivity, humanity vs environment.
It goes further. Not only it doesn't matter what most will do with tech, tech itself doesn't matter, even in the worst hands.
Tech is irrelevant.
The problem is the combination of the people who want to abuse others, and the vast majority who allow it.
Or maybe better, the people who are willing to manipulate others, and the majority who are manipulable.
That problem exists regardless of tech and isn't created or even amplified by tech.
I say it's not even amplified by tech because techs power amplifying nature works for everyone. Every power tech gives to some asshole, it also gives to you and me to get around the asshole.
The only thing we can't get around is the complacency of everyone else who allows Amazon and WalMart to erase all the other shops. A douchebag figures out a button to press, like convenience and price, and most people happily just respond to that button without thought or resistance. And once enough of them do, the environment changes and no one else has any choice but to exist in the new worse environment.
The problem of most people not being skeptical and pessimistic and assuming the worst and rejecting shiny offers and propositions and ideas by default, resulting in them all being taken advantaged of and resulting in terrible systems that you and I have to live in, has always been, since there was no tech beyond mere language.
>That among humans rich with agency, there's inevitably someone who can see the opportunity in restricting access to the unlimited supply of ice cream as a way to control others and reduce their agency.
I think that is a real phenomenon, but different. The article seemed to focus on new technologies that are being brought into existence. It is one thing invent ice cream. It is another to figure out a mechanism to monopolize ice cream access.
If someone invents a new flavor of ice cream and monopolizes it, are they reducing the options available to individuals? What about if it is so good that nobody wants other flavors? What if the other flavors stop being made?
That is a very cynical take. Optimism and adaptation also have a basis in neuroscience.
Neuroscience does not mean that everyone does the most degenerate thing possible at every opportunity. There are people that don't eat themselves to death when presented the option.
There is also neuroscience behind discipline, self sacrifice, and every other positive virtue you can imagine.
Well said. The author's argument is equivalent to saying advanced medical technology (or any advanced technology) is dystopian because it's in a bunch of scifi movies that happened to be dystopian.
I think the simpler explanation is that they’re struck by the style of these artworks, and aren’t interested in referencing the cautionary tale aspect.
Bladerunner, Snowcrash, The Matrix, feature ‘cool’ characters and settings. It’s cool to imagine being a crew member on the Nebuchadnezzar or whatever.
No one references 1984, Brazil, or Children of Men to advertise their product, because those artworks create an uninviting, flat bleakness.
Maybe some blame should go toward authors who work hard to draw you in with style and vibe, in order to tell you a cautionary tale. Many in the audience probably miss the tale entirely, and are just immersed in the looks and vibe.
At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus because they had really cool looking uniforms.
Pretty much every possible word is an English word, it seems.
"English is a whore, sleeping with every language she encounters, and with a very successful sideline in pickpocketing, rifling through all their pockets for any word she can find", or however the old quote goes.
Was I the only one who actually watched this movie? Her is not a dystopia. It's barely even cautionary. The main character seems better off having loved and lost than never having loved at all.
I agree with you about it not being a dystopia but clearly it's a cautionary tale. The message I got from Her is not "Don't create the Torment Nexus" but "Touch Grass" or maybe "Don't force it." The narrator is profoundly socially alienated after his divorce. He's hung up on his ex-wife he's lonely, so he goes on all these dates with other people like Olivia Munn which are horribly awkward so he withdraws even further. Finally he finds Scarlett JohanssAIn and everything is perfect blah blah blah. But clearly it's not going to last as AI Scarlett clearly is in a different league and goes on to cheat on him and accelerate the singularity. He ends up with his awkward neighbor who isn't perfect and has her own skeletons but is more fundamentally compatible than other people he's been with.
I actually just read the Wikipedia article about the writing of the script and Spike Jonze has explicitly said he was inspired by the false allure of chatting with a chatbot. So clearly the intent of the work is in strict tension with the GPT-4o demo.
I'm honestly pretty puzzled by the tech CEOs constantly referencing fictional dystopias. I honestly cannot believe they're ignorant of the artistic intent of most of these works. I reread Snow Crash after the original Metaverse announcement and I kept thinking "Did he actually read this book?"
> Was I the only one who actually watched this movie? Her is not a dystopia.
I watched it, and it kind of was a dystopia. Not in the classic jackboots and oppression sense, but one of pervasive social disconnection and isolation. I mean, the main character's job was literally to write "heartfelt" letters to other people's loved ones on their behalf, because they couldn't be bothered to do it themselves.
You don't learn reliable lessons from fiction. The author of "Don't create the Torment Nexus" may have intended it as a cautionary tale whose lesson is not to create the Torment Nexus, but that doesn't obligate the reader to come to that conclusion.Why shouldn't we say with a couple minor tweaks and a sexier name this "torment nexus" would be pretty cool?
Conversely, most "utopias" sound like they would be pretty horrible to actually live in.
> The author of "Don't create the Torment Nexus" may have intended it as a cautionary tale whose lesson is not to create the Torment Nexus, but that doesn't obligate the reader to come to that conclusion.
One of the points of the Torment Nexus joke is that the Tech Company is totally oblivious to the fact that "Don't create the Torment Nexus" was describing something bad.
> Why shouldn't we say with a couple minor tweaks and a sexier name this "torment nexus" would be pretty cool?
I actually think your defense there actually gets to the core error: things can be simultaneously cool and awful. For instance: nuclear war. On the one hand, you have high tech rockets buzzing around insanely fast while precisely dropping incredibly powerful fireworks; which is admittedly pretty "cool," on the other hand, you have mass death and suffering. The Tech Company people are shallowly responding to just the "cool" part, and missing all the rest.
There are plenty of examples of "critical" utopias which do not try to be absolutely perfect, and acknowledge where they fall down. Anarres in the Dispossessed by Le Guin, and the Culture novels by Iain M Banks might be two examples. They're still vastly preferable (imo) to current life on earth.
Huxley's _Island_ didn't seem too bad. I think the general problem is, if you want to completely eliminate crime, poverty, and conflict in general, you have to have a society that is extremely regimented, and you have to be willing to kill off anyone who seems "defective".
Well, just because he made iPads inspired by Star Trek doesn't mean Steve Jobs wanted war with the Klingons. Although apparently his company wants musical instruments and works of art to be crushed out of existence and replaced with a cold digital slab. So regardless of the level of malice, it's obvious that the tech industry is getting creepily out of touch with non-virtual reality.
- Siri had come out two years before "Her" was released to theaters, so the critique was more about voice assistants (which is a form of AI) than AGI or the AI sota.
- The closest thing to "Her" would be a Humane Pin with 4o or higher. A Humane Pin like experience that didn't suck would honestly be super cool. That would allow me to leave the phone at home!
> So it turns out it is aspirational branding, it’s just a deeply
misanthropic variety
Not misanthropic. Masochistic. I will to be writing something more
substantial on this soon, but the gist is; a lot of these projects are
knowingly self-destructive, and what their leaders crave is for others
to join them in their expressins of self-harm.
I disagree: The real goal is to get others to endorse and support their worldview about how the world works and what their own place within it is.
That doesn't come from the organizer enjoying pain/harm, but from desperately seeking to avoid experiencing a certain kind of it, emotional pain or harm to their ego.
That said, if someone else is willing to harm themselves (physically, financially, legal jeopardy) to "prove" their allegiance to the narrative, that may be considered a more powerful and pleasurable endorsement. Also a sunk-cost also makes it harder for the person to change their mind later.
In other words, more like "messianic cult-leader" and less "Pinhead from Hellraiser."
Yes, there's an element of messiah and saviour complex in there.
> That doesn't come from the organizer enjoying pain/harm, but from
desperately seeking to avoid experiencing a certain kind of it,
emotional pain or harm to their ego.
I think you actually describe a good picture of sado-masochism, the
protective mechanism against emotions of unworthiness, though maybe
you assumed I was emphasising the sadism side.
Of course no actual tech leaders are exactly like that (they're real
people with their own subtle issues that are personal, private and
none of our business - even if they would not allow the same dignity
for everyone else). We are talking about archetypes here. Jobs was
paradigmatic. And not limited to tech, things like drugs (and the
media/arts) are a space where you can achieve mass-reach much more
easily. For me, the portrayal of Richard Sackler (Michael Stuhlbar)
in Dopesick [0] is an even better depiction of the tech leader
archetype than the fictional Peter Isherwood (Mark Rylance) in Don't
Look Up [1].
We talk about "disruption" as if it were a synonym for progress but
what I see is how many tech-obsessed "creator types" actually delight
in destruction, and the pain experienced by others who they affect.
Of course they would say, with that twisted Nietzschian faux-Christian
'compassion', that they "love" humanity and don't want to see anyone
"left behind" (whether they want to be left out of an insane project
or not).
The recent Apple advert was a timely expression of this. I'm glad they
made it. Its better to have this "out" than ignored.
I only saw Her once, when it came out, but I didn't view it as an AI dystopia. The dystopia existed before the AI, and was mostly meant to represent the world some of us already lived in, but which was not evenly distributed at the time. I thought the AI part was meant to be a metaphor for personal growth. I agree that trying to recreate the world of Snow Crash or Neuromancer is missing the point, and some people do seem to be trying to do that, but the central example used in this essay also feels like it's missing the point, just in a different way.
> What’s the common denominator [...]? That the presumed user or owner of the product is the protagonist! [...] the narcissistic, us-against-the-collapsing-world mentality
Another media tie-in here might be Atlas Shrugged. It too is a narrative to elevate an inventor--or more often investor--into an ubermensch hero role, one capable of projecting power against a dilemma that is mostly comprised of other people who failed to give the protagonist the respect/worship they were due.
I don't think it's a cynical strategy to sell products per se, but it comes about because the person with influence over the marketing is able to pick and promote a flattering story or framing.
____________
Also, as referenced in the article, the classic:
> Sci-Fi Author: "In my book I invented the Torment Nexus as a cautionary tale."
> Tech Company: "At long last, we have created the Torment Nexus from the classic sci-fi novel Don't Create The Torment Nexus."
> It too is a narrative to elevate an inventor--or more often investor--into an ubermensch hero role, one capable of projecting power against a dilemma that is mostly comprised of other people who failed to give the protagonist the respect/worship they were due
I've never read atlas shrugged and don't intend on it. But responding to this specific comment, if this planet had 0 creators and only bureaucrats, critics and consumers it would quickly go to hell.
Meanwhile, if we had no bureaucrats, critics, and consumers and only people that created things I'm fairly certain the world would be fine.
> Sci-Fi Author: "In my book I invented the Torment Nexus as a cautionary tale."
Just to be clear, we aren't talking about the torment nexus. We are talking about robots that can be interacted with using human language.
Mark Zuckerberg has worked his ass off for like 20 years to go live in an underground bunker. Granted, it's a large and luxurious underground bunker, but this does not seem like an improvement.
Mark Zuckerberg controls the largest mass-persuasion device on earth - literally controls, he's got more than 50% of the shareholder votes - and is using the proceeds from owning the globe-spanning apocalypse machine that he's turned Facebook into to build himself a bunker underground to survive the apocalypse his machine is creating.
Just turn off the goddamn machine and you can actually enjoy Hawaii, Mark. All the good stuff is above ground!