The first runaway "AIs", so to speak, may be https://en.wikipedia.org/wiki/Waqf which are like an early Islamic version of the modern trust.
These Waqfs grew so large that at times, they controlled a significant fraction of the economy of various Islamic countries. Many functions like social welfare programs were completely funded by these entities. One of the reasons Waqfs gradually declined is that they threatened the authority of governments in the same region.
There are certain US states which do not have the Rule Against Perpetuities, so it's possible that even in the absence of modern AI, we would have seen the resurgence of Waqf-style runaway perpetual autonomous corporations.
The failure mode of governments is to become “exploitative and corrupt,” Runciman notes. The failure mode of corporations, as extensions of an independent civil society, is that “their independence undoes social stability by allowing those making the money to make their own rules.” There is only a “narrow corridor”—a term Runciman borrows from the economists Daron Acemoglu and James A. Robinson—in which the artificial agents balance each other out, and citizens get to enjoy the sense of control that emerges from an atmosphere of freedom and security. The ideal scenario is, in other words, a kludgy equilibrium.
This is really interesting; to draw parallels between AI and the emergence of States and Corporations, which are two other inventions that we have lost control of. You could argue that Climate Change is an X-risk bought about by Corporations, which lumber around our civil landscape like single minded giants pursuing profit above all else and using regulatory capture to push externalities onto the populace.
I'm a big fan of The Corporation (2003), a documentary that explores how Corporations came to have legal rights, and asks, if Corporations are legal people, what sort of people are they? They exhibit reckless disregard for the safety of others, deceitfulness, the incapacity to experience guilt, and the failure to conform to social norms and respect the law, which matches the DSM-IV's symptoms of psychopathy.
I'm sorry to always inject nuclear weapons into everything[1], but the relationship with nuclear technology is another one.
Nukes in the silo essentially act as an enforcement penalty on their own people. No dissident in a nuclear state is willing to openly accept foreign aid for fear of the enemy; no foreign party is willing to openly aid dissidents in a nuclear state for fear of the enemy. Effectively, the stability of both nuclear states is created by the technology that they are both forced to maintain. They're now invasion-proof - the absolute core selling point of getting these things - but the technology over time inherently creates a stratified state. Effectively, it stands as its own system, the weapons creating hubris which creates fear which creates more nuclear states.
Veterans often state that in theatre, combat conditions, there is a sensation that the war is also a self-guiding machine, somehow separate from the men fighting it. While evocative, it is, of course, an illusion. The ones making war aren't the ones fighting it, so it seems guided by unseen malevolence.
[1] Kinda-not-really-sorry. I still don't think any human civilization has come to grips with the tech yet. It's disconcertingly like machine guns in 1897.
> Kinda-not-really-sorry. I still don't think any human civilization has come to grips with the tech yet. It's disconcertingly like machine guns in 1897.
Funny that you mention machine guns. There is a thesis to be written (if not already existing) on how both Truman and Eisenhower viewed tactical nukes as, well, very large conventional weapons and potentially usable as such on the battlefield, while still viewing strategic nukes as something different, akin to what you described in your second paragraph.
The thesis would deal with both presidents' military backgrounds affecting their opinions, and how being intimately involved with the discussion of just how much more powerful thermonuclear weapons are than fission weapons affected taking such a nuanced view, an ability that I think has been lost to later generations (I doubt 5% ordinary citizens can explain the difference between fission and fusion bombs I'd be amazed).
Lost on later generations in the west, I think. I have a confession to make: that nuke - machine gun analogy is not mine, but that of a military officer from a non-Chinese southeast Asian navy I had chatted up[1] at a conference.
I would not be terribly surprised if the showdown in East Asia sees unconventional usage of tactical weapons, as area denial or similar.
Even if a tactical weapon was employed against a USN carrier in theatre, it's honestly a hard call if that warrants tactical weapons against PLA land targets, because those will have inevitable civilian casualties. Probably mass casualties. If we happen to have decent US executive leadership at that moment, they'll probably see that the risk of a countervalue exchange is too high, and opt for conventional land strikes, a terse warning[2], and some serious international consequences. Otherwise the only other good option is total commitment, and praying for forgiveness.
In fact, laying out that red line the instant shooting starts is probably not a bad idea. Let's play with planes and boats (and sailor's lives) for a while, but keep your big guns in the holster.
[1] After he had left some burns on the introductory speaker at the Q&A session regarding mil-std specifications.
[2] "Any future usage of nuclear weapons will warrant a full strategic strike on your country. Not even rubble."
>because those will have inevitable civilian casualties. Probably mass casualties
I suspect civilian casualties will not be a concern of the US in such a scenario, even if the US carrier group is a military target.
> Otherwise the only other good option is total commitment, and praying for forgiveness.
I think that "total commitment" can be done, in theory. I suspect that the US and probably UK achieved the ability to follow enemy SSBNs before the end of the Cold War. A first strike by SSBNs to minimize delivery time followed up by other delivery vehicles, combined with SSNs taking out every Chinese SSBN. In theory.
I don't think anyone's going to be concerned about much of anything regarding civilians, but mass civilian casualties have a reasonable probability to trigger a Chinese countervalue strike. Sidenote: that's something they've been arming up for intensely the last few years, as well as preparing a vast number of "dummy" (or are they dummies?) siloes and some extensive infrastructure building for TELs. Anyway, with that calculus in mind, the US might just decide to skip the middleman and go for total (or as near-as-total as they can manage) commitment, or, alternatively, broaden the conventional war to include a wider range of land targets, underline the red line of further nuke usage, and something that would look like a total embargo (using superior USN sub forces) and cutoff from the world financial system.
I personally think China would fear option 1 enough to not employ tactical weapons in any application, unless they were looking at invasion of the mainland. I'm simultaneously worried - although I don't think it likely - that poor leadership on the US side could retaliate with tactical weapons for any carrier sinking, conventional or not. It's gonna be a tense time regardless - hopefully the world navigates it[1] without the atom baring its fangs once again. Because when I say "nukes are like machine guns", that's not really selling it. A better comparison would be "the invention of fire", which gave rise to genus Homo.
Also yeah, the "in theory" part is a rough one, given the chips being put on the table.
[1] Which would be as simple as just saying out loud what country Taiwan is. Country or no? As with so many other ambiguous zones causing problems around the world. Oy.
These Waqfs grew so large that at times, they controlled a significant fraction of the economy of various Islamic countries. Many functions like social welfare programs were completely funded by these entities. One of the reasons Waqfs gradually declined is that they threatened the authority of governments in the same region.
1600s Europeans were worried enough about this (runaway legal entities caused by interest-bearing trusts) that they invented the https://en.wikipedia.org/wiki/Rule_against_perpetuities
There are certain US states which do not have the Rule Against Perpetuities, so it's possible that even in the absence of modern AI, we would have seen the resurgence of Waqf-style runaway perpetual autonomous corporations.