
US experts propose having AI control nuclear weapons - andyjohnson0
https://thebulletin.org/2019/08/strangelove-redux-us-experts-propose-having-ai-control-nuclear-weapons
======
pdkl95
That might be one of the worst ideas I've ever heard. Do you really want to be
near a nuclear weapon that is controlled by an opaque, unexplainable process?
Even without the insane unobservable complexity of machine learning, do you
want to be near a nuclear weapon co9ntrolled by a simpler, _theoretically_
explainable expert system "AI"?

Safety critical systems - including nuclear bombs - need to be simple, with
realistically-understandable deterministic behavior. The answer to "what would
our nuclear weapons do in ${any_hypothetical_situation}" _should never be_ "I
don't know".

Even worse, if an "AI" is involved (by any definition of "AI"), that means the
control system involves a Turing complete language. Asking questions about the
behavior of the control system for a given set of inputs probably shouldn't
require solving the Halting Problem.

edit: I suspect these so-called experts need to read the report[1] from the
Therac-25 investigation.

[1]
[http://sunnyday.mit.edu/papers/therac.pdf](http://sunnyday.mit.edu/papers/therac.pdf)

~~~
7952
According to MAD principles a "safe" deterrent is less effective. Because your
adversary must always think you are capable of delivering a retaliation. A
perfectly safe system, may prevent that retaliation and invalidate the
detterent effect. This issue applies to all technology and human systems
connected with nuclear weapons. The outcome of nuclear war is so horrific that
no technoogy would ever be good enough.

The solution is to eliminate them entirely and stop creating such insane
choices.

~~~
kranner
> The solution is to eliminate them entirely and stop creating such insane
> choices.

That solution entails magically ensuring that no one else has such weapons or
will develop them.

If we had a way of doing that, we wouldn't even need armies protecting
borders. You'd still need minimal border security to ensure individual
intruders don't make it through, but why maintain a hugely expensive army if
you could ensure that nobody else could create an army to invade you (or
anyone else?)

~~~
7952
Yeah, there isn't really a solution.

Personally I don't see long term national security as possible with nukes
either. Sooner or later a situation will exist where a bad actor is in power.
Or they are just launched by accident. And large countries are not guaranteed
to be politically stable. These large nations can and do disintegrate.

I think the risk of obliteration without nukes is less bad. I would rather see
my country destroyed in a first strike or invaded by a foreign power. I think
that would be less bad than a nuclear war.

The funny things is that the USA could easily be defended from conventional
attack. And a small sub based detterant would be good enough. And there is a
risk in stoking paranoia in your enemy by having large stockpiles. The vast
majority of nations have far less conventional defence than the US and still
don't want nukes.

------
umeshunni
Link to the actual proposal rather than the clickbait article:

[https://warontherocks.com/2019/08/america-needs-a-dead-
hand/](https://warontherocks.com/2019/08/america-needs-a-dead-hand/)

~~~
carlmr
I was a bit disappointed that while they did cite this they didn't link to it.

~~~
danielbln
They do though, in the second paragraph.

~~~
carlmr
If it is, it doesn't work on my phone or computer.

------
jacquesm
Of all the stupid proposals I've seen over a lifetime this one has to be the
worst of all. So much for the 'experts'.

------
mc32
Automate defensive systems, don’t automate the offensive launch of nuclear
weapon warheads. Whoever came up with that idea should be fired. You have to
know what the taboos are, what lines not to cross.

What’s next, simulated casualties are exchanged for actual casualties? I think
one star trek series had a plot reaching that logical conclusion.

------
xkcd-sucks
In the past 100 years, there have been several incidents where humans have
averted war by refusing to follow orders. There have been far fewer incidents
of humans causing nuclear war by either following or disobeying orders

~~~
LyndsySimon
> There have been far fewer incidents of humans causing nuclear war by either
> following or disobeying orders

True - but if you add the words "nearly" and "allegedly", then it's not such a
rosy picture.

The book "Red Star Rogue" proposes the idea that the Soviet submarine K-129
was commandeered and attempted to launch a first strike on the United States
in 1968. She was very far off her expected course when sunk, and allegedly
there was a fail-safe system that scuttled the boat when an unauthorized
attempt was made to launch her missiles.[1]

The US's nuclear weapons were capable of being used by a single individual for
quite some time, until the 1960s. Even with the advent of the Permissive
Action Link, until 1977 the 8-digit arming code was - seriously - "00000000".
They were worried they wouldn't have the code available if the weapons were
needed.[2]

There are numerous "close calls" that occurred, but those two came to mind
right away in the context of human beings causing or having nearly caused a
nuclear attack by disobeying orders in particular.

1: [https://www.kobo.com/us/en/ebook/red-star-
rogue](https://www.kobo.com/us/en/ebook/red-star-rogue) 2:
[https://en.wikipedia.org/wiki/Permissive_Action_Link#Develop...](https://en.wikipedia.org/wiki/Permissive_Action_Link#Development_and_dissemination)

------
tempodox
This proposal is so obviously and completely insane that it would only make
sense as a sort of distraction. I wonder from what.

~~~
credit_guy
Maybe it’s some form of argument where you present three hard-to-swallow
options (“Admittedly, each of the three options — robust second strike,
preemption, and equivalent danger — has drawbacks” [1]), and then you pretend
to offer a better alternative, which is actually so preposterous that the
first three options start looking perfectly reasonable.

[1][https://warontherocks.com/2019/08/america-needs-a-dead-
hand/](https://warontherocks.com/2019/08/america-needs-a-dead-hand/)

------
dogma1138
Terminator 6 marketing is getting out of hand.

------
inimino
Theory: they know something about AI that we don't and because of what they
know, decoded to scare all the world's AI practitioners into getting their
shit together at the same time. Alternate theory: they are idiots. Alternate
theory number two: they know something about AI and a lot about nuclear
doctrine that most people don't, and they are actually onto something...

~~~
nighthawk648
Even then it is irresponsible. Dogma is rapid in industries filled with
compartmentalizations.

To best avert disaster tech needs to first be open to the public and
submisiable to regulation.

~~~
inimino
> Dogma is rapid in industries filled with compartmentalizations.

Yes 100% agreed.

> To best avert disaster tech needs to first be open to the public and
> submisiable to regulation.

But we know that's never gonna happen with defence tech so...

------
danielmg
Why not remove PAL* and delegate authority to local commanders? I'd rather
take the risk of trusting a human over a human designed, highly complex
computer system.

In the UK, due to the lack of land-mass to absorb a strike, the PM issues
final orders to the commanders of the Trident subs __. This is part of the
deterrence strategy.

* [https://en.wikipedia.org/wiki/Permissive_Action_Link](https://en.wikipedia.org/wiki/Permissive_Action_Link) __[https://en.wikipedia.org/wiki/Letters_of_last_resort](https://en.wikipedia.org/wiki/Letters_of_last_resort)

~~~
michaelanckaert
The article on Letters of Last Resort made me think how absurd democracy can
be at times.

It looks like PM May wrote her Letters of Last Resort, and then Johnson did
the same. But from the looks of it, new Letters of Last Resort could be
written pretty soon. Just thinking of those sub commanders, receiving new
letters last week and then getting another set of letters just a couple of
weeks later.

------
credit_guy
The proposal is indisputably harebrained, but the question is why the authors
decided to publish it? One of the authors is a former US Air Force colonel.

And in particular why they choose to propagate falsehoods like "Hypersonic
cruise missiles are powered all the way to their targets using an advanced
propulsion system called a SCRAMJET. These are very, very, fast. You may have
six minutes from the time it’s launched until the time it strikes." Lest it be
any confusion, scramjets are fast, but are nowhere as fast as ICBM's (Mach
5-10 vs Mach 20-23).

~~~
LyndsySimon
> One of the authors is a former US Air Force colonel.

I was curious, so I looked it up. There are about 3,300 colonels in the US Air
Force:

[https://www.statista.com/statistics/239365/total-military-
pe...](https://www.statista.com/statistics/239365/total-military-personnel-of-
the-us-air-force-by-grade/)

> Lest it be any confusion, scramjets are fast, but are nowhere as fast as
> ICBM's

I agree that's weird phrasing and incorrect on its face. That said - as I
understand it, cruise missiles don't have the launch signature ICBMs have.
While it would take longer for a cruise missile to travel the same distance as
an ICBM, it might in fact give the target country less time to react if they
aren't detected until they are well on their way.

------
jacknews
What could go wrong?

[https://www.rottentomatoes.com/m/colossus_the_forbin_project](https://www.rottentomatoes.com/m/colossus_the_forbin_project)

~~~
lostjohnny
I came here to see if someone already mentioned Colossus.

Funny trivia: there are parts of that movie shot in Rome, my city; you can see
the Vatican, San Peters, the Orange Garden, the Janiculum and Isola Tiberina,
specifically the hospital "Fate bene fratelli", where I was born.

The Americans meet there with Russians, joining forces to stop Colossus, and
you can see written on a wall a mirrored "W Lenin".

The writing was actually there before they shot the movie, it was not added on
purpose and stayed there until not long ago.

[https://i.imgur.com/wIha0mr.png](https://i.imgur.com/wIha0mr.png)

~~~
jacknews
cool! great classic movie, beautiful city!

------
ramblerman
Does the term expert as used in the media still convey any meaning?

~~~
b3orn
The "experts" in this case seem to be defense experts, not AI experts.

~~~
3131s
Not really defense experts either though, eh? Maybe "morons" is more fitting.

~~~
Nasrudith
Defense /industry/ experts are experts at pushing fear to sell expensive
"solutions" of marginal to negative value.

------
a3n
The problem with AI is that it doesn't get cold feet, and can't intuit the
likely truth against the present "facts."

[https://duckduckgo.com/?q=Russian+officer+prevented+wwiii&t=...](https://duckduckgo.com/?q=Russian+officer+prevented+wwiii&t=fpas&ia=web)

~~~
icebraining
I'm not saying it's a good idea, but software can and has been written to
evaluate the likelihood of the validity of certain facts using heuristics.
Petrov said the official training taught that an American first attack would
be massive, not just a few missiles; that same knowledge could be used in an
algorithm.

~~~
Nasrudith
Doesn't that suggest the worst possible act of mass slaughter in that system
would be to get a "massive" array of fake missle drones or dud missles from
the direction of a nuclear power to spark dual MAD strikes from a third party?

It would shift it to a hackable spoofable system. Said bad actor could state a
"technically I didn't kill millions/billions you did" and be right in a way.

~~~
icebraining
Yes, but that's a failure mode of any system, including with humans: a
realistic false flag attack can provoke a counter-attack against your real
target. The solution is to improve your capacity of distinguishing true
attacks, regardless of whether the analysis is made by a human or software.

~~~
Nasrudith
I suspect the real danger of the automated system is speed and with it being
beyond the ability of others to counter it - which means the system may
overreact immediately with nothing to be done to stop it in matters of
ultimate stake.

------
imvetri
AI do not have a thinking mind like humans do.

Developer/ Researcher: Hold down boy, we can do that.

Me: Can you create with a goodness in mind?

Developer/ Researcher: Oh yeah, we can train it speaking to a good mind.

Me: Wrong!. you gotta ask whats a good mind first!. Define it, you will stand
corrected and you will realise whole concept of tech is absurd.

------
dimitar
I don't see how faster ballistic missiles change the game. In the past you
still could do surprise attacks and example eliminate the chain of command, or
the lines of communication. While there are reports that suitcase nuclear
devices were developed.

However what ultimately fixed this issue in the 60s was the adoption of
nuclear submarines - you can surprise the adversary, but because you cannot
find the subs the retaliatory strikes are assured.

------
melling
In the near future, we are going to have autonomous battlefield weapons:

[https://www.theatlantic.com/technology/archive/2019/09/kille...](https://www.theatlantic.com/technology/archive/2019/09/killer-
robots-and-new-era-machine-driven-warfare/597130/)

------
Anarch157a
Čapek warned us of this in 1921. Give a machine inteligence enough to be
dangerous, without the emotional safeguards of humans and disaster is certain
to follow.

AI controlled weapons... Holy shit... Stupid ideas like that makes contemplate
moving to the middle of the Pantanal or Amazon jungle to live of the land,
away from all this madness.

------
mlang23
Maybe people should watch more old movies from the 20th century. This reads
like satire, but its true!

------
heyflyguy
Shall we play a game?

~~~
_iyig
For the unfamiliar:

[https://youtu.be/NHWjlCaIrQo](https://youtu.be/NHWjlCaIrQo)

EDIT: Well sue me for footnoting a cultural reference, I guess.

------
ddtaylor
This article seems like rage bait.

------
mnm1
I think Congress needs to pass legislation banning such an insane, stupid
idea. And we should stop calling it "artificial intelligence," as if some
software is actually intelligent comparable to humans. This is just software
and as with all software, it has bugs. Trusting it to not start a nuclear war
is essentially saying, "this software, which we can't even explain how it
works (cause ai), is bug free." Insane. This whole ai movement is dangerous
but not because something like Hal or the terminator will be created but
because stupid people like these will convince the government to put such
weapons under the control of buggy software no one, including its creators,
understands with horrific consequences. All so we can make sure that the whole
world is destroyed in case of nuclear war. The insanity is truly mind
boggling.

~~~
inimino
Congress isn't gonna pass legislation to ban the military from suggesting
stupid ideas, or the executive branch from approving them, because that itself
would be a stupid idea that reduces American defence readiness and suppresses
worthwhile ideas as well as bad ones.

> as if some software is actually intelligent comparable to humans

How's your Go game coming along over there? (Have you ever even played?
Because if you had I think you'd have more respect for the intelligence
required (for a human) to succeed even moderately at the game.)

> not because something like Hal or the terminator will be created

Actually it is pretty dangerous for those reasons too.

~~~
mnm1
Playing go or chess or any other game isn't intelligence. The computer isn't
intelligent. It's programmed for one task and one task only. That's great
programming and use of statistics etc. but that's not intelligent.

Anyway, if you're worried about terminator and Hal like beings, it's futile to
discuss intelligence when you are living in a science fiction world. In our
world, those things do not exist and there is no sign that they will exist
anytime soon. What does exist and is scary is bug ridden software that
controls weapons. That doesn't make for great science fiction though.

~~~
inimino
If your private definition of intelligence excludes everything machines have
already been made to do, I suspect it is indeed pointless to discuss machine
intelligence with you.

Personally, I'm not hung up on whether or not submarines can swim. If you are,
please give me your clear definition of intelligence and explain how Go and
chess and "any other game" are excluded by it.

~~~
mnm1
If submarines are intelligent then cars and bikes must be intelligent also
since they operate on similar principles. If a bike is intelligent, then pogo
sticks are intelligent too. I suppose at that point, anything can be
intelligent. What kind of intelligent things will bikes and cars and pogo
sticks and submarines do if all humans ceased to exist? Let me guess: none.

~~~
inimino
I was referring to the famous quote from Dijkstra:

"The Fathers of the field had been pretty confusing: John von Neumann
speculated about computers and the human brain in analogies sufficiently wild
to be worthy of a medieval thinker and Alan M. Turing thought about criteria
to settle the question of whether Machines Can Think, a question of which we
now know that it is about as relevant as the question of whether Submarines
Can Swim."

[http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EW...](http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD898.html)

Nobody is suggesting submarines can think. The actual point that Dijkstra was
making is that it matters what the machine is capable of, but it doesn't
matter what we call the thing that it does.

If your private definition of intelligence doesn't allow you to apply it to
machines, that's irrelevant.

------
sweeneyrod
This must surely be an instance of "add 'AI' to whatever topic" in order to
get attention, rather than a serious proposal, right? I'm sure we'll see "put
nuclear weapons on blockchain" soon.

------
amai
See also the Russian Perimeter system:
[https://en.wikipedia.org/wiki/Dead_Hand](https://en.wikipedia.org/wiki/Dead_Hand)

------
Havoc
Tried that one already. Works great...if your name is Hollywood and you're
making a action movie

------
fauigerzigerk
Better hope that zero shot learning works. Otherwise it's one shot
annihilation.

~~~
inimino
quote from the actual article:

> Unlike the game of Go, which the current world champion is a supercomputer,
> Alpha Go Zero, that learned through an iterative process, in nuclear
> conflict there is no iterative learning process.

------
mozey
Was the "expert" suggesting this perhaps also same the AI?

------
alfiedotwtf
I want off this planet.

Software engineers: ... Politicians: AI all the things

------
Peej255
Colossus: The Forbin Project (1970) addressed this method.
[https://www.imdb.com/title/tt0064177/](https://www.imdb.com/title/tt0064177/)

------
lwhi
Step 1 .. Connect nuclear weapon to network.

Step 2 .. End civilisation.

------
mschuster91
What the flying fuck? We don't even understand how modern AI algorithms
actually work and how they decide stuff. And we want to give such blackboxes
the power to annihilate all life on Earth?

~~~
jacobwilliamroy
I think the offense industry will read your comment and get all big-headed
about how much stuff they think they can kill. They can definitely kill off
homo sapien, but earth life will go on. There's still a few billion year left
in the sun's main sequence. Maybe something more compassionate will evolve out
of the leftovers.

~~~
klyrs
> Maybe something more compassionate will evolve out of the leftovers.

My money is on squid... and that they'll be no better than us.

------
ohiovr
The experts should go to the movies some time.

------
vincnetas
nice try skynet.

------
JTechno
Terminator hasn't taught them nothing?

------
ekianjo
Is this a modern version of Skynet?

------
dusted
SkyNet? :D

------
dublor
people who think AI can't hurt seems to neglect human stupidity

------
dfawcus
So - Skynet or WOPR?

------
codecrusade
logistic regression based nuclear missiles eh

------
omarforgotpwd
fire them all

------
spsrich2
Come on it's not as if there is any chance something might go wrong after all!

