Yeah, this argument is interesting. There are good moral rebuttals. Someone on HN said recently, “your actions remain immoral even if others are ready to be just as immoral as you are.” I haven’t heard a good practical rebuttal though :/
W.r.t. nukes, for example, there is a good chance that mutually assured destruction prevented war despite how terrifying the policy is.
It's also worth considering if the act of inventing something horrifying is actually immoral, or if using it for immoral purposes is.
For instance, dynamite. We would not have the society we do now without dynamite. The fact that it can also be used to destroy property and kill people makes it a deadly invention, but was it immoral to invent?
Of course dynamite is nothing compared to the Atomic Bomb.
Or maybe inventing it for the purposes of killing people makes it immoral regardless of other possible applications.
How much does intent matter, versus outcome?
And then, does it absolve the inventor if they had pure intentions but others used their inventions to do harm?
Is it all moot because trying to live a moral life when others are willing to be immoral will basically never work out in your favor?
I think a good starting point for this discussion is clarifying the implicit false dichotomy: it's either moral or moral, technology is either good or bad.
To quote Melvin Kranzberg "Technology is neither good nor bad; nor is it neutral".
Just because technology requires people to create it and to use it, does not mean that it has no impact on us (an by extension it also means that it's creation has impacts).
To take an obvious example, guns are not inherently good or bad. But they unlock the possibility of shooting people.
If I invent something that can be used for good, but others use it for bad, does it reflect on my character?
Does my good intentions absolve me of the harm my invention did?
Or, the opposite? If I invent something fully intending to do harm with it, but others find non-harmful uses for it, does that absolve me of my original intentions to cause harm?
A gun may not be good or bad, but the intention of the person who invented guns likely were good or bad.
This takes us back down the road of binary outcomes though.
The intentions of the inventor are one thing to examine. The outcomes of the inventions another. And the notion that something is “moral”, may not have answers in practice, even if there might be an answer in principle. Even if those answers could be found, they could only be found after enough time had passed to judge the outcome fairly. But it is unpalatable to retroactively assign moral judgement for actions the originator could not have possibly predicted (there’s a different version of this thought experiment that imagines we could perfectly predict the future, but that seems less useful here).
I do think that the moral peril of certain actions/decisions is proportional to the potential risk. There are few discoveries as dangerous as that of splitting atoms, and the more obvious the danger, the more reasonable it seems to judge the original act of invention.
When questions like “will we explode the universe?” seem reasonable to ask, that seems like a good sign an invention is not something that should exist in an ideal world.
Most inventions don’t have these kinds of stakes, and the effects of their use are local to the actors using them, at which point it seems more reasonable to judge the actions of individual users than the invention/inventor.
(1) if one day a nuke was used to deflect an asteroid and save the entire planet, would that then outweigh any deaths caused by nukes and ottenheimer would be a hero rather than a villain? That's a tricky road to go down.
(2) as is trying to focus on intent. Do you want to risk absolving Hitler or Pol Pot because they thought they were making the world a better place?
We could also consider failure to prevent bad actors from doing immoral things as immoral itself. The problem with this line of thinking is there is no tight definition of immorality.
Regardless of how the sentence continues, there is no such thing. There is making inventions in the world as it currently is, and what follows from that.
> People have a series of rationalizations. People say for example that science and technology have their own logic, that they are in fact autonomous. This particular rationalization is profoundly false. It is not true that science marches on in defiance of human will, independent of human will, that just is not the case. But it is comfortable, as I said: it leads to the position that "if I don't do it, someone else will."
> Of course if one takes that as an ethical principle then obviously it can serve as a license to do anything at all. "People will be murdered; if I don't do it, someone else will." "Women will be raped; if I don't do it, someone else will." That is just a license for violence.
> Other people say, and I think this is a widely used rationalization, that fundamentally the tools we work on are "mere" tools; This means that whether they get use for good or evil depends on the person who ultimately buys them and so on.
> There's nothing bad about working in computer vision, for example. Computer vision may very well some day be used to heal people who would otherwise die. Of course, it could also be used to guide missiles, cruise missiles for example, to their destination, and all that. You see, the technology itself is neutral and value-free and it just depends how one uses it. And besides -- consistent with that -- we can't know, we scientists cannot know how it is going to be used. So therefore we have no responsibility.
> Well, that is false. It is true that a computer, for example, can be used for good or evil. It is true that a helicopter can be used as a gunship and it can also be used to rescue people from a mountain pass. And if the question arises of how a specific device is going to be used, in what I call an abstract ideal society, then one might very well say one cannot know.
> But we live in a concrete society, [and] with concrete social and historical circumstances and political realities in this society, it is perfectly obvious that when something like a computer is invented, then it is going to be adopted will be for military purposes. It follows from the concrete realities in which we live, it does not follow from pure logic. But we're not living in an abstract society, we're living in the society in which we in fact live.
> If you look at the enormous fruits of human genius that mankind has developed in the last 50 years, atomic energy and rocketry and flying to the moon and coherent light, and it goes on and on and on -- and then it turns out that every one of these triumphs is used primarily in military terms. So it is not reasonable for a scientist or technologist to insist that he or she does not know -- or cannot know -- how it is going to be used.
> Regardless of how the sentence continues, there is no such thing.
Maybe you misunderstood what I was trying to say?
I was not talking about the inventions themselves but the intent of the inventor
Sometimes a person (or organization) sits down and says "I want to accomplish <goal> so I am going to put resources towards finding a way to do that"
<Goal> can be anything from "a new way of preserving apples" to "a way to kill a whole bunch of people at once"
If it turns out that your new way of preserving apples also can be weaponized, is that really your fault for inventing it? That doesn't seem right to me.
Otherwise the only moral action is to not invent anything? Absurd.
> I was not talking about the inventions themselves but the intent of the inventor
I know. And I say that "there is no such thing" in the sense of it having a meaningful impact on the outside world. It's solely something for the inventor in private to be happy or sad about. There isn't even a point in telling the world their intention. That's like telling the desert to be green and thinking you did something.
> If it turns out that your new way of preserving apples also can be weaponized, is that really your fault for inventing it?
Yes. Partly anyway, of course the main fault is always with people "pulling the trigger". For the reasons laid out above. Weizenbaum couldn't possibly spell it out more clearly, and your response was basically giving an example for what intent is. We know, the response is that it's a fallacy. To just repeat the original thing doesn't change the response to it.
> I haven’t heard a good practical rebuttal though :/
The practical rebuttal to people justifying their deeds with sophistry a teenager can see through is to stop them, regardless of whether they like it or not, and not getting too tangled up in the chaff they came up with to rationalize what they are doing.
A teenager probably would tell that ChatGPT helps her do homework and acts as a Pokémon. And she doesn’t understand why it’s compared to a bomb that was dropped onto Hiroshima (by the military) to stop the aggressor that had attacked the United States.
>there is a good chance that mutually assured destruction prevented war despite how terrifying the policy is.
If I bet at the Casino using a martingale system I'll usually leave the Casino with more money than I came with. Doesn't mean that the Martingale system is a winning strategy or that I'm actually beating the house.
MAD certainly prevents war, but that doesn't prove it's worth it.
> W.r.t. nukes, for example, there is a good chance that mutually assured destruction prevented war despite how terrifying the policy is.
But Oppenheimer didn't spy for the Soviet Union. He worked to make the US the sole nuclear power, not to create a nuclear stand-off between opposing powers.
I’m certainly no expert on this, but even then surely this was seen as an inevitability? The US scientists knew Russians and Germans were approaching this, did they think their opponents would stop developing the capability just because they got there first? Seems unlikely that they would think that.
There is none for nukes as it involves the zero sum game of war. Because you'll die if you don't. Russia and Japan were already developing nukes independent of the US, as the theory for nuclear
fission was well known. Logic says to fire immediately if you opponent can't, as eventually you'll lose advantage, a scenario that scared the Soviets to whits end, despite the US not doing so. So you either make your defense or be at the whim of your opponent. Being moral and dead isn't a very good outcome is it?
Overall, the history of nukes lends credence to the idea of keeping doors open, to naively and hopefully trying to avoid zero sum games. The US decided against Kyoto to keep Japan postwar. It's willing to go fire the nukes in defense of Japan today. The US didn't annihilate the Soviets post WW2, despite offense. Khrushchev talked an idiot out of nuclear war and a Soviet operator said the sensors were wrong. But it's only something a single side can control, and it's not assured that you won't end up with a cruel or zealous antagonist, who at best will pause to thoughts of their own death, or you need to make pay to have any hope afterwards.
So you managed to rationalize not building a nuke through your superior morals. Now inevitably there will be a different group that operate on different moral principles that will eventually develop a nuke. (Unless you somehow think they are too stupid and not capable)
I guess have fun being right and on the side with no nukes?
I suspect that when German scientists reported that it was infeasible to create a fission chain reaction they knew. Knew that there were ways and knew not to turn their brilliance towards the creation of terrible weapons.
The last step in creation is relinquishing control. And while it is possible to use weapons wisely, an inventor must realize that once a thing is created he has no say in whether it’s used wisely.
For this reason I caution you all against the creation of any weapon you wouldn’t wish to face in the darkness of night, be it in the receiving end or as the responsible party.
Don’t create superhuman targeting devices. Don’t create targeted germ warfare. Don’t create antimatter devices. Don’t use AI to kill.
You are responsible for your own actions and your actions matter. Let us begin there.
Not really, because becoming a rapist won't reduce the number of rapes committed by others. On the other hand, inventions can really only be invented once, and successfully inventing or commercialising something _will_ usually reduce the profitability for other companies entering the space (because they'll have to compete, or they don't have the patents, etc).
I think that his statement highlights a misunderstanding of the capabilities of ML in the near future. It's an honest mistake since all the startups chasing pumps in investment are selling the idea of terrifying power.
A more apt parallel, in my opinion, is that of solar geoengineering. We are doing what Oppenheimer wish we did: ignore its existence. However, the allure, now as it was then, is that it's too little effort for too much power to ignore. It isn't an existential imperative to the global North so it has been dismissed. Make no mistake if climate change posed a direct and immediate existential risk to nations of the global North we would have stopped global warming already. Instead we are waiting for the collision of "nations that can afford solar geoengineering" and "global warming posing a direct and immediate existential risk" in 2-5 decades and doing none of the legwork to avoid unintended consequences because the notion of doing the science is not popular politically.
Fortunately we are all pretty far away from an existential war. That's the only thing keeping us alive after Oppenheimer's grand success. Even if a nation crumbles the things that replace it need to at least offer the trappings of a legitimate state if they don't wan to be immediately crushed by nuclear superpowers (assuming they themselves are a nuclear superpower or adjacent to one).
So the cutting edge isn't what's physically most effective, but what's most effective while also being within the gray and constantly shifting "acceptable" range. It's nice that we don't have to go all out, but as soon as scarcity or existential threats enter the picture things can get very ugly quickly. Best to steer clear of such things if possible. I consider chemical, biological, and weather weapons all in the same class of "you better not if you don't want to start a world war".
I think of the film The Bridge on the River Kwai as being a great cautionary tale for engineers, the myopic pursuit and pride in solving engineering problems at the expense of one's larger mission.
Had the atom bomb never been built we'd be on WW5 by now, not much of a 'cautionary' tale. I for one have learned to stop worrying and love the bomb, I'll take the current situation over rotting in a foxhole any day
It turns out you have to abstract put the gun to the head of humanity form them to realize that pulling the trigger is a bad idea.
It's sad, but its very true. I also stopped worrying and love the bomb, and I'm willing to justify the US nuclear bomb usages on many grounds, namely that even after they were dropped, Japan almost did a military coup against their own emperor to keep the war going. https://en.wikipedia.org/wiki/Ky%C5%ABj%C5%8D_incident
Japan surrendered to the US to not be invaded by the Soviets, who had just called off their truce, swept the Kwantung army in Manchuria, and were on the verge of doing the same through Japan's undefended north. Stalin would have treated the emperor far differently from MacArthur, and the best case scenario for Japan had it not surrendered to the US prior to that invasion would have us talking about the possibility of reunification of North Japan and South Japan today.
It's both actions, the dropping of the atomic bombs and the the Soviet invasion of Manchuria. The Soviets, like the US, would have needed to prepare for a land invasion of Japan, probably about 9 months to 1 year of so of prep, which would be behind a US invasion, but it helped close in the walls. The Soviets had agreed to the war with Japan, but another view is to look at it as a quick spoils at the end of the war and to help mao. See the yalta negotiations. Still it was brought up in their surrender discussions.
You can crawl through the memories/literature to see what weighed on their minds more, but it's the bombings. Japan knew what nukes were as they were developing themselves. And as pointed out by the poster, the coup attempt happened after both the Soviet involvement and Hiroshima, August 12th, so the point still stands, that their was a willingness to fight to annihilation, which would include a Soviet invasion. But your right to point out that Japan considered the better offer that stood at that pivot moment, rather than the triple scenario of more bombings, a US land invasion, and even later Soviet invasion.
Be aware, overtly strong emphasis of the Soviet invasion and de-emphasis of the role in the bombings is often a tactic to dismiss the usage of the atomic bomb, that's not borne out by what we know of the internal dialogue of Japan's surrender. And on the outside and without hindsight, to both the US and soviets, it looked very much that Japan could/would fight to the death, and this reflects in the decision making of those few weeks. Still, perhaps the Soviet invasion would have been enough, but we will never know. And it would have not worked if the coup had worked.
It's also worth repeated yet again as many seem to have either forgotton or in many cases never knew that when the atomic bombs were dropped on japan it was literally " just " two more cities after 72 cities had already been bombed with a similar degree of destruction, death, and injury stats.
The firebombing of Tokyo was on the same order as H & N.
Cost wise conventional bombing was (at that time) far far cheaper and posed a similar risk to US air forces as there was little to no credible fight back against carpet air bombing at the stage.
I address more in the other comment. I was more trying to head off the oliver stone, "it was exclusively the Soviets", which was the original post. The article is okay, but spends too much time on the mental faculties of those involved.
No, I have not forgot the firebombings. They should have weighed the firebombings as the same. But they didn't. Some of it was that it was novel and a single bomb. New capabilities seem to leave an impression, such as V2's for the ability to hit a target without warning, even though they were cost ineffective. But, they were not grasping their predicament.
But take the Soviet case to it's conclusion, and that the bombs had no impact on whether they thought the war should be continued. They could have waited more if it was just the soviets, as all the stated Soviet plans do not involve Honshu. You likely had a couple months at the minimum, even if the soviets rushed. And it's not like I hear any worries of communism in anything, Japan had a pact with them and tried to use them as a third party. So why not try to negotiate openly with the US/GB/China/Soviets then? Half their cabinet thought to try even after everything. No, hirohito, thought the US was going to bomb them to bits in the interim, whether by firebomb or atom bomb, and the offer was only going to get worse. So they ended it.
This isn't to say that maybe the atom bombs were needed to come to that position, as the previous bombings might have been enough. Or maybe a few weeks of waiting would have been enough. I ponder if their had been no bombings of any kind, per the rules of war, would Japan have surrendered given the Soviet involvement. I would guess no, considering the margin given and the longevity of such an event creating a way to rationalize out, but still something to think about.
Even Japan's surrender didn't stop the Soviet invasion, with the Soviets taking the Kuril islands and Sakhalin after the Japanese had surrendered. A planned invasion of Hokkaido on August 24 had to be scrapped after the Japanese surrender only because of the terms of the Potsdam Declaration.
Ugh. Should have left that part out, and now, I spent too much time trying to give you a good response. ;)
Japan was hedging it's bets at the top, giving the option to explore a soviet mediated conditional surrender, ie. the maintenance of their post ww1 gains, maintenance of government, etc. They, Togo and Suzuki, had been considering it for a while. However, it wasn't well recieved by the mil, who thought it at best a tactic to garner an ally and to delay for time to make a defense or to get a new opening. And that is key, as it's the military had the majority of spots and overall control of decisions, only breaking after the initial bombing and soviet involvement (to a degree the soviet involvement might have been more shocking for the mil, as it wasn't known if the US had anymore bombs). And a military coup to continue the war was a chief concern, so much so that any discussion of it was done in secret. This is also likely the reason for their screwups on not responding to/inability to negotiate, as once it was made known, a coup could happen. The disconnect between the cabinet levels pragmatism and the lowers zeal didn't help any outside impression, and is one reason they had both a botched coup and yet an orderly surrender.
The soviets disregarded the request and invaded soon after it was made. This removed Japan's possibility of using the Soviets as a negotiating party/leverage with US/GB/China. This likely means a combination of soviet views/reasons:
a. the soviets thought their unconditional surrender was imminent and needed to grab land quickly
b. They didn't think a unconditional surrender was imminent but needed to get on with their agreement with the others (They had to go to war on August 9th as that was the yalta agreement, 3 months after may 9th, and it's not like the sped up thinking they were close to surrender and needed to get in quickly)
c. That the mullings of conditional surrender was a delay tactic to regroup/dig in and so should be ignored
d. Some combination of US/GB/China would or could not entertain a conditional surrender and thus it was moot, potsdam for instance
e. Japan was just looking for a new partner to help with US/GB/China and it would be not kosher to entertain it as they already had Yalta/disagreements in Europe/fights over the germans surrender
f. Japan was weak and offered nothing, especially consider russo Japanese history
But consensus is just what's being carried out, not the state of all opinions. It's the reason I included "could". For example, the US may have thought that it was a strong possibility they may not surrender to any reactions, but tried anyway by dropping leaflets and more. Japans offers for a soviet negotiated conditional surrender were known to the other allies as they were decoded. Now, you could go down the path of if the US/GB/China should have gone the route of entertaining a conditional surrender to avoid continuing. But conditional surrenders can drag on for years, the war continuing all the while, and used to delay for an opening or a peacetime regrouping. It's why at all the conferences they insisted on unconditional.
Some the cynical attitude towards it was that japan wanted to play the cost game near the end, where they held that they'd fight vigorously unless they got good terms or something changed. And it's not a terrible plan mind you, as for example, the US has shown weakness to such costs in subsequent years, though for less popular wars. Or that Great Britain for example, despite being alone and under imminent invasion, saw the entrance of the US and the Soviet Union into the war. Some of this is also likely influenced by the potsdam terms, indicated the allies in the pacific did not seek Japans complete destruction. Some of this was to make surrender look attractive but it also betrays a weak stomach (despite the firebombings indicating otherwise). It only appears that threat turned into a real thought post atomic bombings (per their comments in the meetings about the bombs).
It's betrays the delusion of were they thought the war stood, that the atomic bombs meant anything changed or had even considered the USSR as a negotiator. They had already been firebombed, and thus looked at possible annihilation without the atomic bombs, they had no allies, the soviets had given warning that they were not on their side by not renewing the non aggression pact in april though they did not weigh it like they should have. Sorta the last desperate and illogical actions of a flailing government. It's one reason why the whole episode is confounding, especially later, as a "rational" actor would have surrendered earlier.
Per your point on the soviet invasions.
The soviets got bogged down on both landings quickly and were helped by the surrender of a large portion of the japanese forces starting on the 15th and still taking till well after the surrender to take sakhalin and kuril. This likely would have happened to hakkido, if they weren't repulsed (doubtful they'd be fully dislodged, but overextended supplies are bad when getting bogged down). I'd have to dive more into the planning on hakkido to see the state of it, as Sakhalin/kuril are often seen as quick risky land grabs as they were launched after the bombings/invasion of Manchuria, and surrender on the 15th. Still the Soviet style of pressing might have made some gains, though it would have been limited to Hokkido. For your original post, it depends on if Japan thought Soviet occupation of Hakkido as a breaking point to take an unconditional surrender, (Okinawa comes to mind as a failed breaking point, though it did seem to set some stuff in motion), but I doubt it as, as it's always been sorta a backwater for Japan.
Kinda odd phrasing from Nolan there. "The rise of companies over the last 15 years bandying about words like ‘algorithm,’ not knowing what they mean in any kind of meaningful, mathematical sense."
That's a description of scriptwriters and journalists. Tech companies know well what they mean when they say algorithm, especially what they mean mathematically. It's the non-tech class that talks about algorithms as if they are magical spells of mind control which is why they think that "Meta’s contribution to genocide in Myanmar" is a real thing and why they are shocked that Threads, being a social network, uses a "purely algorithmic feed" (emphasis in the article).
"as AI systems go into the defense infrastructure, ultimately they’ll be charged with nuclear weapons"
The logic of MAD is simple that's why it's a three letter acronym. Nukes could be put under automated control for decades now and AI wouldn't change anything about what those control systems would do. Detect a pile of nukes heading your way from Russia/China -> fire back. No fancy schmancy AI needed for that!
“When I talk to the leading researchers in the field of AI they literally refer to this right now as their Oppenheimer moment,” Nolan said. “They’re looking to his story to say what are the responsibilities for scientists developing new technologies that may have unintended consequences.”
So much more exciting than the banality of the truth: neural networks are useful algorithms but not nuclear weapons. Not even close.
I agree. It doesn't make Nolan look good. But he's probably just trying to generate publicity.
I suspect that facebook's contribution to Myanmar was more as a place of organization and evidence of opponents, than the feeds somehow generated most of the zeitgeist to slaughter. It kinda ignores the other causes of zeitgeist/organization and absolves the villains of the affair in doing so. But the idea latched on to some journalist.
The Soviet Union had a semi/automated system for firing if Moscow got blown up before making a decision.
They're is an air of vanity to the media and researchers who compare AI to nukes. It's sorta like Nolan making a film about Oppenheimer, in the hope to associate with them.
Why a cautionary tale? The manhattan project was a succees. Does anyone doubt that if a similar war broke out nukes won't be used?
Just look at the number of civilians the japanese killed (nanjing massacare is arguably worse than the holocaust) and they would have kept doing that. Assuming all lives are equal, nukes saved a crapton of lives.
I think it is hypocritical to say cluster and fire bombs that have high rates od civilian casualty and even chemical weapons were ok but nukes somehow are not.
Moreover, can you imagine the world we would live in without nukes? A whole lot of countries would be fighting a whole lot of wars were it not for the fear of starting a nuclear war. Even now, without nukes, do you think there is a chance the US would be attacking Russia over Ukraine? Or if there is a war with China, do you think there would be more or less casualities if there was no nukes? Short of multipe nukes hitting multiple cities on both sides continually, a drawn out conventional war would kill more people, especially with other WMDs still in play. Cholrine gas was used on civilians as early as WW1.
You gotta understand, in a major war like WW2 where losing means nazis and imperial japanese invade the US homeland, most people have no problem nuking their cities (self included). Youe choice is between your children and family or theirs. This isn't like vietnam/iraq/afghanistan which are wars of ideology and interference not wars of self defense. Also keep in mind that most japanese would have fought tooth and nail (civilians included) against an american invasion. Many killed themselves and their own families simply because their land was invaded and they felt dishonored.
I think Black Mirror is neither a deterrent nor a guide, and 1984 is more on the "deterrent" side. A ton of policy debates tend to get shut down almost immediately whenever someone says "this is 1984". Our level of surveillance is way, way, way below what our technology allows (yet still above what is desirable in a free society), and I suspect 1984's existence is a major reason for that fact.
It depends on your country of residence. In certain authoritarian countries, 1984 was definitely used as a guide on what to implement in terms of facial recognition, scanning of device data, always on recording in public, etc.
Your comment made me wonder if there's an alternate history work of fiction where the atom bomb wasn't built.
Something like Man in the High Castle for WW2 victory or For All Mankind for the Space Race.
That would be a very interesting divergence to explore, with very interesting ramifications for the Cold War, especially if nuclear weapons do get invented later on, but without two bloody live-fire demonstrations to shock the world.
