
Ask HN: Are there real-world examples of moral failures of software engineers? - objectiveariel
Consider the following projects:<p>* Facebook&#x27;s Free Basics initiative in India:
http:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;technology-35169226<p>* The Stingray phone surveillance device: 
https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Stingray_phone_tracker<p>* North Korea&#x27;s Red Star operating system: 
https:&#x2F;&#x2F;media.ccc.de&#x2F;v&#x2F;32c3-7174-lifting_the_fog_on_red_star_os<p>* Hacking Team&#x27;s surveillance software (sold to countries with a poor human rights track record): 
https:&#x2F;&#x2F;theintercept.com&#x2F;2015&#x2F;07&#x2F;07&#x2F;leaked-documents-confirm-hacking-team-sells-spyware-repressive-countries&#x2F;<p>Do you think that the software engineers who consented to work on the above projects acted ethically?
======
jerf
Errr, define "ethically".

I'm not trying to trap you. I'm willing to work with whatever definition you
specify, and I won't try to play semantic games with the definition if it's at
least close enough to something specific to work with. I'm not asking for a
universality claim. But without some specification of what you mean the
question is vague to the point of unanswerability.

The North Korean programmers may well have truly believed in what they were
doing. A utilitarian may well truly believe that even if Facebook Free Basics
isn't a perfect program, it's a net good for the participants. The
"surveillance software" can be seen as just a tool and whether the tool makers
are responsible for its misuse is ethically debatable. (And let me be clear I
mean that literally, not as an attempt to rhetorically state a position. I
could write a coherent argument both ways.) After all, many people even in
free countries end up calling for strict regulation of corporations and that
same "surveillance software" is pretty much the way you instantiate such
regulation, so, is it _really_ clear that it's intrinsically evil?

And again let me emphasize the point I'm making here is just the width of
possible arguments about ethics that can be made. My previous paragraph is
itself ethically incoherent, inasmuch I'm not even trying to take a consistent
stand overall, but merely trying to highlight the most obvious problem per
issue where ill-defined "ethics" makes it hard to even debate the matter.

~~~
RodericDay
Why? Defining ethics is extremely difficult, but volunteering our intuitions
via examples helps us get closer and closer to narrowing what it is exactly
that we're talking about.

It's mighty hard to pin down a universal definition for "art", "love", and
even "game"\- and yet we use these words regularly and mostly very
successfully to communicate.

Discussing "ethics" is difficult too, but I am unconvinced by arguments of the
"it's obviously all subjective" variety.

------
SideburnsOfDoom
How about the engineers who wrote this code?
[https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal](https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal)

I can't put all the blame on the software engineers involved: the decision to
do this surely lies higher up the chain. But did anyone of the coders object
or blow the whistle beforehand? Or were they all fine with "just following
orders"?

Now that more is known, have any come forward or even given an anonymous
interview about it afterwards? AFAIK,no.

~~~
bakhy
good point. although this actually crosses the line from "just unethical" to
straight up "illegal", which makes the question perhaps less interesting.
developers who wrote that code are accessories in crime. and the defense of
"just following orders" might only be considered for soldiers, not for
civilian workers.

~~~
pc86
It's typically not even considered and excuse for soldiers, is it? Just
because someone may be immune from civil prosecution does not mean there are
not equally (or more) severe consequences in the military justice system.

~~~
ni-hil
I find it hard to believe you could be punished in the military system if you
followed your superior orders.

Edit: my bad, in fact you can, if the order is illegal, and it has been
unsuccessfully used as a legal defense in hundreds of cases. But, considering
what you risk when you don't follow orders, it seems like a tought decision to
take. If you don't follow orders, you are done and will probably be punished.
And if you follow and you get caught it's on you and you will have to assume
full responsability...

Here is more info on this subject:
[http://usmilitary.about.com/cs/militarylaw1/a/obeyingorders....](http://usmilitary.about.com/cs/militarylaw1/a/obeyingorders.htm)

~~~
SideburnsOfDoom
I used the phrase "just following orders" knowing that it referred to military
history.

You don't have to work on the basis of _belief_ , there's a lot of history on
this topic:
[https://en.wikipedia.org/wiki/Superior_orders](https://en.wikipedia.org/wiki/Superior_orders)

------
ryandrake
I've always thought we could benefit from having a Hippocratic Oath for
software developers and taking it seriously. The tough part would be agreeing
what kinds of projects are and are not ethical to work on.

I personally have quit a job in the past because (among sever other important
reasons) I felt the projects' primary application (surveillance) was not
something I wanted to be associated with. I have also chosen not to
voluntarily participate in the patent process for any software I've designed
(foregoing those slimy patent "bonuses" other engineers seem to like to gobble
up). But not every software developer shares my particular set of ethics. How
would you come up with a definitive list of what does and does not violate the
"Software Hippocratic Oath"?

~~~
gorbachev
My Comp.Sci. degree actually included one. All graduating students swore an
oath to not use our skills for evil (paraphrased). It's been while since that
happened, so I don't remember the specifics any more, but I do remember
thinking what a great thing to pass onto new software engineers.

------
bcg1
Its nearly impossible to answer your question, because the examples you give
all have an element of "its bigger than one person".

In all of those cases, it it is conceivable that programmers could easily
justify the "larger purpose" in their own moral/ethical framework. This a
problem inherent in all collective efforts.

 _" There's always the same amount of good luck and bad luck in the world. If
one person doesn't get the bad luck, somebody else will have to get it in
their place. There's always the same amount of good and evil, too. We can't
eradicate evil, we can only evict it, force it to move across town. And when
evil moves, some good always goes with it. But we can never alter the ratio of
good to evil. All we can do is keep things stirred up so neither good nor evil
solidifies. That's when things get scary. Life is like a stew, you have to
stir it frequently, or all the scum rises to the top."_[1]

[1]: [http://www.goodreads.com/quotes/12020-there-s-always-the-
sam...](http://www.goodreads.com/quotes/12020-there-s-always-the-same-amount-
of-good-luck-and-bad)

------
Cyph0n
How about the extensive surveillance software developed by the NSA (PRISM et.
al.)? That's probably the biggest example of ethical and moral failure in
modern history.

~~~
edanm
I think that most people disagree with you about that. So it's hard to call
the people working on that immoral.

~~~
Cyph0n
I call them immoral because I think the work they are doing simply is immoral.
Nothing "hard" about that.

------
hendzen
For an older example, see [0].

[0] -
[https://en.wikipedia.org/wiki/IBM_and_the_Holocaust](https://en.wikipedia.org/wiki/IBM_and_the_Holocaust)

------
ethbro
There's a lot of ethical failure in failed implementations.

I realize these kind of projects are big consulting firms' bread and butter,
but if you as an engineer continue to work on a project that's being managed
with an eye towards "Failure is okay, because our lawyers wrote the contract
to cover that contingency" then that's pretty scummy from an ethical
perspective.

[https://en.wikipedia.org/wiki/Virtual_Case_File](https://en.wikipedia.org/wiki/Virtual_Case_File)

 _Edit_ : On the flipside, I feel a lot more comfortable when employers I've
worked for have sat down with customers and had the "Look, this just isn't
working out. We recommend you cancel this project and we tie off our
relationship, because it's not going to end well for either of us if we
continue" talk.

------
pmoriarty
I'd add:

\- Anyone who works for or supports militaries who wage aggressive wars or
military actions. Arguably, depending on your stance on pacifism, anyone who
works for or supports the military at all.

\- Anyone who supports torture or "intensive interrogation".

\- Anyone who helps to imprison people for victimless crimes, or otherwise
supports the injustices of the legal system.

\- Anyone who works in or supports unsolicited advertising which tries to
manipulate people in to buying junk they don't need, often lying in the
process.

\- Anyone who works to support the security, surveillance, or police state
aparatus.

Given certain viewpoints these are ethically indefensible. From other
viewpoints these can be not only ethical, but righteous. A lot depends on
where you're coming from and what your politics are.

Unfortunately, I don't think most people care or think much about the ethical
implications of their work or what they do. And if they do think about it,
they usually just throw up their hands and say there's nothing they can do
about it, or everyone else is doing it and if they don't do it somebody else
will, or that they got to make a living somehow, or that at least they're not
doing something even worse, or that they were just following orders. The
excuses, even in the rare cases when people recognize there might be an
ethical problem with their actions, are legion.

------
dreen
It is subjective, but you can argue that any device ever made with intention
to kill or harm people had people involved who knew what they were doing and
created it anyway. Everything from Gas chambers in concentration camps (made
supposedly by Siemens or Bosch), bombs, guns, even swords.

And then there is Java EE...

------
ultramancool
Why focus on these iffy things rather than say - ransomware, someone basically
wrote a nw.js based ransomware as a service tool even. Are these people not
professional enough or something?

------
logfromblammo
It's impossible to say without knowing more about the alternative courses of
action available to the individuals involved.

You can't fault someone for making a morally wrong choice if they never had a
morally right option to choose.

They might also genuinely believe--perhaps due to misinformation--that they
are doing good rather than evil.

Besides that, I believe that it is acceptable to perform an action for the
benefit of a mutually loyal relationship at the expense of anonymous
strangers.

It may be that someone who has a moral objection to IMSI catchers would,
instead of endangering himself for the sake of strangers, anonymously pass a
tip to a defense attorney containing a suggestion for discovery. Would that
person then be condemned as unethical for supporting the technology, or
expiated by peeling away one of the veils of secrecy? There is never any
shortage of skilled workers who can be psychologically manipulated into acting
against their own interests, after all. If the morally conflicted person never
worked there, the secrecy may have remained intact longer, creating a larger
window for potential abuse.

It's very difficult to condemn someone for trying to collect a regular wage.
"I was only following orders" is not exculpatory, but if the alternative to
following the order is sufficiently dire, it is sufficient to discourage me
from adding moral condemnation to any of the other consequences that may
result.

------
OliverJones
I know somebody who knows somebody :-) who did the following:

In a mad scramble to create some impromptu how-to instructions for a
customer's users, the person made screenshots that revealed a bit of personal
and confidential information.

The person doing that knew it was wrong at the time. This took place in a
dodgy ethical milieu, where the customer's management refused to support the
publication of high-quality how-to materials, and the customer's users were
failing and flailing, and themselves emailing screenshots showing private
data.

In my belief, this kind of breach of ethics is much more common than colossal
systematic moral collapse. The trouble is this: It contributes to a morally
slippery workplace.

------
alexandrerond
The engineers working on those projects may grasp that there are problematic
sides to the wider-scope of what they do, but they are not the ones running
the show and they are just trying to do their job. The ones running the show
usually have very good compelling reasons to keep running it (national
security, avoid company crisis and keep jobs for many families.. etc) even
when things are illegal or borderline, and usually can diffuse responsibility
in a larger board, subordinates etc...

In short, most people won't perceive their job as unethical at all. Those who
see the problems, won't relate them to the tiny bit they work on but to the
whole that doesn't depend on them (and are probably right).

And those who get enough remorse to stop doing their tiny bit, will be
replaced by someone else who will happily continue it anyway.

So I don't blame the engineers working on those systems despite the results
being so immoral.

Appeals to ethics are not very useful. Otherwise most of the Tech giants of
this world, not only the NSA, would have an awful time finding someone to work
for them.

------
dsacco
There are software engineers who follow some definition of ethics, and there
are software engineers who ignore some definition of ethics - just like every
other profession in the world.

What do you hope to achieve by finding real world examples? People do Bad
Things for some definition of bad all the time, no matter their profession,
but good luck trying to make any claim to absolute wrongdoing without the
discussion devolving into semantics.

It will be very difficult for you to ascribe a moral position to any real
world example because the real world isn't binary. To a first approximation,
debates about ethics are usually won by vindicating the majority in-group's
opinion about something they disagree with.

As you can tell, I'm basically saying this discussion isn't productive. You'll
probably get some trendy answers like "NSA" or something to do with
"surveillance" but I'll bite - how about the blackhat hackers who are employed
by organized crime to de-anonymize "problem people" so those people can be
found and "dealt with"? But that's just my opinion.

One of the biggest problems with this sort of thing is that no matter how
powerfully you might believe someone working at e.g. the NSA is doing
something evil, that individual likely feels just as powerfully that they are
working to increase the net good in the world. In fact, they probably have an
objectively coherent argument in favor of their position.

------
gorbachev
[https://en.wikipedia.org/wiki/Room_641A](https://en.wikipedia.org/wiki/Room_641A)

Well, not really about SOFTWARE engineers, I suppose.

------
davismwfl
My question is slightly different. Is it the writing of the software that is
ethically at issue, or is it how the software is utilized?

Are the scientists that develop new drugs which have potential side affects,
even death, murders, are they ethically challenged, morally corrupt? Some
people died using their invention, some could have even have been
intentionally killed or accidentally killed. Rogue governments could use the
drugs to hurt or abuse people. I'd say no, the scientists (like engineers)
aren't the problem, they did something that served some valid purpose but
their invention (creativity) can be abused or misused.

Stingray devices, while I totally disagree with how they are used, I can see
that validity for their use for some specific law enforcement cases. The
problem to me isn't the technology, but the lack of ethics and morales in the
people using them. Lying to the judges about the usage, lying about it's
capabilities etc.

Hacking Team's surveillance software, having read just a little about it in
the past. It appeared again that the software was being used appropriately for
a genuine and valid purpose, at least at first. However, then it was sold to
people who planned to mis-use it and to people who have a track record of
abusing human rights. So is it the software's creation that is the problem or
the assholes that sold it to dictators and abusers?

A syringe isn't an evil device, it is necessary for the medical community to
do its' job, but it is also a device that can be used to kill, commit suicide,
overdose on drugs etc. Is the person/people who invented it morally
corrupt/ethically challenged, or does their invention just have a potential to
be abused?

~~~
ponderatul
Well if you are knowingly creating software that you can reasonably expect
someone to use it unethically then you are responsible.

You might say that you are not directly responsible and that there are second
or third order effects. Or you might even say every man has his own judgement,
I cannot be made ethically responsible for his actions with the software I
created.

Sure, no you are not responsible, but you were an enabler of his behaviour,
you brought that man/company whatever one step closer.

To answer your question, I think you don't need to create separation between
the two, they are intertwined. The software would be less likely to have been
created if you didn't write it in the first place and you probably wouldn't
have written it if someone wasn't going to use it.

So as a software engineer I think it's reasonable to expect to try to
understand who is it that you are creating the software for, how are they
planning to use it, who is selling it to whom, what kind of people they are
etc.

------
bakhy
although i would personally consider all of the above unethical to some
degree, i think i could understand someone thinking differently about Facebook
Free Basics, Stingray, and Red Star. someone might put more weight to giving
the poor any "internet", than giving them proper, neutral internet. Stingrays
are used by legitimate law enforcement agencies, although i don't know much
about how and under what conditions. and Red Star, well, all of that is legal
there and not complying with the Kim can cost NK citizens their heads, so i'd
cut a lot of slack there.

but investing your work in a company which sells surveillance equipment to
dictators is downright disgusting. people are being tortured and killed
because their governments managed to track them down thanks to that company.
that seems like complete moral ambivalence at work. even if one could have
somehow rationalized enabling their own government to spy, upon finding out
who the company sells to, they must quit. and perhaps also leak everything.

------
s_kilk
Potentially debatable, but:

Google outs transgender woman:
[http://www.theinquirer.net/inquirer/news/2321446/transgender...](http://www.theinquirer.net/inquirer/news/2321446/transgender-
woman-is-outed-by-google-hangouts)

~~~
pc86
Certainly debatable. Unintended consequences do not "moral failures" make.

~~~
s_kilk
One could argue that neglecting to take care when handling users identities is
a moral failing.

~~~
pc86
Hence being debatable :)

------
burner_1000
I heard a rumour (well people were talking in abstract terms about "the secret
project" so I'd say 90%+ true) that a certain french owned software
consultancy was building systems for the Saudi government to monitor
homosexuals and journalists. Mass surveillance FTW!

------
Lawtonfogle
Anyone who worked to keep Tor secure from the US government.

Anyone who worked to aid the US government to circumvent Tor.

The problem with ethical actions is that it depends upon one's framework. Now
there are areas we can tend to all agree on. Taking money to sabotage software
to kill someone we agree is innocent is probably a good example. But once you
get past the clear cut examples, it begins to be clear that ethical behavior
is subjective and our subjective views do not always agree. For example, I can
find people who would agree strongly with one of the above two that I listed
while disagreeing strongly with the other, and I can find people who would do
the opposite.

------
l3m0ndr0p
IBM & nazis: [http://www.nytimes.com/books/first/b/black-
ibm.html](http://www.nytimes.com/books/first/b/black-ibm.html)

------
SuperKlaus
How about anybody and everybody working on surveillance software for the NSA?

------
stewartbutler
I would add Zynga, FanDuel, Draft Kings.

All three prey on people with addictive personality.

------
threesixandnine
Why would software engineers be any different than other people? As in any
field you will find the whole spectrum of humans...

------
cju
There has been a conference by Bertrand Meyer on the topic but I don't know if
there is a video of it : [https://bertrandmeyer.com/2015/10/12/danke-
sehr/](https://bertrandmeyer.com/2015/10/12/danke-sehr/)

------
tetraodonpuffer
don't you think that, especially in a large organization, the work can be
partitioned in such a way that nobody knows they are working on something
objectionable?

There are cases where it is obvious and people will quit over it, but if you
are given the task, say, to change the firmware on this particular device to
execute these external calls for conditions A, B, C, how do you know if this
particular check is used for something you don't agree with?

What if somebody had you implement a bunch of logging and management
interfaces telling you it's for QA when instead it is for mass surveillance?
how are you going to know?

I think the ethics come more into play after the fact, if you find out that
your company has done something you don't agree with (and you might have taken
part in it without knowing), are you going to quit over it or not?

------
lectrick
Zynga

And I would personally argue for Microsoft's
[https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish](https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish)
program

------
bobby_9x
What about people that work at knife manufacturers? A portion of their product
will be used to kill/hurt someone.

Most engineers build something for a paycheck and the company paying them ends
up using it for something unethical or illegal.

------
taveras
I've always felt a little strange about TEMPEST:

[https://en.wikipedia.org/wiki/Tempest_(codename)](https://en.wikipedia.org/wiki/Tempest_\(codename\))

------
jmkni
Relevant Defcon talk - My life as a spyware developer

[https://www.youtube.com/watch?v=k2mdUcOXW6I](https://www.youtube.com/watch?v=k2mdUcOXW6I)

------
ryanmcdonough
[http://www.wired.com/2010/09/google-
spy/](http://www.wired.com/2010/09/google-spy/) like him?

------
lfender6445
slightly related, but check out consequences of an insightful algorithm -
which speaks to the unintentional ethical failures we make as technology
continues to progress and become more powerful.

[https://www.youtube.com/watch?v=NheE6udjfGI](https://www.youtube.com/watch?v=NheE6udjfGI)

------
lwyr
Nobody mentions software freedom? [0] Arguably any software that restricts
user freedom is the result of a moral failure somewhere in the process.

[0]
[https://en.wikipedia.org/wiki/Free_software](https://en.wikipedia.org/wiki/Free_software)

------
jimmaswell
Various applications refusing to allow you to run them as root

------
Grue3
Let the one without sin cast the first stone.

~~~
AnimalMuppet
That means don't condemn others. It does _not_ mean "don't bother cleaning up
your own act". It doesn't even mean "don't make moral judgments about what
other people do".

~~~
Grue3
No, that means everybody is a "sinner". It can be argued that anyone who
writes closed-source software is unethical (see Stallman et al). But open
source programmers are themselves unethical, being privileged enough to give
away their work for free which drives down salaries of less fortunate coders.
There's simply no programmer who can be described as completely ethical, and
your helloworld.c is the Original Sin.

~~~
icebraining
Why is lowering the salaries of other coders unethical?

------
J_Darnley
No, no, yes (I assume people from NK wrote it), no.

Add to that list everyone who works for a social network company or who works
for an ad company.

~~~
drdeca
What about social networks that aren't ad supported ? ( e.g. donation funded )

So, do you think providing a social networking site is inherently unethical,
or just information selling advert stuff?

I ask because I've been kinda working on an alternative to a particular one,
and if you think that is unethical, I'd like to know why, in case your reasons
why are convincing.

(because I don't want to do something unethical, unless not doing it would be
morally worse than doing it.)

~~~
J_Darnley
These other social networks ("they really exist, trust me") probably don't try
to actively spy on the entire world and follow every person across the entire
internet. They probably don't try to hoard content into their silo and try to
keep the users locked in. Clearly I am describing the big three here.

Now a not ad funded network may not do these things. That removes my largest
complaints about them. It is these things which make them so grossly
unethical.

As for your project I wish you luck. You will need it.

------
liviu
Like Patrick Naughton (Java creator) beeing a pedophile? But instead of
13-year-old girl it was an FBI agent.

[https://en.wikipedia.org/wiki/Patrick_Naughton#Sex_crime_arr...](https://en.wikipedia.org/wiki/Patrick_Naughton#Sex_crime_arrest_and_conviction)

