
The Human Toll of Protecting the Internet from the Worst of Humanity - mathgenius
http://www.newyorker.com/tech/elements/the-human-toll-of-protecting-the-internet-from-the-worst-of-humanity
======
kartan
> constant exposure to the worst of humanity on a daily basis takes an
> undeniable toll.

This is one of the troubles police officers find in their daily work. When
they are not part of the society where they work it is hard for them to
empathize, as they just see the worst parts of that society.

In the police case also to have members of the police that are part of the
communities they serve. But counseling also is needed to be able to deal with
a biased view of reality. People don't call the police to show them how nice a
neighbor is, or to invite them to a party. Their experience is, like for the
guy in the article, to see only the bad parts as the good ones are filtered
away.

~~~
Broken_Hippo
This is exactly why I wish we'd just hire more police so the existing ones can
spend a shift a week doing positive community work.

Helping folks, getting to know the 'normal' folks. This is when I get to the
problem: I don't know what _exactly_ I'd ask them to do, but it would be
things like: Free (or very low cost) 2-hour windows of child care on the
weekends to give parents a stress break, delivering hot food to elderly folks,
helping some folks with lawn care or organize to repair some houses (at little
to no cost to the dweller). Actual community improvements.

What is more common is to have well-advertised PR events and regulations that
they live in the area. Simply living in the area without being exposed to the
positive sides is not enough in my opinion.

~~~
chordatum
I don't feel that the second paragraph is realistic. But where to start in
explaining why?

Cop life is so complex that a weekly home economics shift isn't going to
repair relationships, normalize psyches or adjust attitudes.

There's a depth of behavioral psychology to this problem that extends well
beyond imagining a one-size-fits all therapy plan.

This can't be approached from the perspective of an outsider's view of what
plays well to the public concept of what ordinary people imagine about police.

~~~
smhost
It's complex enough that the etiology might well be rooted in our entire
culture. There are movements of people trying to approach the problem at that
level. Their goal is to destigmatize pain, attack anti-pessimism, etc.

~~~
LanguageGamer
This sounds fascinating but I have no idea what it means. Could you point me
someplace where I could learn more about it?

------
bidoof___
Disqus and social sign-on are the great cancer. Every local news site on the
planet that outsources authentication to whoever's got a valid Facebook,
Google+, Twitter, etc., is opening themselves to troll farms landsliding both
opinions and values out of existence in favor of mindless repeatable bites,
"the Sleeper Effect." This is twice as true of Twitter discussions.

Hostile nations are employing legions of both paid users with a posting quota
and bots. They are taking advantage of globalism to spread fear and infest
rivals with a self-destructive nationalistic ideology which appears to be
grassroots and organic.

Just the other day I saw "easyvpn" on here, a one command shell utility to hop
VPNs to exit from any desired country. I assume far more anonymous and
powerful alternatives exist if you have the cash. This kind of thing paired
with "persona management software" is a real problem.

Step back, stop out-sourcing authentication on your websites (even if it means
you have to do the time learning some amount of practical cryptography, it's
safer than handing the keys off in this climate), and stop conceding massive
tracts of digital "space" to a hundred thousand Ru & Romanian twitter accounts
claiming to be Joe America tired of liberals while deflecting to "Correct the
Record" and "but emails" every time you call them out. Step it up with
security, boys, that means some degree of human intelligence. The Internet is
growing up and it's going through a pretty nasty phase right now.

Build a wall? How about build a wall around your website so Facebook social
sign-on isn't letting Ivan roleplay half of Texas on a popular news story.

~~~
empressplay
It's a good thing all those Trump supporters were bots, if they had been real
he might have got elected! (Oh, wait...)

~~~
bidoof___
You misunderstand the impact of social media marketing.

------
jedanbik
Can we get a collective thank you for the moderators on this website?

It's a hard job with lots of nuance-against-shouting-people, and now has
probably been the hardest time to be a moderator in the history of the
internet.

~~~
mr_spothawk
+1.

I'm constantly delighted (and informed/encouraged/challenged/etc...) by the
content available to me through this community, and by the hard work of its
moderators.

Thanks.

------
_callcc
I think we'll see more stratification on the internet in the coming years with
subscriptions, etc.

There's probably even a market right now for a paid version of reddit or HN.

The sad fact is that the internet has gradually become a shithole as the
entirety of humanity has joined in.

Twitter, for example, is being dragged down by low quality user accounts that
add nothing but noise. It's actively hurting the stock price.

~~~
bitlax
> The sad fact is that the internet has gradually become a shithole as the
> entirety of humanity has joined in.

In what year do you feel the Internet was better than it is now?

~~~
wott
Pre- or beside-Internet, the days of FidoNet; the days of Usenet before its
decadence; the days before Web 2.0; the days before business shifted from
traditional forms to Internet; the days before smartphones, etc.

There's no clear point, but when it was loosely "reserved" to a certain
"elite" (financial, technical, cultural, scientific) which showed genuine
interest in it, it was in a much better condition than now. It sounds terrible
to say that, but as the masses joined in, the media dumbed down and what it
carries dumbed down as well. Another step towards dumbness happened when kids
were massively allowed to enter the arena. Again, this sounds harsh, but
that's the pitiful reality of humankind. And businesses and capitalism know
very well how to encourage the most vile behaviours in order to make more
profit.

It also shifted from a confidence & collaboration mode to insecurity &
defiance. Now it's "oh my God how this protocol / library is insecure, how
could one design such a stupid thing?" Well, I am not sure who is most stupid.

Remember when a 'troll' was just someone who was just pulling a prank teasing
a few other people? Well now the same word means a stupid, racist, hateful,
mean guy who drowns out a forum or a site with his feces and whose attacks
impact real life of its victims.

Which brings me to tell that another big difference is that when Internet was
disconnected from real life, it was much better. That was a big turning point
too. You used to be able to write everything under your real name and not fear
some acquaintance, school mate, employer or any other peeping Tom or enemy
would search for it.

So there's not a single year to point out, but during the last 15 (20?) years
a series of steps gradually turned Internet in the shithole (and supermarket)
it is now. And none of those steps was positive.

~~~
pjc50
The first time I got called a "troll" because someone disagreed with my
criticism of a computer program was on FidoNet. That was probably over 20
years ago.

I'm not sure it's got worse intrinsically, but it has certainly got _larger_ ,
so the badness has scaled along with it.

~~~
mgkimsal
You're on to something, but even if the 'badness' on an individual basis is
the same as 20 years ago, the volume has gone up more than linearly. As more
people have joined and been able to find other like-minded folks, they've
become emboldened to contribute more. And I don't think this has just happened
in 'bad' communities - I've seen it happen in all sorts of communities -
people connecting to other folks online has seemed to have this multiplier
effect all over.

~~~
vivekd
I think a big part of it is anonymity encourages people to let out the worst
aspects of their personality. I think if a forum required people to use their
real identities you would see much less objectionable contact (but probably
also a lot less interesting ideas)

~~~
pjc50
No, you'd see (as with youtube and G+) people putting out objectionable
content under their own names. While genuinely vulnerable people would tend
not to get involved for their own safety.

The problem is one of "lawlessness"; adhoc reprisals from your fellow netizens
aren't an improvement.

------
drinkjuice
Dare I guess that these people generally get paid minimum wage?

> Tech companies don’t like to talk about the details of content moderation,
> so it’s difficult to judge how well they’re caring for the psychological
> health of moderators.

The number of such stories which include said companies declining comment (not
this one, but anecdotally that's how it seems to be) is kind of a red flag
though.

~~~
manarth

      "Often, the moderators are workers in developing countries, like the Philippines or India"
    

It looks like those two countries do have a legal minimum wage [1], although
it's pretty low - $0.28 for India, $0.61 for the Phillipines. No idea whether
those numbers reflect living wages for those countries though, or what a
content moderator might actually earn.

[1] [http://www.businessinsider.com/a-look-at-minimum-wages-
aroun...](http://www.businessinsider.com/a-look-at-minimum-wages-around-the-
world-2013-8?IR=T)

------
losteverything
The internet is a voluntary vice used by some to earn a living.

It's not food, shelter, clothing AND the internet.

Staying away from the internet is the next health craze.

~~~
village-idiot
Maybe. It's also so crucial to so many careers that you need to be in a pretty
special place to pull that off constantly, along with all the money it takes
to entertain yourself sans internet.

~~~
stephancoral
A library card is free in many cities.

------
stephengillie
Are there any projects working to automate the identification of offensive
material with Machine Vision or Machine Learning? Replacing these peoples'
jobs with computers might actually be a humane act.

~~~
shakna
I really wish there was.

I did forensics for a couple years, working with the police force to access
and/or recover incriminating evidence.

I really, really wish that a machine could replace a person in that
circumstance. And to a certain extent, it is possible.

One of the really unfortunate parts of the process, is to know a video is
incriminating, someone at some point has to see that video.

It was part of my job to analyse and catalogue the evidence I recovered, such
as saying who was in a video, and what happened to who inside it.

ML will hopefully take over this at some point, but it is difficult for a
computer, at this stage, to tell if say, penetration, is involved.

It's moving in that direction, and that is absolutely fantastic.

No person should be required to witness this crap. Hopefully, we can offload
it to something that can, one day, be better than us at it.

~~~
jasonkostempski
I'm sorry anyone has to see stuff like that, but there is no way in hell
machines can be allowed to start "confirming" such things without several
people having to review it.

------
apatters
Instead of inflicting PTSD on content moderators, why not give the job to
someone who actually wants to see this type of content? They must exist,
otherwise the content wouldn't spread in the first place.

And/or crowdsource it so that more people are spending less time doing it?
Viewing a little bit of this content is probably less traumatic than viewing
it for 40 hours a week.

Of course I realize it'd probably be a regulatory minefield, a thorny area
morally, and any specific plan would likely draw critics. But it seems a lot
better to me than scarring people for life with a full time barrage of things
that repulse them.

For instance, why not offer convicted pedophiles jobs which involve screening
out child porn? That would save an innocent person from the trauma involved
and it might even allow the pedophile to feel like he was redeeming himself to
some degree by saving others from viewing that material. From my understanding
many people with that fetish feel a lot of remorse about it and they're not
exactly trying to get other people into it.

~~~
exodust
Your idea is bad, but needs only a slight modification to work. Instead of
giving the moderation task to those who are sensitive to the material, find
people who are not so sensitive. Like we can't all tolerate the sight of
blood, but some people can deal with it. Doesn't mean they need to be Dexter,
but they might have a high tolerance to blood and dead bodies, as medical
professionals often do.

Everyone is responsible for monitoring their own job and making sure it isn't
affecting their life in a negative way. I think the moderator should have quit
or raised the issue. Instead it's the usual story... keeps going and going
even after the nightmares and anxiety... keeps working and working...
depression, substance abuse or whatever.

As usual, prevention is best. Change jobs.

~~~
bittercynic
There are many people who struggled for a long time to find their current job,
and feel like it would be impossible to find another now. It would be extra
difficult to find the strength to hunt for jobs when your current one is
giving you depression and PTSD.

------
ajeet_dhaliwal
In the games industry dealing with abusive online players can get demoralising
too. In a lobbby and someone has black sounding voice? Abuse. A woman's voice?
Abuse. It becomes an ethical and business issue because these are paying
customers on services like Xbox Live and PlayStation Plus. There a ToS and
abusive people are a minority but in gaming it can be a large minority.
Developers do write algorithms to help deal with these people (won't go into
details intentionally) but without outright banning them its not always full
proof.

------
RangerScience
Huh. I'm reminded of that story about the guy who's shown surveillance footage
of the middle east, and there's a computer that watches his brain because his
brain will react to there being things like people in the images faster than
he can be consciously aware of it.

Would it work to use such a system to improve filtering while reducing the
human cost? Or would it do something akin to what subliminal advertising is
credited with, and just push the damage to the unconscious level?

~~~
halomru
I don't think such a system would be notably better than standard mashine
learning. Reliable image categorisation means you often have to take a second
look and interpret context. The internet is filled with images that
Intentionally lead to a wrong first impression and can only be recognized when
consciously looking at them

------
zw123456
I have seen articles similar to this one a number of times. The thing that I
wonder about is if there are certain personality types that would be less
sensitive than others while still able to remain objective in terms of
applying the standards. For example, would a sociopath be unaffected by the
content? Just wondering if a psychological screening for the right personality
type to do the job would help.

BTW, thank you to the people that do that hard job including the HN screeners.

~~~
kem
This is a bit within the realm of my research, so I feel confident about
commenting: yes, it's possible. There might be a catch-22 in that the people
who would be most resilient are probably those who would be least concerned
about the welfare of victims. However, I don't know that's really perfectly
accurate: I think you could find some medium, or identify those who empathize
but tend not to become easily upset. Empathy and emotional response are kind
of dissociable.

Interestingly, the rates of stress disorder in those populations that they
cite in the article are kind of in the ballpark of what you see in
epidemiological studies of trauma-exposed populations, like combat exposed
populations, areas subject to frequent terrorism, natural disaster survivors
and so forth. My impression based on what they're seeing is that it's
comparable to a job with high trauma exposure rates.

It might be a prime opportunity to study trauma response and treatment,
actually, in that often these things have an element of unpredictability to
them. If content moderation is something that's reliably characterized by
traumatic exposure, you could maybe try to study content moderators to
identify predictors of and preventions for stress response.

------
EekSnakePond
As always, the risk is simple: deny the expression of human nature at your own
peril.

Those who hunt monsters are apt to become one.

------
ocschwar
What I'm not clear on is why these mods had to look at all at some of these
pictures and videos.

There are published hashes of files known to contain child porn and the like.

sha1sum matches? Bit length matches? Contact LEOs with relevant details,
delete the post, block the accounts, and move on.

------
manigandham
Wired also did a good writeup in 2014: [https://www.wired.com/2014/10/content-
moderation/](https://www.wired.com/2014/10/content-moderation/)

------
uptownfunk
I'm traumatized by the iPhone ui of that site with the ads.. couldn't even
make it through the article..

------
wallace_f
Political censorship cleverly wrapped in a much more palatable package
delivered to you for your acceptance.

~~~
Brakenshire
Yes, in the same way that not inviting a neo-nazi or an anarcho-syndicalist,
or a vegan, to talk on your radio programme is political censorship. Private
people and organizations are well within their rights to set their own rules.
If you want no moderation that's your choice, go to 4chan or a similar site.

~~~
jessaustin
_Private people and organizations are well within their rights to set their
own rules._

Aren't we tired of saying this by now? That's fine, everyone can express her
own views on Facebook, unless one is a Kurd who would like to say something
about Kemal Atatürk? Should the Kurds have to create their own Facebook? Who's
next, Trump voters? Do we really want them to sink further into their filter
bubble?

At some point a service offered to the public must actually serve the public.
We don't allow bakeries to choose which weddings they will cater. If you'll
bake a wedding cake for $200, then you have to get baking, even if the happy
couple with $200 are two dudes or whatever.

~~~
lacampbell
_We don 't allow bakeries to choose which weddings they will cater_

I think it's incredibly entitled to want the government to punish private
businesses for refusing to do a job. Not everyone has to agree with your ideas
or provide you a service. Getting the state involved to literally force a
small business to make a cake is an insane, authoritarian over-reaction. The
fact that it seems OK to you because you have no problem with gay marriage
(neither do I, btw) is really besides the point.

By all means - boycott the bakery. Call them stupid for the bad PR and lack of
work they'll get for not making the cake. But coercing them under the threat
of force to do a job they don't want to do? Absolutely insane, and I can't
agree to it.

~~~
wallace_f
I agree with you. I don't know why or how it came to be that diversity is the
greatest virtue in Western society and every White Western country is being
forced to take in so many that Whites will be a minority. Meanwhile an
educated Chinese or Japanese citizen has an incredibly difficult challenge in
legally migrating to America or Western Europe. What the fuck is going on
here? I don't like bigotry, but this just doesn't seem right.

~~~
up_so_floating
When you criminalize a certain pattern of thought, it seems that its inverse
will eventually get taken to the most extreme position.

~~~
wallace_f
That does appear to be the case

------
sidcool
Sometimes I feel humanity does not deserve all the wonderful things that
science and technology provides because people will find a bad use of it.
Won't be surprised if someone creates a solar bomb.

~~~
20after4
The hydrogen bomb could be thought of as a solar bomb. The sun is made almost
entirely of hydrogen.

------
wtbob
The title is an instance where automatically substituting 'humanity' for
'mankind' leads to a contradiction in terms: humanity is the quality of being
humane, or benevolence, and there's nothing bad or malevolent about it.

~~~
edmccard
>humanity is the quality of being humane, or benevolence

That's one definition; my dictionary also lists "the human race; mankind;
people."

