Hacker News new | past | comments | ask | show | jobs | submit login
The Human Toll of Protecting the Internet from the Worst of Humanity (newyorker.com)
250 points by mathgenius on Jan 29, 2017 | hide | past | favorite | 114 comments

> constant exposure to the worst of humanity on a daily basis takes an undeniable toll.

This is one of the troubles police officers find in their daily work. When they are not part of the society where they work it is hard for them to empathize, as they just see the worst parts of that society.

In the police case also to have members of the police that are part of the communities they serve. But counseling also is needed to be able to deal with a biased view of reality. People don't call the police to show them how nice a neighbor is, or to invite them to a party. Their experience is, like for the guy in the article, to see only the bad parts as the good ones are filtered away.

This is exactly why I wish we'd just hire more police so the existing ones can spend a shift a week doing positive community work.

Helping folks, getting to know the 'normal' folks. This is when I get to the problem: I don't know what exactly I'd ask them to do, but it would be things like: Free (or very low cost) 2-hour windows of child care on the weekends to give parents a stress break, delivering hot food to elderly folks, helping some folks with lawn care or organize to repair some houses (at little to no cost to the dweller). Actual community improvements.

What is more common is to have well-advertised PR events and regulations that they live in the area. Simply living in the area without being exposed to the positive sides is not enough in my opinion.

In some areas (like Sunnyvale CA) you become an Emergency Responder and can alternate from police work to fireman (or EMT) every few years. I heard it described (by someone who was a policeman at the time) as going from hero to villain and back.


I agree. I think it helps them see the people they "serve and protect" as better humans, and themselves as something other than the most righteous thug.

I think that is a wonderful program.

I don't feel that the second paragraph is realistic. But where to start in explaining why?

Cop life is so complex that a weekly home economics shift isn't going to repair relationships, normalize psyches or adjust attitudes.

There's a depth of behavioral psychology to this problem that extends well beyond imagining a one-size-fits all therapy plan.

This can't be approached from the perspective of an outsider's view of what plays well to the public concept of what ordinary people imagine about police.

In theory, police officers (and similar professions, also military) have various privileges, like early retirement and some other benefits (depending on the country).

I'm not sure if there even was a time where these things were enough, but right now they certainly don't outweigh the psychological devastation which almost inevitably follows being on duty for a couple of years.

I don't know the reasons but I'm sure there are many, ranging from cultural shifts in society to particular regulations they use to operate.

Still, the fact remains that the job is nowadays really bad. It's hard for people who never experienced it to understand how bad it is. An analogy which often speaks to programmers is: imagine the worst case of big corporation you can, stuffed with dick-head managers, full of pointless activities which obviously contradict each other, and so on. Now imagine every single manager above you can put you in jail for almost anything - but doesn't have to, if they don't feel like it - and that you are one weird rumor away from being fired (or worse). And that's on top of actual job being shitty, tedious, sometimes pointless, always psychically and emotionally draining.

All this is to say that I agree that "one day a week of social positive work" wouldn't help very much.

As a cure it might not work —existing cops are already biased by their experience. As a vaccine however, it may have a bigger impact.

The bigger problem is to figure out whether the extra expenses are worth it.

It's complex enough that the etiology might well be rooted in our entire culture. There are movements of people trying to approach the problem at that level. Their goal is to destigmatize pain, attack anti-pessimism, etc.

This sounds fascinating but I have no idea what it means. Could you point me someplace where I could learn more about it?

In the 70s when police officers carried a baton and a revolver you'd encounter a lot more talk, diplomacy and second chances. In the age of terrorism and tazers that has become rarer in my experience.

Police officers need to have an 'edge' that is missing in normal people. Normalizing their experience with people and re-humanizing them isn't the best idea for their career and longevity in this day and age. Let them burn out in peace or find a way to cope with their distance from people (usually alcohol, seen that a lot).

I think this hits one something as well. I moved to Norway a few years back - and while the police have access to guns, they don't generally carry them. The public doesn't seem to want it for the most part.

The effect is the one you noticed in the 70s. From a normal citizen's perspective, the fear factor isn't there. You are generally OK so long as you cooporate, even if you tell the police to fuck off while still doing what they say. Some of this is culture: For instance, someone with a bow and quiver on their back walking through the summer palace that is here in town doesn't get harassed. There are fewer guns on the people.

I'm not sure about your outlook, however. I think they need the humanity to be able to treat people well. After all, normal people are the very people that become cops: That "edge" is a learned thing - just like it is in the military. Minimally, I'd go with a few years of training being mandatory before stepping out and becoming a rookie. Things like conflict resolution, dealing with the mentally ill, and interpersonal relations.

Edit: Oh, and stress. I'd include classes on dealing with stress in the training.

Your suggestions are ideal in terms of training and compatibility with society. I just don't think that they are practical (in the US) right now.

The dysfunctional approach between police and citizens seems to be more manageable and cheaper and makes candidates more numerous. My cynicism is showing though.

That was dependent on who you were and the location. A long haired hippy was not getting many second chances in the south. Watch the movie Easy Rider as an example.

I don't need to watch a propaganda film to remember how it was. I was there. How old are you?

That's a pretty good idea IMO. Give cops the same as Google's 20% time, to help with things that bug them (which probably also help the communities they're serving in).

Many do. That's why you have organizations like PAL where police do just this.

You don't know? Here are some easy to implement ideas:


I would dare say nothing of worth being done stop child abuse anywhere in world would compounds the effect of being exposed.

Child advocacy groups are exasperated. The tactics that have been shown to actually help kids aren't sexy enough for law enforcement or politicians. Busting actual abusers is one bust, and its almost always of a parent or other close relative of the victim. Whereas they can get dozens of people trading pictures online in one fell swoop, and the public tends to treat that as just as useful, but research shows it just isn't. Those people aren't the ones abusing kids in the vast majority of cases.

To stop child abuse, the most effective tactics are education and empowerment. Educating kids about the realistic dangers is uncomfortable and decidedly not something people are comfortable with. And empowerment - telling them that their body is their own property and no one, not even their parents, has the right to make them do anything they don't want to - goes against the narrative people want to believe about children being incapable of making those decisions (as if they're difficult).

To stop abuse, you have to realize what abuse is happening. And its abuse by parents first and foremost. A close second is close family members and family friends. Strangers simply aren't the problem. They're a statistical anomaly that is certainly a problem, but one that education does away with quickly and easily. That leaves kids who are afraid of seeing their family destroyed and potentially being tossed into the foster care system. Kids need to know that they will be taken care of, and to be made confident and competent enough to stand up and call out abuse when it happens. It's depressing, slow, intensive work. And law enforcement and politicians are instead too easily convinced to abandon it entirely and just go for the quick busts of extremely dubious benefit to kids.

Education and empowerment would be great, but why should cops be doing it? If DARE has taught us anything, it's that cops are really bad at education. Also it's a fairly privileged point of view that assumes any particular child would want to talk to police about anything. This seems more like a job for teachers and child advocates.

That is, if they can get past the PC "parents first and foremost" BS. First and foremost is stepparents and mom's boyfriends. Actual parents are a distant second, and even then is often a case of "divorced dad lost all custody rights so resorted to kidnapping in order to see children". Concentrating on the wrong thing is just as bad for advocates as it is for police.

You mention children fearing the foster system. I don't think awful foster stories are being traded at kindergarten recess, but it's true that foster care should only be contemplated in the most extreme circumstances. If child advocates are advocating for just turning up the knob on enforcement, and destroying families for such flimsy causes as corporal punishment or parental marijuana enjoyment, they're abandoning that requirement of extremity. In that case, I wonder for whom they're actually advocating, because it sure as hell ain't the children.

> And empowerment - telling them that their body is their own property and no one, not even their parents, has the right to make them do anything they don't want to - goes against the narrative people want to believe about children being incapable of making those decisions (as if they're difficult).

Not only does it go against the "comfortable narrative", it goes against the law. Children are systematically disempowered (often for their own good) until they reach arbitrary magical ages. Minimum driving age, Minimum drinking age, Age of consent, Compulsory school attendance, etc, etc. The idea that children have little agency and few rights is enshrined in law, so it's no wonder they feel powerless when they're abused.

There's a much bigger picture than this. It comes down to governments, media, community leaders, families and individuals. There's a huge problem with apathy toward abuse and it's all around us.

As a case on point, I am extremely disturbed by the way media and governments handle issues like domestic violence. 1800 respect (a government department in Australia) tried to shift the blame onto young boys in a series of heavily promoted TV commercials without any opposition from the community. When a female is caught abusing a family member (child/husband), the media's excuses flow readily "she lost her job", "she was under stress at work". The imagery on TV rarely reflect abused children, despite this being exactly what's happening across the spectrum of society. The Australian Bureau of Statistics says that about 25-33% of domestic violence victims are male (I haven't seen figures for children, but it is likely to be heavily under reported). That's a huge percentage of people who receive little or no representation by all media outlets and the government, let alone charity groups (ie. indigenous people are rarely represented, males are never represented, children are rarely represented - see the imagery as it speaks volumes).

Imagine being a young boy whose mother either mentally or physically abuses you. You turn on the TV and you only see fully grown white women who are (supposedly) the only victims. Politicians (eg. A prominent Queensland politician who should have been dismissed and charged long ago) are legislating for gendered criminal laws because fully grown white women are (supposedly) the only victims and never (supposedly) perpetrators. The people on the TV say how bad males are. The government (1800 respect) saturates TV with domestic violence ads saying "it's a boy thing". You open the newspaper and read the same message almost every week.

Even charity groups chime in. Most charity groups (about 95% of the ones I surveyed, using a public list of every charity that deals with domestic violence) won't even show images of male victims or house male victims. Some charities (like mission Australia) even go as far as to say that "men deny domestic violence happens". Which men? Where? This is almost certainly a ruse to justify almost all of their focus being on fully grown white women (even when Mission Australia say that indigenous women are 24 times more likely to be abused, they still mostly picture _white_ women as victims).

If you really want to talk about abuse and attitudes toward abuse, the problem starts at the top. Governments are prejudiced. The media is morally bankrupt. When somebody knows the government will turn a blind eye and the media will hand out a blank cheque and the courts will say "oh poor woman, she's oppressed/distressed/lost her job", you've got a complete system that's ripe to openly support child abuse.

I find some observations quite fascinating. For example, Steve Altman (Mr Y Combinator) expressed outrage because Muslims can't get on a plane. However, I am yet to see him voice outrage about domestic violence and the appalling attitudes in most media that are open for anyone to see.

The problem doesn't start with parents (although parents contribute to the problem). It starts with apathy.

I'm not under the illusion that this problem of violence and abuse is limited to one gender as that's patently not true. We need society to accept that truth and stop making excuses.

Disqus and social sign-on are the great cancer. Every local news site on the planet that outsources authentication to whoever's got a valid Facebook, Google+, Twitter, etc., is opening themselves to troll farms landsliding both opinions and values out of existence in favor of mindless repeatable bites, "the Sleeper Effect." This is twice as true of Twitter discussions.

Hostile nations are employing legions of both paid users with a posting quota and bots. They are taking advantage of globalism to spread fear and infest rivals with a self-destructive nationalistic ideology which appears to be grassroots and organic.

Just the other day I saw "easyvpn" on here, a one command shell utility to hop VPNs to exit from any desired country. I assume far more anonymous and powerful alternatives exist if you have the cash. This kind of thing paired with "persona management software" is a real problem.

Step back, stop out-sourcing authentication on your websites (even if it means you have to do the time learning some amount of practical cryptography, it's safer than handing the keys off in this climate), and stop conceding massive tracts of digital "space" to a hundred thousand Ru & Romanian twitter accounts claiming to be Joe America tired of liberals while deflecting to "Correct the Record" and "but emails" every time you call them out. Step it up with security, boys, that means some degree of human intelligence. The Internet is growing up and it's going through a pretty nasty phase right now.

Build a wall? How about build a wall around your website so Facebook social sign-on isn't letting Ivan roleplay half of Texas on a popular news story.

I don't think social sign-on is that big of a problem. It certainly makes it easier to comment on news stories, but if you're spending the effort to create thousands of social profiles already, what's a few more e-mail registrations using your Facebook email address?

You could restrict comments to subscribers, but that's anti-social to a degree that I expect is too severe for marketers.

It's a good thing all those Trump supporters were bots, if they had been real he might have got elected! (Oh, wait...)

You misunderstand the impact of social media marketing.

Can we get a collective thank you for the moderators on this website?

It's a hard job with lots of nuance-against-shouting-people, and now has probably been the hardest time to be a moderator in the history of the internet.


I'm constantly delighted (and informed/encouraged/challenged/etc...) by the content available to me through this community, and by the hard work of its moderators.


I think we'll see more stratification on the internet in the coming years with subscriptions, etc.

There's probably even a market right now for a paid version of reddit or HN.

The sad fact is that the internet has gradually become a shithole as the entirety of humanity has joined in.

Twitter, for example, is being dragged down by low quality user accounts that add nothing but noise. It's actively hurting the stock price.

Perhaps it doesn't fit being like HN/Reddit, but Something Awful has been around for a good while, and requires $10 to get access.

At least personally, I feel it worked pretty well... but there was a drop in quality when they introduced probations (a temp ban), meaning breaking the rules stopped hurting your wallet (you could get back in as long as you paid, and didn't do something so bad you got a permaban)

> The sad fact is that the internet has gradually become a shithole as the entirety of humanity has joined in.

In what year do you feel the Internet was better than it is now?

Pre- or beside-Internet, the days of FidoNet; the days of Usenet before its decadence; the days before Web 2.0; the days before business shifted from traditional forms to Internet; the days before smartphones, etc.

There's no clear point, but when it was loosely "reserved" to a certain "elite" (financial, technical, cultural, scientific) which showed genuine interest in it, it was in a much better condition than now. It sounds terrible to say that, but as the masses joined in, the media dumbed down and what it carries dumbed down as well. Another step towards dumbness happened when kids were massively allowed to enter the arena. Again, this sounds harsh, but that's the pitiful reality of humankind. And businesses and capitalism know very well how to encourage the most vile behaviours in order to make more profit.

It also shifted from a confidence & collaboration mode to insecurity & defiance. Now it's "oh my God how this protocol / library is insecure, how could one design such a stupid thing?" Well, I am not sure who is most stupid.

Remember when a 'troll' was just someone who was just pulling a prank teasing a few other people? Well now the same word means a stupid, racist, hateful, mean guy who drowns out a forum or a site with his feces and whose attacks impact real life of its victims.

Which brings me to tell that another big difference is that when Internet was disconnected from real life, it was much better. That was a big turning point too. You used to be able to write everything under your real name and not fear some acquaintance, school mate, employer or any other peeping Tom or enemy would search for it.

So there's not a single year to point out, but during the last 15 (20?) years a series of steps gradually turned Internet in the shithole (and supermarket) it is now. And none of those steps was positive.

The first time I got called a "troll" because someone disagreed with my criticism of a computer program was on FidoNet. That was probably over 20 years ago.

I'm not sure it's got worse intrinsically, but it has certainly got larger, so the badness has scaled along with it.

You're on to something, but even if the 'badness' on an individual basis is the same as 20 years ago, the volume has gone up more than linearly. As more people have joined and been able to find other like-minded folks, they've become emboldened to contribute more. And I don't think this has just happened in 'bad' communities - I've seen it happen in all sorts of communities - people connecting to other folks online has seemed to have this multiplier effect all over.

I think a big part of it is anonymity encourages people to let out the worst aspects of their personality. I think if a forum required people to use their real identities you would see much less objectionable contact (but probably also a lot less interesting ideas)

No, you'd see (as with youtube and G+) people putting out objectionable content under their own names. While genuinely vulnerable people would tend not to get involved for their own safety.

The problem is one of "lawlessness"; adhoc reprisals from your fellow netizens aren't an improvement.

My personal opinion: Before monetization. The Internet I remember was people volunteering their time, because they loved the project that they were working on. If some random blogger wrote an article about something, it was often shit, but it was also sincere. Now that there's money involved, it's less about feeling passionate about something, and more about what brings the most CPM.

This has led to walled-in gardens, where the noise is amplified. I'd even tentatively argue it's because having more users is seen as better, even if those users are fake, trolls, or non-contributers.

1993, the "Eternal September".


1974 or few years after, maybe?

After that a law of large numbers must've kicked in (and I don't think human nature had changed any much in last 3-4 decades).

The fundamental issue up to now is that sites need to make it as easy as possible to join, in order to attract a growing audience, but that invariably means almost complete anonymity, which makes it ruinously expensive or effectively impossible to moderate.

The Something Awful forums have charged 10 dollars for an account (and a few other features) for a long time now, and as a result there are basically no bots and higher quality posts.

It's comedy focused, but there's topical areas too. They remain one of the largest posting boards on the internet.

Isn't reddit already the paid version of reddit?

The key difference is that you can still read/post without ever paying.

Dare I guess that these people generally get paid minimum wage?

> Tech companies don’t like to talk about the details of content moderation, so it’s difficult to judge how well they’re caring for the psychological health of moderators.

The number of such stories which include said companies declining comment (not this one, but anecdotally that's how it seems to be) is kind of a red flag though.

  "Often, the moderators are workers in developing countries, like the Philippines or India"
It looks like those two countries do have a legal minimum wage [1], although it's pretty low - $0.28 for India, $0.61 for the Phillipines. No idea whether those numbers reflect living wages for those countries though, or what a content moderator might actually earn.

[1] http://www.businessinsider.com/a-look-at-minimum-wages-aroun...


Keep in mind cost of living is lower.

The internet is a voluntary vice used by some to earn a living.

It's not food, shelter, clothing AND the internet.

Staying away from the internet is the next health craze.

The internet isn't a vice. Over use of social networks and various applications on it may be. The internet in many forms has become the primary means of human communication.

Communication is a fundamental need of human societies. I view excessive social network usage (to the point of unhealthy) to be analogous with over eating and the obesity it leads to. Underlying both is a basic need. We do better collectively to encourage others to behave responsibly with moderation and enjoyment.

In the extreme we should guard our freedoms, yet penalize harmful abuse.

Future generations will look at the smartphone era the same way we look at the Romans with lead in the water pipes. They too knew it was bad for them but they still did it. So do we. I'm posting from my phone right now.

Maybe. It's also so crucial to so many careers that you need to be in a pretty special place to pull that off constantly, along with all the money it takes to entertain yourself sans internet.

A library card is free in many cities.

Couldn't agree more. It is more like we need to protect humanity from the internet.

Yeah, let only the powerful use it /s

We could put up a giant content firewall, and only allow the well connected to access the "real internet".

Are there any projects working to automate the identification of offensive material with Machine Vision or Machine Learning? Replacing these peoples' jobs with computers might actually be a humane act.

I really wish there was.

I did forensics for a couple years, working with the police force to access and/or recover incriminating evidence.

I really, really wish that a machine could replace a person in that circumstance. And to a certain extent, it is possible.

One of the really unfortunate parts of the process, is to know a video is incriminating, someone at some point has to see that video.

It was part of my job to analyse and catalogue the evidence I recovered, such as saying who was in a video, and what happened to who inside it.

ML will hopefully take over this at some point, but it is difficult for a computer, at this stage, to tell if say, penetration, is involved.

It's moving in that direction, and that is absolutely fantastic.

No person should be required to witness this crap. Hopefully, we can offload it to something that can, one day, be better than us at it.

I'm sorry anyone has to see stuff like that, but there is no way in hell machines can be allowed to start "confirming" such things without several people having to review it.

As the article states, "Although algorithms and artificial intelligence have helped streamline the process of moderation, most technology companies that host user-generated content employ moderators like Soto to screen video, text, and images, to see if they violate company guidelines."

So the algorithms are used to create the list of dubious material. But in the end you need a human being to look at the picture/video and call the police as needed.

The day that AI can do that for us, they will do any other job also.

Yes, there are a lot. For example, NCMEC maintains databases of child porn hashes using photodna and other things.

Machine learning and other algorithms are used to identify images that end up in this database (and others).

Sadly, teaching machines what exploitation looks like is hard and it's not as simple as "identify things that have explicit sexual activity" and then "filter those things by people who appear to be 6 years old".

People still often have to make a final determination because categorization is ... hard

(and honestly, nobody wants to be the guy handling mislabeling, etc)

The cost of false negatives in this area is so painfully high too.

There's this by Yahoo:


> For this reason, the model we describe below focuses only on one type of NSFW content: pornographic images.

It tries to detect a subset of pornography. Others may have different opinions, but personally I wouldn't relate porn to a category of "worst of humanity".

Good point, I guess detecting something as e.g. showing child abuse would be much more difficult.

Most obvious problem is: define offensive. What other people take as offense doesn't look offensive to me at all, and vice versa. There are other problems too.

The only way to fight offenses is to restrict phrasing to Common Universal Neutral Terms. Of course, message consisting only of Common Universal Neutral Term Sequences cannot possibly contain an offense!

Instead of inflicting PTSD on content moderators, why not give the job to someone who actually wants to see this type of content? They must exist, otherwise the content wouldn't spread in the first place.

And/or crowdsource it so that more people are spending less time doing it? Viewing a little bit of this content is probably less traumatic than viewing it for 40 hours a week.

Of course I realize it'd probably be a regulatory minefield, a thorny area morally, and any specific plan would likely draw critics. But it seems a lot better to me than scarring people for life with a full time barrage of things that repulse them.

For instance, why not offer convicted pedophiles jobs which involve screening out child porn? That would save an innocent person from the trauma involved and it might even allow the pedophile to feel like he was redeeming himself to some degree by saving others from viewing that material. From my understanding many people with that fetish feel a lot of remorse about it and they're not exactly trying to get other people into it.

So expose a mentally ill person to more triggers for his mental illness, for something which society despises and condemns him for ? I fail to see how this could ever be a good idea or moral. It's akin to saying lets make serial killers, soldiers. It would be a lot more sensible to have well payed and trained psychiatrists/psychologists do this job, trust me they are trained for, and exposed to far worse things than these moderators see.

Your idea is bad, but needs only a slight modification to work. Instead of giving the moderation task to those who are sensitive to the material, find people who are not so sensitive. Like we can't all tolerate the sight of blood, but some people can deal with it. Doesn't mean they need to be Dexter, but they might have a high tolerance to blood and dead bodies, as medical professionals often do.

Everyone is responsible for monitoring their own job and making sure it isn't affecting their life in a negative way. I think the moderator should have quit or raised the issue. Instead it's the usual story... keeps going and going even after the nightmares and anxiety... keeps working and working... depression, substance abuse or whatever.

As usual, prevention is best. Change jobs.

There are many people who struggled for a long time to find their current job, and feel like it would be impossible to find another now. It would be extra difficult to find the strength to hunt for jobs when your current one is giving you depression and PTSD.

If I recall correctly, one strategy for preventing relapses is to avoid children. Using convicted pedophiles might be counterproductive for some.

It's like a recovering alcoholic taking a job at a liquor store. Not a good plan.

>For instance, why not offer convicted pedophiles jobs which involve screening out child porn? That would save an innocent person from the trauma involved and it might even allow the pedophile to feel like he was redeeming himself to some degree by saving others from viewing that material.

Or just allow the pedophile to indulge in his paraphilia. I'm really not comfortable with this...

At some point it will be possible to create very realistic CP without using children. Actually it might already be possible to do that. One would have thought that would solve most of the problem: create "artificial" CP with strict controls to ensure children are not involved, and let the weirdos watch as much as they want. (Admittedly, there would still probably be some super-weirdos who would be totally turned off by seeing the "Attestation of No Children Involved" message at the beginning of their CP...)

But this comment seems to imply that there would be a problem with that? What is the harm in movies that harm no one?

Because it normalizes that behavior, and may push some to act it out in real life.

I'd be curious to know of that's true. Japan has it everywhere and by everywhere I mean porn manga in every convenience store in the country of which about 3% depicts acts with children. Does Japan have a higher rate of real incidents? I have no idea. If we take violent media normalizes violence afaik we find the opposite. Violent media is up, actual violence is down.

Is this different from other forms of violence ? For example I have been shooting enemies in videogames for more than 20 years, yet I don't feel the urge to shoot anybody in real life.

This would effectively make child pornography "normal" (at least as normal as other extreme fetishes). Many people fear that that would lead to more pedophiles acting on their urges. Then again some think that readily available child porn could reduce cravings and reduce child molestation. I'm not sure if any actual studies exist that shed some light on that problem

Studies are all fairly conclusive: indulging in watching porn related to your paraphilia makes you want to act on that paraphilia. Suggesting otherwise is completely absurd.

Would you care to point us to these studies?

A cursory google about the catharsis myth should do the trick. Not terribly invested in convincing you.

That's fine; you haven't. Googling "catharsis myth" turns up various (unscientific, opinionated) stuff about anger. Perhaps some CP fans are motivated by anger (toward children?), but to generalize from that to all the other motivations for CP is a non sequitur. I'm not normally the "citation needed" dickhead, but you started out by specifically claiming "studies are all fairly conclusive", which seemed like a unique situation for psychiatry so I thought maybe I could learn something...

Freud being wrong about something only tangentially related (what are the chances?) isn't a valid justification for prior restraint. If the work in question hasn't been shown to harm, there would be very little reason to censor it in USA.

In the games industry dealing with abusive online players can get demoralising too. In a lobbby and someone has black sounding voice? Abuse. A woman's voice? Abuse. It becomes an ethical and business issue because these are paying customers on services like Xbox Live and PlayStation Plus. There a ToS and abusive people are a minority but in gaming it can be a large minority. Developers do write algorithms to help deal with these people (won't go into details intentionally) but without outright banning them its not always full proof.

Huh. I'm reminded of that story about the guy who's shown surveillance footage of the middle east, and there's a computer that watches his brain because his brain will react to there being things like people in the images faster than he can be consciously aware of it.

Would it work to use such a system to improve filtering while reducing the human cost? Or would it do something akin to what subliminal advertising is credited with, and just push the damage to the unconscious level?

I don't think such a system would be notably better than standard mashine learning. Reliable image categorisation means you often have to take a second look and interpret context. The internet is filled with images that Intentionally lead to a wrong first impression and can only be recognized when consciously looking at them

I have seen articles similar to this one a number of times. The thing that I wonder about is if there are certain personality types that would be less sensitive than others while still able to remain objective in terms of applying the standards. For example, would a sociopath be unaffected by the content? Just wondering if a psychological screening for the right personality type to do the job would help.

BTW, thank you to the people that do that hard job including the HN screeners.

This is a bit within the realm of my research, so I feel confident about commenting: yes, it's possible. There might be a catch-22 in that the people who would be most resilient are probably those who would be least concerned about the welfare of victims. However, I don't know that's really perfectly accurate: I think you could find some medium, or identify those who empathize but tend not to become easily upset. Empathy and emotional response are kind of dissociable.

Interestingly, the rates of stress disorder in those populations that they cite in the article are kind of in the ballpark of what you see in epidemiological studies of trauma-exposed populations, like combat exposed populations, areas subject to frequent terrorism, natural disaster survivors and so forth. My impression based on what they're seeing is that it's comparable to a job with high trauma exposure rates.

It might be a prime opportunity to study trauma response and treatment, actually, in that often these things have an element of unpredictability to them. If content moderation is something that's reliably characterized by traumatic exposure, you could maybe try to study content moderators to identify predictors of and preventions for stress response.

As always, the risk is simple: deny the expression of human nature at your own peril.

Those who hunt monsters are apt to become one.

What I'm not clear on is why these mods had to look at all at some of these pictures and videos.

There are published hashes of files known to contain child porn and the like.

sha1sum matches? Bit length matches? Contact LEOs with relevant details, delete the post, block the accounts, and move on.

Wired also did a good writeup in 2014: https://www.wired.com/2014/10/content-moderation/

I'm traumatized by the iPhone ui of that site with the ads.. couldn't even make it through the article..

Political censorship cleverly wrapped in a much more palatable package delivered to you for your acceptance.

The original theories of free speech included basically no censorship on the theory that the marketplace of ideas (along with backlash for non-anonymous awful statements) would lead us in the direction of the truth, or at least stability.

It appears that the anonymity of the Internet combined with the echo chamber effect has pretty much completely ruined that.

I don't think it's ruined it at all. I think naive people just like living in a bubble and deplore the truth.

I use the internet and don't often encounter things I didn't expect. Usually what I see on a board, or forum, or sub-reddit etc is the material I expect to see there. Occasionally I see spam or something. Generally the community is probably capable of policing itself on most boards if you give them the tools, downvoting / flagging or something like that.

The crux of the problem is, if the internet remains free then it is also free to "fail". If you can only post lollipops and cat pictures guaranteed not to offend then the internet would contain almost nothing. Comments on such an internet would be so milquetoast as to be not worth reading in the first place.

You can already filter out almost 90% of the content you don't want to see by simply going to certain websites / forums and not others.

Yes, in the same way that not inviting a neo-nazi or an anarcho-syndicalist, or a vegan, to talk on your radio programme is political censorship. Private people and organizations are well within their rights to set their own rules. If you want no moderation that's your choice, go to 4chan or a similar site.

Private people and organizations are well within their rights to set their own rules.

Aren't we tired of saying this by now? That's fine, everyone can express her own views on Facebook, unless one is a Kurd who would like to say something about Kemal Atatürk? Should the Kurds have to create their own Facebook? Who's next, Trump voters? Do we really want them to sink further into their filter bubble?

At some point a service offered to the public must actually serve the public. We don't allow bakeries to choose which weddings they will cater. If you'll bake a wedding cake for $200, then you have to get baking, even if the happy couple with $200 are two dudes or whatever.

We don't allow bakeries to choose which weddings they will cater

I think it's incredibly entitled to want the government to punish private businesses for refusing to do a job. Not everyone has to agree with your ideas or provide you a service. Getting the state involved to literally force a small business to make a cake is an insane, authoritarian over-reaction. The fact that it seems OK to you because you have no problem with gay marriage (neither do I, btw) is really besides the point.

By all means - boycott the bakery. Call them stupid for the bad PR and lack of work they'll get for not making the cake. But coercing them under the threat of force to do a job they don't want to do? Absolutely insane, and I can't agree to it.

I sympathize with that point of view, and I think e.g. discriminatory preachers and wedding photographers are protected by the 1st Amendment (and rightly so). However I find myself fairly indifferent on the topic of wedding cakes (and gay marriage, for that matter). Baking a cake is not a religious act; it is a mundane service provided in exchange for payment. No one has objected to baking e.g. Super Bowl party cakes, even though those don't honor St. Paul the Judgmental Ass either. Lots of other mundane service providers, like mechanics, taxi drivers, grocers, etc. might like to reject the custom of those of particular races or religions, but we have laws against that too. There are well-known reasons for those laws; we don't live in an ahistorical libertopia.

Still, yours is an salutary consistency. My original statement wasn't directed at you but rather at the sort of SV person who "boycotts" Indiana and North Carolina but still talks about the "rights" of a social media service.

I agree with you. I don't know why or how it came to be that diversity is the greatest virtue in Western society and every White Western country is being forced to take in so many that Whites will be a minority. Meanwhile an educated Chinese or Japanese citizen has an incredibly difficult challenge in legally migrating to America or Western Europe. What the fuck is going on here? I don't like bigotry, but this just doesn't seem right.

When you criminalize a certain pattern of thought, it seems that its inverse will eventually get taken to the most extreme position.

That does appear to be the case

Sorry, what? I was talking about baking cakes. Did you mean to reply to someone else?

You don't see the connection? You were really talking about government mandates that: 'if you provide a service or product, you must perform the same service for everyone,' which typically fall under Equal Opportunity and Fair Housing laws(I believe?). I don't see the point in you being disingenuous here? Anyways, those laws protect diversity. Diversity is protected more than free speech. I don't know if that's the right priority. I just hold an opinion that fundamental human rights and liberties should be the primary role of government's use of their monopoly on violence.

>At some point a service offered to the public must actually serve the public. We don't allow bakeries to choose which weddings they will cater. If you'll bake a wedding cake for $200, then you have to get baking, even if the happy couple with $200 are two dudes or whatever.

Excellent point. It is very well worth noting that pragmatically speaking, current law treats diversity > free speech, as is illustrated in your example.

Jessaustin that's a well written expression of that idea, thank you.

[Edit: parent was edited, I was replying to a house example] Not exactly. The comparision doesn't account for scale, and in case of largest platforms (like Facebook) it's more like "leave our country bloc"... except for such comparisions are probably a bad idea, because there are always too many differences that break them.


But parent comment has a point anyway. Political censorship had (ab)used moderation systems in that way, at the very least in some countries. It's not unheard of cases of censorship slipping under disguise of "protecting from bad/unhealthy things". I can think of a few examples from Russia.

Yes, apologies for that, I changed it a few minutes after posting to be more constructive/reasonable, the original comment was flamebait on reflection.

Facebook shouldn't be some kind of universal substrate on the internet anyway. Alternatives should be encouraged, and people can make their own choice about the kind of environment they want to spend their time in.

If you're saying that's impossible, and facebook is some sort of utility, or common carrier, then you can have rights which are enforced upon the space.

Ultimately, either it is a public space (which needs public rights analogous to what exists in the real world, enforced through constitutional or democratic means) or it is a private space, and the owners are free to regulate as they wish, and customers are free to make their own decisions, where to visit and where to spend their time.

> Facebook shouldn't be some kind of universal substrate on the internet anyway. Alternatives should be encouraged, and people can make their own choice about the kind of environment they want to spend their time in.

Wholeheartedly agree that this is the ultimate resolution, but in order to actually get to that point we need a fundamentally different software architecture than the current centralized approach of "the web".

> Private people and organizations are well within their rights to set their own rules

Sure, but that does not imply that it isn't censorship. It's just within their rights to censor.

> If you're saying that's impossible, and facebook is some sort of utility, or common carrier, then you can have rights which are enforced upon the space.

It's not impossible, but it is Facebook's goal to have this level of necessity in people's lives. So, by their own framework, it does make sense to talk about rights on their service. The purpose of this isn't to incite government action of enforcing such rights, but to keep users' and devs' perspectives clear about the lack of freedom on centralized platforms.

There's moderation (bans for being off topic, bating, coordinated websit raiding, NSFW in the wrong place, etc)

But yeah, there's less in the way of censorship. The odd moderator with an agenda still sneaks in, but they seem to get outed.

Label it what you want, I'm glad that steps are taken to get rid of videos of girls being raped and murdered, there are limits to free speech, both legally speaking as well as philosophically speaking as well as in terms of values. Neither the founding fathers nor any of the early proponents intended free speech to be a free for all of any kind of content that some advocate for today.

Sometimes I feel humanity does not deserve all the wonderful things that science and technology provides because people will find a bad use of it. Won't be surprised if someone creates a solar bomb.

The hydrogen bomb could be thought of as a solar bomb. The sun is made almost entirely of hydrogen.

The title is an instance where automatically substituting 'humanity' for 'mankind' leads to a contradiction in terms: humanity is the quality of being humane, or benevolence, and there's nothing bad or malevolent about it.

>humanity is the quality of being humane, or benevolence

That's one definition; my dictionary also lists "the human race; mankind; people."

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact