I've thought that it's hard to appreciate how much large tech companies (and communications and payment intermediaries) are in a terrible double bind about many content and policy issues.
If they enforce some community's or culture's norms, or proactively try to avoid enabling any violations of some jurisdiction's laws, people will find things about that that outrage them, including misunderstandings, or ways that those norms aren't shared by others, or ways that they can be oppressive.
If they don't do that, other people will get outraged about how the companies are allowing or facilitating "bad stuff" which they obviously could have chosen not to allow or facilitate to the same extent.
But there might be no actual possible course of action which would avoid both forms of outrage!
And, at a large scale, false positive rates and false negative rates are usually in a trade-off, even when people supposedly agree on what is supposed to happen. (Which they also don't actually agree on.) That's especially true when companies are using software and statistics to help them make judgments, or when they're not spending billions of dollars to have decisions made by domain experts, at leisure, with rights of appeal.
Sure, business is hard. When doing business you have to spend money and labor solving problems for customers. This is one of the problems you have to solve when you store people's data for them.
We don't need to "appreciate it" any more than we need to appreciate how hard it is to get gasoline to the pump, I appreciate that by buying the gas, I appreciate Google's service by buying Workspace (or if you use Docs for free, which you probably shouldn't, granting them a license to use and resell some of your data).
Now specifically the problem with Google is that they don't want to do all the work involved in their line of business. They invest very little time in doing effective content moderation and support with humans and that's why you get ridiculous events like this one. Their model, and this is the key thing to understand, is to be large enough that they can cut costs and deliver worse service but you stay with them anyway because switching is hard and you don't have a lot of alternatives.
This is pretty much the textbook definition of monopoly-style market power abuse, and it's essentially illegal, so another way of putting this is Google's business strategy is to be large enough that they can violate the law without consequences.
They are a mafia.
These types of people are always saying "data is the new oil," well that means Google is the new Standard Oil. It needs to be broken up otherwise it will continue to violate the law willfully at scale.
Not really; pretty much any service that will allow contacting random people will have the shit abused out of it.
And to be clear: what happened is that she was prevented from sending more share links after sending 80 or so of them, with "spicy" content. She wasn't "locked out". Title is wrong. This is basic spam filtering, which does have the occasional false positive.
> But there might be no actual possible course of action which would avoid both forms of outrage!
> And, at a large scale, false positive rates and false negative rates are usually in a trade-off, even when people supposedly agree on what is supposed to happen. (Which they also don't actually agree on.) That's especially true when companies are using software and statistics to help them make judgments, or when they're not spending billions of dollars to have decisions made by domain experts, at leisure, with rights of appeal.
There is a course of action that leads to far better results; human review. But that cuts into the $billions of quarterly profits, so we don't want that.
The idea that it's unaffordable to these companies is a complete fabrication. It should be forced upon them. If that halves their profit, so be it, it just means their margins have been too high due to offloading negative externalities unto society.
Having built tools to organize and drive human curation, human review only means that you get different errors and different people get fired as scapegoats. When you're doing content moderation, rather than simple data curation, you don't even get "far" better results - you frequently get worse results! - and you start traumatizing your moderators enough to destroy them. On top of that, who the fuck would ever use a product if their private notepad was going to be read by a human moderator? There is no easy answer here.
The very least you could do is have human review of these cases where accounts are shut down, have an actual human appeal system. You can't convince me that this wouldn't significantly decrease the false positive rate (without significantly increasing false negatives), learning to better outcomes. Obviously when appealing you'd consent to having your content reviewed by a human.
> On top of that, who the fuck would ever use a product if their private notepad was going to be read by a human moderator? There is no easy answer here.
Besides, we've gotten to a point where it's not clear if A. every single thing being ingested as data to create a profile of me off (current situation, e.g. Gmail) is better than B. Only a portion that has been flagged being reviewed by a human.
There's a lot of low-hanging fruit which humans-in-the-loop would be used for to give far better results. A very obvious example being Meta's key role in the Rohingya genocide. Plenty of outsiders banged on Facebook's door, but they wouldn't listen, costing thousands of people their lives. There's lots of such "obvious" cases. Content moderation is extremely difficult when it comes to things that toe the line, sure, but I wasn't really talking about those.
> But there might be no actual possible course of action which would avoid both forms of outrage!
The only possible course of action (which upends may a business plans/models) is to not host the data in the first place; not even encrypted.
Offer software that stores stuff where users choose to (local drive, NAS, private server, s3 bucket/backblaze/etc although that's getting close to "hosting data")... How computing used to be - and should hopefully one day become, again... Assuming we veer off this dystopian timeline.
That won't protect from outrage, as inevitably something really bad will be exposed to be on your service. You'll have to either defend keeping your position or creep up surveillance on user content to avoid a similar thing to happen.
Users, regulators, investors etc. all want different things that are mostly incompatible, and you're bound to piss some and please others to varying degrees.
> other people will get outraged about how the companies are allowing or facilitating "bad stuff" which they obviously could have chosen not to allow or facilitate to the same extent.
They won't get outraged, they'll just not use it if they feel unsafe. That means less ad money, which is death for a monopolistic ad company, that's all. No outrage required.
There was this old site called voat.co, a reddit clone with almost no restrictions to what you could post. IIRC, only child porn and bestiality was off limits. It was located in Switzerland.
Soon the site filled with nazi stuff among other things, so the author in fear of getting in trouble, relocated the site to a more free-speech country. Where? To the USA.
A free speech country is not the same as a country where some companies have restrictive policies. You cannot post a nipple in Instagram and then cry "muh freedom of speech". Use one of the endless sites to post such things or roll your own: the government won't stop you.
The problem is specifically that the government won't stop you. American free speech principles have been warped and distorted to give platforms total control and no responsibility over what they host. Hell, back during the last wave of Net Neutrality regulation, Comcast was arguing in court that common carrier regulations were a free speech violation. Which is absolutely insane.
Additionally, we should stop measuring the size of a country's free speech commitment by how legally easy it is to host Nazi bullshit. Let us not forget that Nazis hate freedom of speech and use all sorts of extralegal bullshit to stifle it. A country or platform in which it is legal or permitted to express Nazi political opinions is one where you have less freedom of speech, not more.
Freedom of speech is a balancing act in which we tie the hands of people in power to allow those out of power to speak their mind.
I live in a country with laws against certain hate speech forms. All we really got is not less hate (we have plenty of nazi bullshit), but comedians, musicians, journalists... heavily fined, exiled or in jail.
A woman almost two years to jail for dark joking about a guy killed in 1975: https://es.m.wikipedia.org/wiki/Caso_Cassandra. Later revoked in the Supreme Court. The jokes were all well known, I've been hearing them since the 1980's.
There's even a lawyer firm that only exists to sue people for offending christian beliefs (https://abogadoscristianos.es/), the modernized version of the spanish inquisition you didn't expect.
If you live in the USA be careful if you want to limit freedom of expression, because you will end with a tool to bully people. You dislike a newspaper? Sue them and hope it goes well. Nazis are at least gone? I'm afraid, but no: they are stronger than ever in the last 50 years.
> But there might be no actual possible course of action which would avoid both forms of outrage!
Is weird to think outrage is normal ?
There's no pleasing of every single human on earth, and any company with a large enough audience needs to balance their position and have priorities. Once they made a choice, it's natural they get the scrutiny for the consequences, and if they're not happy with the tradeoff they're free to adjust and piss off another group of people.
If the goal is to give them a pass because choices are hard, I'd say no. They should deal with the outrage.
Cynically I've come to the conclusion that the only proper solution to pleasing everyone is to start telling every "activist" to just go fuck themselves, irrespective of the contents of their demands because the perpetual outrage cycle is just that bad. Sure they'll get mad but they have no attention span and will move on to more pliant entities when they don't get any more stimulation.
If they enforce some community's or culture's norms, or proactively try to avoid enabling any violations of some jurisdiction's laws, people will find things about that that outrage them, including misunderstandings, or ways that those norms aren't shared by others, or ways that they can be oppressive.
If they don't do that, other people will get outraged about how the companies are allowing or facilitating "bad stuff" which they obviously could have chosen not to allow or facilitate to the same extent.
But there might be no actual possible course of action which would avoid both forms of outrage!
And, at a large scale, false positive rates and false negative rates are usually in a trade-off, even when people supposedly agree on what is supposed to happen. (Which they also don't actually agree on.) That's especially true when companies are using software and statistics to help them make judgments, or when they're not spending billions of dollars to have decisions made by domain experts, at leisure, with rights of appeal.