
Content moderation at scale is impossible to do well (2019) - dredmorbius
https://www.techdirt.com/articles/20191111/23032743367/masnicks-impossibility-theorem-content-moderation-scale-is-impossible-to-do-well.shtml
======
sova
Score Voting notably defies Arrow's Paradox.

Score Voting (= Range Voting) is where candidates are given a score from, say
zero to 99, and an average is taken. Effectively the same style as in figure
skating. Yes, a once-every-four-years global aesthetic athletic event is the
only place to get voting right.*

*what do you mean by "right?" Well, there is something called Bayesian regret and you can minimize it by having Score Voting (=Range Voting) or by having the magical harry potter wizard hat choose the perfect candidate. Other voting systems have more Bayesian Regret and the current First-Past-the-Post voting style has quite a lot, actually. [https://rangevoting.org/BayRegsFig.html](https://rangevoting.org/BayRegsFig.html)

~~~
ulucs
Any voting system that incorporates mean scores will in no way be immune to
strategic deviations. (imagine a tie between your second ranked candidate and
some other one, you simply inflate your score of your second ranked candidate
by one) Also isn't achieving the lowest regret trivial, as the other voting
systems never report anything cardinally?

~~~
sova
In regards to your first statement, that's absolutely the point of Score
Voting, to have a clear leaderboard with 1st, 2nd, 3rd and so forth. With
regards to your second statement, "because we cannot access data means the
data never existed?" Bayesian regret exists in all elections with all
candidates, just because it's not "reported" doesn't mean it's not
statistically there. The point is that Score Voting is a superset of all
voting systems, and everything else is a blurrying of the detail that is
available only in Score Voting. Bayesian Regret is just used to demonstrate
that it will yield the highest rated candidate overall, which is logical given
how it works. Nevertheless, examine the science for yourself and I believe you
will soon find that all voting systems are some sort of blind or ignorant form
of Score Voting. Means or averages are convenient, but you could simply take
the greatest total score as well.

------
auganov
Obviously right on the numbers but I have to disagree with the notion that
reporters and people are just making a lot of noise over the 0.1% of
inevitable "bad calls".

Usually only notable popular accounts get coverage over their censorship (or
lack of). The number of such accounts is a few orders of magnitude smaller
than the general userbase. Any network would want to protect these accounts
from arbitrary bans and they probably do.

Most big stories about social media censorship often include comments from
company reps who stand by their decision. Stories ending in "Oops we made a
mistake" tend to be much less common as far as I can tell.

As such I'd say people are actually pretty good at identifying the cases that
aren't just a "bad call" but a contentious matter.

------
chr1
Centralized moderation at scale is impossible to do for the same reason that
centralized planning of economy is impossible to do: which is that goals,
knowledge and resources are not known to the centralized authority that is
supposed to handle the task.

But it is possible to give everyone tools to modereate for themeselves
[https://adecentralizedworld.com/2020/06/a-trust-and-
moderati...](https://adecentralizedworld.com/2020/06/a-trust-and-moderation-
system-for-the-decentralized-web/)

~~~
xg15
> _The first obvious issue is the creation of echo chambers or online cults,
> where instead of choosing truth and seeking the best information, people
> trust those who confirm their existing beliefs.

This can be somewhat mitigated with the ability to set a trust filter level to
a low level, even in the negatives, to see posts and opinions one may not
usually be exposed to. This again is up to the end user if they wish to expose
themselves to differing opinions or not.

It is up to individuals to determine what’s best for them. Of course, one
would hope they make good choices, but it is not society’s job to save people
from themselves. It’s far better to have this freedom and the power to use or
misuse it, than to allow companies to build the echo chamber behind closed
doors._

That's a nice dogma - which breaks down as soon as one of the cults/echo
chambers starts harrassing or doxxing or otherwise hurting someone they
declared an enemy.

This whole concept is based on an assumption that if someone bothers you
online, this is always a personal issue that can be resolved by you pretending
they didn't exist. By now we have enough experience with online communities to
know that things don't work that way.

------
dang
I always look at these things from a parochial perspective, i.e. how does it
apply to HN.

His first two points ("moderation is likely to end up pissing off those who
are moderated" and "content moderation is always going to rely on judgment
calls") are true of all moderation and certainly HN. They have nothing to do
with scale though.

His third point is more interesting: at scale, even 99.9% accuracy is going to
leave hundreds of thousands of mistakes a day. No one will care about the
99.9% you got right, only the things you missed. Reporters will call public
attention to these.

That's a good point and not widely understood. But he stops there. All that
shows is that content moderation is impossible to do perfectly, which we
already know because of points 1 and 2. More interesting would be the part he
didn't write: given that a large number of misses is inevitable and reporters
will call attention to them, what follows next? Probably that the public image
will be that moderation sucks. That's rough, but it doesn't necessarily follow
that content moderation is impossible to do well.

~~~
pbhjpbhj
Surely the 'scale' issue's answer is for a community to moderate itself. That
way, the community should be happy. That's what Slashdot did (does?). All
users take part in moderation [if they wish], and take part in meta-
moderation.

Using a statistical consensus, and correction of outliers, but doing so
through the userbase such that as the community grows so too does the base of
moderators.

Making it statistical means that no one moderator can have undue influence
(other than site owners).

HN does some 2nd order stuff, flags/vouches from trusted users i think are
weighted (?) but doesn't go to the next level of then decreasing the weight of
votes for those whose votes were meta-moderated as 'wrong' (ie contrary to
community consensus).

In theory that should handle brigading, for example, because you don't know
what you'll moderate before you say yes, and you can't choose to moderate -
the site chooses you.

Slashdot also had several dimensions to it's moderation and allowed users to
choose to consume content contrary to the community norm - eg you can choose
to see more humour, even if the community in general doesn't like to view
humour.

Still the best system, on paper (I have no idea if they have to constantly fix
things!?!) that I've seen.

~~~
jasode
_> Surely the 'scale' issue's answer is for a community to moderate itself.
That way, the community should be happy. _

Self-moderation is basically what Stackoverflow is and a lot of users are not
happy with the outcome. Most of the votes that close questions as "duplicates"
and "not constructive" are done by _fellow users_. I tried to explain this
previously[0] but unhappy Stackoverflow users still insist there's an "us vs
them".

And consider that Stackoverflow's userbase is a much smaller scale than
Facebook.

[0]
[https://news.ycombinator.com/item?id=23977165](https://news.ycombinator.com/item?id=23977165)

~~~
pbhjpbhj
Under a Slashdot style system the "closed-duplicate" questions would be
downgraded for those users who choose; you might like dupes and so you could
choose to boost dupes for yourself.

The community provides a default position (duplicates closed), provides the
information ('this is duplicate'), but under a Slashdot style system you can
choose that content that others 'hide'.

------
jimnotgym
Not impossible at all. Divide the subject matter into thousands of specialist
areas, and let each area be moderated by it's own community
moderators/editors. It only becomes a problem when you try and publish every
comment on every subject through a single organisation.

~~~
pbhjpbhj
Doesn't that reinforce extremes and encourage enclaves: which is why Reddit
end up closing whole communities (the good and bad in those communities).
You're just putting off moderation of extreme content that doesn't confirm to
a community consensus at all by hiding it in a corner and letting it grow
without control.

~~~
jimnotgym
Are the readers of 'Nature' or 'The Economist' extreme? Quality curation and
editorial control are possible at those levels. I can write a letter, or
indeed an article or paper for either, but it only gets accepted if it is good
enough.

~~~
ceejayoz
> I can write a letter, or indeed an article or paper for either, but it only
> gets accepted if it is good enough.

Sure. With a maximum bandwidth of what, dozens a month?

~~~
jimnotgym
Sounds like a feature

------
djoldman
I've always been curious about "content moderation." Theoretically, if you
could perfectly allow only content that was Legal in the jurisdiction,
wouldn't every website allow everything Legal and be able to just point at the
laws and say, "if you don't like it, change the law"?

If so, is the crux of the problem that 1. we don't know what is legal, 2. We
can't perfectly allow it, and or 3. Something else.

If 3, is it that websites don't really fear legal repercussions, but fear bad
PR, thus alienating users/revenue? If so, doesn't it follow that the level of
content moderation will track the projected cost of losing the projected
users/revenue?

~~~
krapp
The crux of the problem is that limiting moderation only to allowing content
which is illegal and banning what is illegal still allows content many people
may find objectionable, spam, trolling, racist and hateful speech and off
topic content.

Not every online community wants to be a 4chan or Voat, which is what that
sort of hands off moderation inevitably leads to.

~~~
searchableguy
What if you just give each person a way to censor the content they don't want
to see?

Something like show dead posts on hn.

Although, why host content that isn't profitable?

~~~
krapp
> What if you just gave each person a way to censor the content they don't
> want to see?

That works to a degree, but it isn't foolproof because it still requires
people to see that content to begin with. It isn't proactive.

------
mountainboy
my approach:

let everyone rate content on multiple relevant criteria. Eg, quality, grammar,
agreement, sexuality, violence, hate, etc.

Make some default filters that choose societally acceptable settings for each
filter, and hides new content that is unrated.

Enable individuals to tweak the filters to any setting they wish, for a
totally custom view, including viewing unrated content.

Enable individuals to share their filters with others.

Voila! A system that allows _any_ content to be posted, but shows only "safe"
content by default and allows individuals to bypass the filters if they wish.

~~~
jorvi
At least for voting, the fix is simple and I don’t understand why both Reddit
and HN (with the default settings) don’t do it: make voting invisible. That
is, you sort the comments by the vote value they have, as is done now, but you
don’t display the value. Reddit displays the value directly, and HN by
gradually greying out comments. The problem with doing that is that it leads
to bandwagoning. If I’d had to hazard a guess, a solid 50% of downvotes are
people blindly ramming the downvote button because that comment already has a
negative vote value.

~~~
krapp
>the fix is simple and I don’t understand why both Reddit and HN (with the
default settings) don’t do it: make voting invisible

Because part of the purpose of voting/karma is operant conditioning - visually
marking certain comments as "low quality" and others as "high quality" and to
encourage commenters to prefer creating and engaging with the latter over the
former through the constant feedback loop of reward and punishment.

> If I’d had to hazard a guess, a solid 50% of downvotes are people blindly
> ramming the downvote button because that comment already has a negative vote
> value.

That's the system operating as intended, ruthlessly separating the signal from
the noise. I don't particularly like it either but Hacker News isn't about
_free_ discussion so much as _curated_ discussion.

------
sova
This title is sarcastic right? It's doable just not well because there is no
way to create a totalitarian panopticon without having your state first be a
totalitarian panoply of optics and cons.

------
ameliaquining
Given the "impossibility theorem" headline, I was hoping for at least a sketch
of a formalizable argument. I wasn't really _expecting_ one, and maybe this is
a silly thing to be annoyed about, but I think it's better to be clear about
what exactly you have backing up a particular argument.

------
erichocean
Can GPT-3 be re-purposed to do automatic content moderation?

------
new_guy
Moderation, even at scale, is a decades old solved problem, and not even a
particularly hard one. The reason places like FB etc say it's 'impossible' so
they don't open themselves up to legal liability.

~~~
pbhjpbhj
Perhaps you could expand on that with details of a solution you have in mind,
references to a moderation system using that solution would be useful too.

