
How Crowdsourcing Turned on Me (2014) - grkvlt
http://nautil.us/issue/18/genius/how-crowdsourcing-turned-on-me
======
mapt
So the author decided to rely on "Crowdsourcing"... without actually building
any of the features needed to make crowdsourcing work... and was surprised at
the outcome?

He appears to have supplied the task as a wide open Drawball or Google Docs
type interface. This is great if you have some lasting leverage over the
participants (such as "I can punch them in the face" or "I can fire them"),
but otherwise some form of advanced moderation (Wikipedia with moderator bots)
would be step zero, and a validation process would be step 1. The
possibilities for, eg, anonymous drawings of dongs, are early and obvious in
any open system, to the point that I start out collaborative document editing
sessions in gaming by telling people to get it out of their systems at the
beginning.

Mechanical Turk is great, but the cost of paying people <$1/hr is you have to
assume some percentage of them are unreliable or bots just trying to make
money (if not actively malicious), and designing this expectation into your
workflow is nearly the whole task at hand. There will be some multiplier where
you pay for c*N hours of labor in order to perform N hours of work. Randomized
peer review methods combined with long-term engagement can keep c quite low,
far lower than management expenses in an enterprise, but it will always be
greater than 1.

------
biot
It would have been interesting to add in a validation layer, letting the crowd
evaluate each others moves, creating a virtual trust score for each player.
Those with high trust scores get to meta-moderate others moves before those
moves can be applied to the board. You can also try to catch trolls here by
presenting them with a known good matching and the system blackholes those who
reject it. Similarly, if their overall trust score dips below a certain
threshold, whether due to poor skill or malicious activity, they get
blackholed then too and any subsequent change they make to the board is only
visible to them privately though they think they're causing havoc.

Combine a system like that with a requirement that _N_ (some reasonable number
of randomly chosen people) must agree when combining pieces or splitting
pieces before the change is applied to the master board. And add the ability
of the project admins to scan the board and lock in place obvious good
solutions. That might work.

~~~
LionessLover
We have such systems in place - see Wikipedia and/or Stackoverflow. Both
platforms IMHO have become stagnant messes. A group of people who spend
waaayyyyy too much of their life on those platforms and by now identify with
it way too much get power-hungry. It's "their" platform now. While still being
able to deliver some value a lot of the fun and "feeling easy" about or when
using the platforms actively (instead of just passively looking at what they
produce) is gone.

It's equivalent to managers or politicians who are in power for too long.
Basically, the 0.1% who contribute the most are the ones who should actually
get thrown out of "office" occasionally.

~~~
jlg23
> A group of people who spend waaayyyyy too much of their life on those
> platforms and by now identify with it way too much get power-hungry.

Though, what is "waaayyyyy too much"? Someone who consistently responds to a
question on SO every day or checks 10 pages on WP for language issues every
day?

And are long timers really power-hungry or just fed up with discussing the
same things over and over again? (Not saying the latter is better.)

I'm sure what you say is true for some, but I think the vast majority is
actually civilized and most of the time quietly doing their thing for a
project.

What I am missing with most "power awarding" algorithms is a time factor:
Respect has to be earned again and again and some "powerful" person who has
been inactive for a year can easily come across as condescending to someone
who has been very active and engaged in a project for just the last 9 months
and thus does not have more than baseline respect for the old timer because
s/he never saw him/her in action. An example of a, in my opinion, better
solution would be to lose 10% of your karma for every month of inactivity on
HN (just an example, I don't think HN suffers from the problems described) or
removal of the "auto patrol" flag for users who have been inactive on WP for
some time.

~~~
brashrat
my definition of "waaayyy too much of their life" in the context of this topic
is, people who don't have much going on IRL due to social adjustment
impairments and personality disorders discover these online communities and
have their emotional needs met by fitting into an online community.

Nothing wrong with all that, some hermit toiling over research into an obscure
topic can do great work, but the same sorts of neuroticisms and anti social
behaviors that can plague real life can more easily over plague online.

------
metafunctor
Most crowdsourcing systems hinge on trust. Too much trust, and exploitation
and sabotage will occur. Too little trust, and nobody gets excited.

You cannot just give trust away like hugs. Trust must be earned. Trust is a
social construct, and if that's not built into your system, your system is
bound to fail.

Almost everyone will be honest and genuine, but that's not enough. It takes
just one asshole in a thousand to burn down the building, if they have nothing
to lose.

------
DanBC
This story is interesting because of the hate generated in a small number of
people around the perceived cheating.

Or maybe the stuff around anonymous crowds will always contain destructive
assholes.

People might be interested in the previous discussion of this article (only 25
comments)
[https://news.ycombinator.com/item?id=8499452](https://news.ycombinator.com/item?id=8499452)

Here's a different post about the 2011 challenge, with some interesting
comments:
[https://news.ycombinator.com/item?id=9021383](https://news.ycombinator.com/item?id=9021383)

And here's another one with 50 comments:
[https://news.ycombinator.com/item?id=3164466](https://news.ycombinator.com/item?id=3164466)

~~~
grkvlt
The previous submission had exactly the same URL _and_ title, so why did the
duplicate detection system allow me to submit this? Still, there's some
interesting discussion there, thanks for pointing it out.

~~~
LukeB_UK
Because it was 584 days ago. HN lets you repost something after a while.

~~~
retox
If a link was posted that long ago maybe the original year could be
automatically appended.

~~~
LukeB_UK
Usually the year means the year it was posted on the source site. That doesn't
necessarily mean it'll be posted here the same year.

------
kriro
The shredding sabotage seems very interesting. There is probably a lot of
unexplored space in creating resilient (self repairing) crowds and/or the
tradeoff between opening up the crowdsourcing process and security/safety
concerns.

~~~
grkvlt
Yes. I wonder why they didn't just roll back the attacker's moves? It seems
like the problem is one of those NP-hard type situations, where the solution
is very complex (hence crowdsourcing as a brute-force solution, rather than
anything algorithmic) but _checking_ a proposed solution is tractable. You
might be able to validate a move (or perhaps a block of moves) and either keep
or reject.

------
6stringmerc
After reading the attacker's note, which I'm very thankful the author
included, I'm inclined to agree with the negative perception of
"crowdsourcing" in contests as mentioned. There's merit in researching the
effectiveness of groups, but, I mean, sort of harnessing a mass of people for
individual ends can be altruistic or simply a mechanical turk like the
examples, but it can also be used for mob-like tactics. What I'm saying is
that it's simply such a wide platform that I agree it has the potential to
solve most any problem thrown its way, provided the conduit leader provides
the right incentives.

As for those who might take issue with the style of the writing, hey it's just
got some flourishes. A bit above and beyond the usual newspaper dryness. I can
appreciate the effort to put together something with more than just a one
sided, personally glorifying take. This felt a bit more in touch with balance
on its own.

------
jlarocco
I'm not sure I agree with the tactics, but I agree with the "attacker." It
seems pretty obvious that the intent of the challenge was to find an efficient
_automated_ way of reproducing the shredded documents.

The author even points out the movie where a bunch of people put together
shredded documents. So everybody already knows it's possible with enough
people working on it. But it's boring, labor intensive, very expensive, and
doesn't scale at all. Sure, it's faster on 5 documents over a few days when
everybody's excited to be competing, but how many people are going to stick
around working for free for 50, 500, or 5000 documents when the system is used
in real life?

------
psb31
The tradeoff between verification/validation and openness is definitely a
tricky one. I'm the co-founder of a crowdsourcing platform prolific.ac and we
settled on system with minimal verification (via sms) with background
screening and shadow banning. It's been effective so far at limiting bad
actors though I think there are definitely some more effective ways to address
this problem.

------
UhUhUhUh
One could look at the crowdsourcing strategy as a darwinist approach... It
works in nature...

------
natch
It's hard to see how the author thinks that "harnessing" humans can be a legit
solution for this type of challenge.

Think about it... putting harnesses on humans, if you take it literally,
actually sounds pretty disturbing. I realize it's not literal, but the mind-
numbing work he has envisioned for large numbers of people is not something I
would wish for as a future solution for anything.

~~~
draw_down
Well, that's not a very good point.

