

Does lack of reputation help the crowdsourcing industry? - yummyfajitas
http://www.behind-the-enemy-lines.com/2011/11/does-lack-of-reputation-help.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+AComputerScientistInABusinessSchool+%28A+Computer+Scientist+in+a+Business+School%29&utm_content=Google+Feedfetcher

======
anandkulkarni
Indeed, Panos's remarks are spot-on here. Reputation and quality management on
small amounts of work are really only the smallest problem in crowd computing,
and have been artificially created by the lack of simple reputation systems in
broken markets like Turk.

This is one reason why next-generation crowd platforms like our own,
MobileWorks, (and for that matter, why users of crowd platforms) should make
use of a high-quality workforce as a starting point, rather than as their
ultimate goal.

Once you assume that you have a high-quality workforce, a host of other
problems become more relevant and valuable.

\- How can we make it simple for a user to specify their tasks in a way that
crowd workers can understand? \- How can we use crowds to solve arbitrary
tasks, rather than a limited range of tasks? \- How can we give back responses
in real time?

One of the reasons that MobileWorks (www.mobileworks.com) has been so
effective as an alternative to Mechanical Turk is that it goes beyond
reputation to address these problems.

------
PaulHoule
here's my take, for the tasks I've done with turks. Maybe I'm no expert, but
I've run enough HITs to get an account rep.

A certain group of turks are just criminals who do totally bogus work. Maybe
they use automation or maybe they just click as fast they can by hand, but
their work is worse than an optimal model that does work by chance. They're
easy to catch, both technically and emotionally.

The people I have a hard time with are the "superturks" who do a lot of work
and manage to do it just barely well enough that it's hard to kick them out.
Technically it's easy to catch them because they are the N people who did the
most work. Maybe I could just be tough and kick them out, but I'd get the work
done more slowly without them.

My guess is that a turk who's good at task A might not be good as task B and
vice versa. I know that turks like my tasks enough that I get regular emails
begging me for more HITs...

... but then, I hate to break the news that I built an A.I. that's more
accurate than the average Turk and 1000 times cheaper.

