

Case Study: Accuracy & Fraud on MTurk - geofflewis
http://tumblr.gethumanoid.com/post/16940419386/accuracy-and-fraud-on-mturk-case-study

======
sheraz
Interesting story -- I've attempted to use mturk a few times in the past, but
it is actually a lot of work to get it right. Good instructions are really
important, as well as a way to measure performance.

Good luck to them as I think they might have made mturk easier to use for
people like myself who see the value but couldn't figure out how to "make it
work."

~~~
MediaSquirrel
good instructions are crucial.

------
chrisyeh
I just used Humanoid for the first time. I had a job that would have taken
about 5-10 hours (look up 50 ad agencies, record their Twitter account, blog
URL, and their CEO's Twitter account and blog URL).

Humanoid accomplished the job for under $35. Considering I would have had to
do it myself, or nag an intern for a week to do it, I consider it money well
spent.

P.S. I don't know what results I would have gotten from going to Mturk
directly, but the results from Humanoid were better/cheaper than my old habit
of outsourcing to an Indian body shop.

(Disclosure: I am an investor. But I'm giving honest feedback.)

------
bitstormer
Great data, very cool how sophisticated algorithms can increase performance
dramatically for human-powere tasks

------
garethsprice
What does the author mean by fraud? Do people submit junk results and hope
they slip through the screening process, or it it something more insidious?

~~~
kailashbadu
Junk results, mostly, yes. That way they can complete more taks faster and
still make money because most of the time their work is not manually approved.

------
MediaSquirrel
"Humanoid’s software identified 8% of the workforce as engaged in fraud."
<\--that is a big number!

------
gcb
Fails to mention how much they cost on top of the mturk prices to the client

