At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app).
Then I tried a new tactic. I hired some pros from various translation websites. All were more expensive (money-wise) than MT, but the results were better, and I spent a lot less time. Some were even close to MTurk prices.
So, I don't use MTurk anymore. Maybe if I had a more mindless task I'd try again, but it is a lot of hassle.
I know this isn't exactly what the author means by "spam" - but even the workers on that site are often dishonest. It's a total black market.
I had good success stopping cheating by using images of text instead of actual text. Typing in a foreign language is not trivial if it's not something you're used to doing.
It seems to be a failed social experiment to me, there is definitely a negative feedback loop there. Bad HITS, low wages, withheld payments, bad workers, sweatshops. Hard to imagine it will get any better.
I wonder if you reversed the process and had projects bid on workers if it could counter the race to the bottom effect. Include a system where work could be divided into common types and include worker metrics and feedback that captured any specialization. Perhaps allow some workers to fill a managerial role, where they acquire large blocks and have a trusted group they farm out work to. Presumably wages would go up but bad work would go down, perhaps offsetting it somewhat.
Not 100% relevant to this thread, but if it helps, I've had success with http://www.icanlocalize.com for localizing apps. They assign a human translator for your strings and are accommodating with custom directions you might have. They'll even take your strings file from Xcode.
(Disclaimer: I'm not affiliated with this site, your mileage may wary.)
My experience with MTurk suggests that Amazon has mostly abandoned the product. Amazon still uses it for product classification, but does not participate in the community forums (even on the official AWS MTurk developer forum). No new features have appeared in several years, and enhancement requests have fallen on deaf ears.
That said, MTurk still works. The API still does what its documentation claims. People still perform tasks. Payment still works.
Nathan, thank you for the kind words -- but I really can't take any credit. Just know we are actively listening and we take all the feedback, even the controversial bits, very seriously.
The obvious question: do you have a computer program set to alert you if "natala" appears on a tech site in a mturk context, or a mechanical turk task set with a 10 cent payment to email you whenever you're mentioned?
;-) No. I found out the old fashioned way -- a friend saw the post and sent it to me. He reads YCombinator news much more frequently than me! It does help to have a unique name...
But what have they done lately? Do you seriously consider opening a blog and a twitter account to be "sort of active"? The blog contains a single fluffy marketing piece.
I think this is one of those examples where nobody's rushing in because no matter how you slice it, the product is just not that useful.
I've used it a few times and because the eco system is truly about hiring dirt, dirt cheap labor you kind of get what you pay for.
Beyond that the revenue model sucks - the concept of running millions of micro-transactions and having a little piece of all the action is compelling, but again because the product use is so limited it's difficult to make signifigant revenue.
It seems like going through CrowdFlower could help a lot. Has anyone used it? I guess it worked for the author.
At my last job I was involved in a project where we were going to use CrowdFlower, but it ended up being canceled. I was always curious about how good their results are.
I think mturk is vastly underdeveloped, there are all sorts of things people could be using it for, but don't simply because they don't know about it or it's too complex.
Before I started my current startup, creating an mturk competitor was something I considered, but the fundamental problem is handling the payments. Most payment processors wouldn't touch you due to the risk of fraud without a huge (>1mm) deposit.
Paying individuals small amount of money is hard, you pretty much have to use paypal, and if you're lucky enough not to have your account suspended they'll completely rip you on the charges. And you'll still carry all the risk of people trying to use your system for money laundering.
Then I tried a new tactic. I hired some pros from various translation websites. All were more expensive (money-wise) than MT, but the results were better, and I spent a lot less time. Some were even close to MTurk prices.
So, I don't use MTurk anymore. Maybe if I had a more mindless task I'd try again, but it is a lot of hassle.
I know this isn't exactly what the author means by "spam" - but even the workers on that site are often dishonest. It's a total black market.