

A Plea to Amazon: Fix Mechanical Turk - ananthrk
http://behind-the-enemy-lines.blogspot.com/2010/10/plea-to-amazon-fix-mechanical-turk.html

======
terra_t
Aw dammit, Turk is great.

I spent $300 in one day on Turk and Amazon gave me a personal account
manager... And that's the problem, if you spend $300 on Turk in a day you're a
high roller, Turk just isn't that big of a market. I've had days when I've
been responsible for 20% of the HITs in the system and I'm just a poor
computer programmer who writes code for a penny a line. I just got an email
the other day from a Turk who was begging for more HITs.

Personally I've got no problem with the API and for certain kinds of tasks
I've got good answers for the quality problems. For other ones, I haven't
figured it out yet.

If you think Amazon's offering a service that's too bare bones, you ought to
start a start-up that layers services on top of it. It may be a tough row to
hoe though... My layer that interfaces to turk is specialized to my purposes
and isn't spiffy, commercial and easy to use. It would probably cost you 20-50
times as much to develop something "usable" than it cost me to develop
something that fits my workflow. Unless you can amortize this cost over a
LARGE number of customers, your spiffy new system is going to cost more than
it costs to pay the Turks. And good luck getting Angel or VC money to develop
something that might never get all that big... And that will probably get
regulated with the government when they find out that you're getting people to
work for you for $2.50 an hour!

For now I'm tickled pink because I can get labor cheaper than my competitors
can -- I can escape the economic problems that keep silicon valley companies
going in circles chasing each other's tails.

What I really wish is that I could pay people Facebook credits to do work for
me: my guess is that I could be paying people $1.25 an hour that way...
However, Facebook knows the first thing people would do if it was easy to pay
credits out to people is create gambling apps, so they don't make it easy.

------
froo
My main issue with Mturk is not being able to sign up as a non-US company.

It's not that I'm angry at Amazon or anything, I just wish that they would
come to the conclusion that our money is just as green as everyone elses.

~~~
Caligula
Same with me. I spent over 1k with a friend on turk, even reported blatant
security issues to one of the PM's yet I was unable to sign up and had to get
my American friend to do it.

I see no reason why they can't open it up to Canada. Terribly frustrating.
Take my money!

------
PanMan
I have used Mechanical Turk quite extensively the last few months. But I have
always used it via Crowdflower. They offer a more sensible API and interface.
And they keep scores on all workers, and calculate how well they perform. This
influcences the weight of their job submissions. It takes a bit getting used
to their terms, but I'm quite happy with the end results.

~~~
bravura
CrowdFlower has some annoying particularities which are undocumented.

In particular:

The CSV parser doesn't work / is too strict / something. The only way to
upload complicated data files in as "JSON". The only problem is they don't
accept standard JSON. You cannot have a comma separating the rows. (!?) I have
no idea how much time I spent trying to figure this out.

Additionally, if your task is too hard, and more than 50% of judgments are
rejected, they auto-pause your task. When you resume it, it will simply pause
a few minutes later. So, for certain tasks, you simply _cannot finish the
job_. Their rationale is that CrowdFlower pays for the judgments that are
discarded, and you pay for only the judgments they keep, so if they discard
too many judgments they lose money. I would rather just pay extra money for
tasks that are very hard.

------
grandalf
I have a product that I hope to launch before Jan 1 that will address these
issues.

~~~
davi
Tell us where to sign up to be notified of beta.

~~~
grandalf
Please sign up for the beta here:

[https://spreadsheets.google.com/viewform?formkey=dDNkei1Hb0l...](https://spreadsheets.google.com/viewform?formkey=dDNkei1Hb0lKYURlSmhoVVlNVTNOeXc6MQ)

------
aantix
Completely agree with his sentiment regarding TurkIt and the prospect of
making iterative tasks easily accessible to the masses. Read some of the
papers that Greg Little has put out from MIT to open up your eyes to some
advanced uses of Mechanical Turk.

Using the RTurk gem and my recently released Turkee gem (
<http://github.com/aantix/turkee> ), I've been able to quickly throw together
a demo Rails application that allows Turkers to vote on previous Turker
submissions or use other turker's submissions as the basis of their own own
submission.

With my experience under this type of interface most Turkers will vote for one
of the previous submissions but if you offer a small bonus for the highest
voted submission, more turkers will take a chance at submitting something
original (even if it's a trivial 50 cents bonus).

The demo's code is available under my github account (aantix/Turkee-Iterator).
Here's some of screenshots of my demo; the top image shows my initial
parameters for setting up my Turkee task. The second screen shows the
responses I received from the Turkers. The third screen shows the form that
was presented to the Turkers (either vote or submit a joke).

<http://imgur.com/13rCM.png>

<http://imgur.com/ghveH.png>

<http://imgur.com/229uc.png>

Right now I am writing a game that generates content using a couple iterative
approaches to content generation with Mechanical Turk (hence my interest in
this area).

Don't hesitate to drop me a line if you have any problems with getting my demo
running.

------
matt1
Chris Conley, a member of this community and a well known Philly developer,
has been building a powerful API for developers to use to interact with
Mechanical Turk (it started with his own frustrations using theirs). You can
check it out here: <http://houdinihq.com/>.

If you're looking for a solution to some of the problems addressed in this
article, his API (and the corresponding Rails gem) are a good place to start.

------
iwr
I think MT is failing because they only allow busiless in the US, which is a
tiny market for jobs that pay pennies/hour. But if Amazon managed to open up
say, India or China, there would be a much richer ecosystem in place.

An interesting finding is that it costs about $5K-10K to open a US bank
account from overseas, post 9/11. Pre 9/11 it was probably around $100.

Although you can use MT credit to buy stuff out of the Amazon store.

------
akozak
These change requests are (mostly) from one particular perspective, so it
ignores some other problems from the worker's perspective; namely, that the
market hasn't quite settled on reasonable rates for work, and that a worker
has insufficient protection from exploitive work providers. These are also
serious problems that could hinder it as a platform.

~~~
john_horton
I think the relative supply and demand are going to determine prices here,
just like any other market. A price floor could be imposed, but this would
likely price some work out of the market or lead to other work-arounds as
trading partners try to avoid constraints.

As far as protecting workers, in a survey I ran last year, there wasn't any
evidence that workers on MTurk feel more exploited by their online bosses
compared to their offline bosses:

<http://www.technologyreview.com/blog/arxiv/24646/>

Though obviously those that feel particularly mistreated could be selecting
out of the market.

------
paul9290
Great community and service, terrible and confusing interface. I wonder maybe
their UI is so bad as they only want to attract a certain demographic, though
it seems their demographic(turkers) runs beyond just computer nerds.

------
bradgessler
Fix these problems and you may have yourself a viable startup.

------
pitdesi
Has anyone tried CloudCrowd? They have a similar system to MTurk but it's
built on facebook and it's a managed process similar to crowdflower...
Interested to hear any experiences.

