
Ask HN: Does anyone use the Amazon Mechanical Turk API? - gcv
I'm working on a project that relies on a human workforce to perform some simple image identification tasks. AWS Turk seemed to fit the requirements pretty well, and I have the basics working with externally-hosted tasks.<p>Does anyone here actually use Turk, though? The forum seems nearly abandoned (http://developer.amazonwebservices.com/connect/forum.jspa?forumID=11), and no Amazon employees have responded to any questions. The API has not had any updates in years. Is the Mechanical Turk now a walking corpse?<p>Read on for a description of the problem I've run into. Maybe a fellow enterprising HN reader dealt with something similar and found a solution. :)<p>I need to enforce that the same worker does not look at closely-related images. I can enforce that in my app, and basically show a task that says "please return this task and try some of our other tasks in a couple of hours". This is annoying, and drives up the "returned tasks" statistic for the workers. I would very much like to make Amazon filter out tasks for users who have run out of valid work in my system.<p>At first, it seemed that Turk qualifications fit the bill. Unfortunately, the Turk system does not support "negative" qualification, e.g., it cannot only assign tasks to workers who do <i>not</i> have a particular qualification. Amazon employees have not addressed this question from a year ago: http://developer.amazonwebservices.com/connect/thread.jspa?threadID=32744&#38;tstart=0<p>Then, I tried blocking workers once they run of tasks, and unblocking them once work shows up. This does not affect task filtering, and instead tells the user about a block. Additionally, blocking may or may not have really bad side effects for workers, another question Amazon employees have ignored: http://developer.amazonwebservices.com/connect/thread.jspa?threadID=48851&#38;tstart=0<p>Advice and suggestions welcome.
======
raffi
I use it for <http://www.feedbackarmy.com> and other tasks. I recommend using
an ExternalQuestion. This will let you host the worker form on your own site.
I recommend using JavaScript and whatever you want server side to let the user
know whether they can work on your stuff or not. Maybe set a cookie with an ID
so you can know what kinds of images they've looked at.

Doing something on your end is your best bet. I like MTurk but the API is not
so great and support does not exist. Good luck.

~~~
gcv
Thank you! I am already using ExternalQuestion forms and generating the exact
questions at view time. In your experience (or opinion), workers do not object
if a task just says "sorry, please return this HIT and try again later"?

~~~
raffi
I don't know the answer because I haven't been in your situation (yet). I'm
thinking of offering support for worker demographics and the scheme I
described is how I plan to implement it.

a. Do I know about the worker? Yes - go to c. b. survey the worker with some
demo questions and store them c. Does the worker's demographics match what the
user wants? Yes - let them through d. Prompt worker--sorry, you're cute and
all, but not what this user wants.

------
camtarn
Check out <http://www.turkernation.com/> \- still seems active.

------
maccman
We using ExternalQuestions with the Ruby API lib for Socialmod. We're pretty
impressed the quality of work that comes from MTurk - questions get answered
under 5 mins. All the logic is our side, especially the banning/blocking
logic, which gives us a lot of control.

As to your problem, why not create a new hit-type for each group of similar
images, and then restrict worker access based on that type?

<http://www.socialmod.com>

~~~
gcv
Thanks, that promising and helpful. How do I restrict worker access by HIT
type? I don't see anything in the documentation that lets me do this —
negative qualifications would work, but the MTurk system does not support
them.

In your blocking logic, do you simply show a HIT that says "please return this
HIT"? I already implemented this, but I'm worried that it is bad etiquette.

------
pgbovine
Check out this research project from MIT ... it aims to provide a more sane
interface to MTurk:

<http://groups.csail.mit.edu/uid/turkit/>

------
pedoh
I'm not familiar with AMT, but you might check out <http://crowdflower.com>
and see if they might be a good fit for your needs.

------
lukejduncan
I know speakertext.com uses it, and for a cool product!

------
jey
You should have 1 task with many instances, and dynamically assign the
specific task only after you have a specific user who has accepted it. This
way you can use the user ID in your task-assigning logic.

~~~
gcv
That's exactly what I did. The question is, how do I prevent people who have
no more valid tasks available from seeing or accepting HITs? I can lock them
out in my own logic, but their Amazon records will show that they abandoned a
task — and it wasn't actually their fault.

