

Poll: What experiment to run on AWS mechanical turk? - adatta02

After our experience using the mechanical turk we've become a bit obsessed with trying to find a really awesome use case for the service - even if its just experimental.<p>We have some downtime right now and we are narrowing down the experiment between the following:<p>1. Some sort of cooperative art project - inspired by http://www.tenthousandcents.com
2. A massive chess game that forks every time a move is made (probably a poor idea)
3. Some implementation of the Prisoner's dilemma game.
4. Any time of game to get a sense of how risk seeking or risk averse the Turks are - possibly a tie in with the Monty Hall Problem.<p>Most of these ideas are centered around game theory but we are pretty much open to anything (within reason). Any votes or opinions on what would be the most interesting/entertaining?<p>-Thanks
======
AlexeyMK
I'd love to see a prisoner's dilemma experiment via MTurk. These tend to be
financial in nature, though; can you set variable payment rates per task,
depending on external factors?

~~~
sigstoat
you can pay out arbitrary bonuses once they've answered.

so you could easily do single-round prisoner's dilemma, and have amazon pair
up 2 (different) folks at a time, and when you get the results back, look to
see what the folks in each pair did, and award bonuses appropriately.

i'm not sure if there is a way to do an iterated version.

------
iamelgringo
A friend last week told me about a business he's working with. They do Natural
Language Processing to scan thousands of articles about a specific
topic/product to see if they are positive, negative or neutral. They are then
hoping to turn around and sell the business intelligence.

I was wondering how it would compare on a cost/effectiveness basis with the
Turk.

Let us know what your experiment is and how you fared.

------
MaysonL
See if you can use MT to provide the fitness function for some genetic
algorithm, evaluate the results, tune it, and profit!

------
Deadsunrise
Some months ago I tried to set up a project to transcribe Alexander Shulgin's
Lab books but Mturk wasn't available outside US.

<http://www.erowid.org/library/books_online/shulgin_labbooks/>

------
lacker
Go meta - use Mechanical Turk to ask what would be a really cool idea to run
using Mechanical Turk.

------
andr
Liar's poker. Ask 2 people to pick a number and the one that picks the lower
doesn't get paid. The number of people that actually play the game is the
result of the experiment.

~~~
eru
How about a dollar auction?

------
zitterbewegung
How about creating an experiment and designing it right on turk? You create a
rough sketch and then you auto-correct it using people?

~~~
antigravity
You mean keep tinkering with the study until it "works"?

If you're not careful, you could get
<http://en.wikipedia.org/wiki/Publication_bias> on a massive new scale...

It would be interesting to try to find scientifically optimal ways of doing
study iteration, or approaches to interpreting the complete (experiment_rev,
results)-list from another researcher...

------
bjclark
I'd pay $.05 to get the them to up vote my site for certain search terms on
Google's new "Search Wiki". See what happens.

------
shafqat
What was your previous experience?

~~~
adatta02
we used the Turk service to basically do OCR on about 3k images -
[http://shout.setfive.com/2008/11/20/artificial-artificial-
in...](http://shout.setfive.com/2008/11/20/artificial-artificial-intelligence-
our-experiance-with-mechanical-turk/)

