
Mechanical Turk: The New Face of Behavioral Science? - ryan_j_naughton
http://priceonomics.com/mechanical-turk-new-face-of-behavioral-science/
======
jbarrios
Crowdsourcing marketplaces offer unique advantages to researchers that are
simply impossible to achieve with traditional methodologies, here's a few:

1) External Validity: Today, almost all research labs in Universities have to
source their participants from their respective undergraduate population.
Studies suggest that the results gathered from 'convenience samples' cannot be
generalized to the general population. That is, the results from experiments
conducted on 19-23 year old Stanford undergraduates may not be representative
of the general population
([http://www.mccombs.utexas.edu/research/~/media//Files/MSB/Re...](http://www.mccombs.utexas.edu/research/~/media//Files/MSB/Research/Recent-
Publications/2014Q1/PetersonR)). MTurk offers researchers a more diverse
subject pool in age, geography, language, and culture.
([https://ccrma.stanford.edu/~jieun5/publications/2012-icmpc-m...](https://ccrma.stanford.edu/~jieun5/publications/2012-icmpc-
mturk.pdf))

2) Cross-cultural and International Studies: Conducting cross-cultural
research though traditional methodologies requires dealing with differing
national requirements for protecting human subjects;variation in data
collection processes in primary care practices; data transmission among
participants; duties and tariffs on necessary instruments; fluctuation in
currency exchange rates; incapacitation of coinvestigators; complex
administration of funds; financing the additional, legitimate costs of
collaboration; sustaining strong personal relationships among coinvestigators;
and accepting longer time frames than would otherwise be expected.
([http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466730/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466730/)).
The diversity on Mechanical Turk facilitates cross-cultural and international
research at a very low cost (labs.yahoo.com/files/mturkmethods.pdf)

3) Faster Theory/Experiment Loop: One implicit goal in research is to maximize
the efficiency with which one can go from generating hypotheses to testing
them, analyzing the results, and updating the theory. Ideally, the limiting
factor in this process is the time it takes to do careful science, but all too
often, research is delayed because of the time it takes to recruit subjects
and recover from errors in the methodology. With access to a large pool of
subjects online, recruitment is vastly simplified
(labs.yahoo.com/files/mturkmethods.pdf)

Traditional researchers have raised questions about the quality associated
with the data collected from participants. The empirical evidence strongly
suggests that the quality of data collected from mturk participants meets or
exceeds the standards associated with published research
([http://www.pomona.edu/Academics/departments/psychology/files...](http://www.pomona.edu/Academics/departments/psychology/files/Buhrmester%20-Crowdsourcing-
Amazon-MTurk.pdf))
([http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...](http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0057410)).
Evidence that Mechanical Turk is a valid means of collecting data is
consistent and continues to accumulate (Mason & Suri, 2012)

Disclaimer: I'm a crowdsourcing technology advisor to the department of
Psychology at Vanderbilt University, VizCogLab, and UVic Center for Autism
Research. I'm also the co-founder and CEO of Cognilab Technologies, an online
platform that allows researchers to publish their experiments on mturk.

------
ramidarigaz
I'm doing a Master's thesis in the intersection between Cognitive Science and
Machine Learning. The group I work with looooves Mechanical Turk. It's such a
cheap way to do small validation studies.

~~~
tectonic
I've used it for data gathering, but found it to be a pretty homogenous group
of people. What has your experience been?

~~~
sireat
There are various clusters of homogeneous people. You have your classic
/r/beermoney students, same as you would get from regular studies on campus.
Then you get your bored housewives of US, and oversees workers mostly
Indians/Philipinos.

Once you realize these large groups of people exist, you can still get useful
work done by accounting for obvious biases.

Disclaimer: I've used MTurk both for surveys as a requester and also used it
as a Turker for a month two years ago, whenever I got bored.

I saw so many Kahneman inspired studies it was not even funny at the end.
(Monty Hall problems, priming, $1.10 bat+ball problem, etc etc) . I ended up
making around $6 an hour.

The best problems were porno classification problems.

I suppose that is standard stuff you do in psychology these days.

------
john_horton
If you're looking for a discussion of running economics-style experiments on
MTurk (as well as some advice on keeping online experiments internally valid):
[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202)

------
runeks
I think Amazon's Mechanical Turk (MT) is fascinating. I really wish Amazon
would release more figures on its usage, rather than only the number of
registered users[1].

It would be really interesting to see how many people earn their wages through
this service. The chart in the article says 16% in Feb '10 [2], but I think it
would be really interesting to see 1) how much money they earn, 2) how many
hours they work, 3) where they live, 4) how much Amazon pays out in total in
MT wages (all anonymized data, of course).

[1] > The only numbers that we share regarding our Worker population are these
two -- which reflect our current state: Over 500K registered Workers from over
190 countries worldwide.

[https://forums.aws.amazon.com/thread.jspa?threadID=58891](https://forums.aws.amazon.com/thread.jspa?threadID=58891)

[2] [http://pix-
media.s3.amazonaws.com/blog/835/ScreenShot2014-10...](http://pix-
media.s3.amazonaws.com/blog/835/ScreenShot2014-10-14at4.58.57PM.png)

~~~
mandalar12
Here is an article [1] trying to estimate that kind of numbers (wage, number
of active workers, etc.) from various sources and questionning the ethics of
using MTurk for science purposes.

The bottom line is: one third of tasks are achieved by people relying on MTurk
(as primary or secondary money input) having working conditions that would not
be considered acceptable in any first-world country (hourly wages around $2,
working long hours with, obviously, no kind of worker advantage or
protection).

[1] [https://hal.archives-
ouvertes.fr/file/index/docid/569450/fil...](https://hal.archives-
ouvertes.fr/file/index/docid/569450/filename/LastWords_AMT.pdf)

~~~
jbarrios
Harvard recently published a paper on the mturk demographics, you can find it
here:
[http://scholar.harvard.edu/files/dtingley/files/whoarethesep...](http://scholar.harvard.edu/files/dtingley/files/whoarethesepeople.pdf)

If you're looking for something less academic, try out this great blog:
[http://experimentalturk.wordpress.com/](http://experimentalturk.wordpress.com/)

