
Amazon’s Mechanical Turk Has Reinvented Research - DmenshunlAnlsis
https://daily.jstor.org/amazons-mechanical-turk-has-reinvented-research/
======
Radim
Mechanical Turk is great for "open", public research. We used to use them a
lot for machine learning tasks (data cleanup, model comparisons, label
annotations), along with similar services like CrowdFlower / Figure Eight. We
saw two primarily issues when applied to "non-open" (commercial) projects:

\- business-related data too sensitive to share with strangers (contractual
obligations, too much risk)

\- some tasks required non-trivial subject matter expertise and context to
annotate properly (quality control issues)

For this reason, we gradually moved to an in-house team of long-term
annotators. It's not much more expensive (moms on maternity leave, students…),
but infinitely more flexible and safer for our purposes. YMMV.

~~~
therealmarv
try PYBOSSA, open source crowdsourcing platform, you host it yourself, you are
in control of your data.

~~~
k__
I think the problem is more that you give the data to the turkers than to give
it to Amazon.

------
rococode
MTurk is a godsend for ML research and is a huge game-changer. For every other
project where the problem is "that sounds cool but we don't have enough
labeled data" the answer nowadays is "just turk it". Sentiment labeling,
qualitative comparison, error identification, and tons of other traditionally
data-scarce tasks are made trivially easy (at the cost of some money) with
MTurk, and it's pretty much a win-win for everyone involved too!

Now the ethics as far as exploitation are definitely important, but I think
the design of the site handles things quite well and makes everything fair for
all parties. If you feel a task is underpaid, there are enough alternatives
that you can just not do it. It's also true that there are many international
turkers for whom $8/hour or less is still solid pay. Then there are also many
third-party tools that allow turkers to see which HIT (task) requesters have
good track records (low rejection ratio, good pay, etc.), and the site's own
tools allow requesters to avoid turkers with bad track records. In my
experience just browsing through tasks, heavily underpaid tasks don't tend to
get done (for example, writing a 100-word summary for $0.50).

~~~
krageon
You can't receive any actual money (only amazon money, which is much less
useful) as a worker outside of the US as far as I know.

~~~
lozenge
You can in India as well which must be where most of the work is coming from.

------
chx
Data cleanup is also a great use. For example, we needed to parse an amount of
address data, we just needed country, state (or equivalent), city, we gave out
each address three times, whatever results were the same at least twice was
accepted. We had over 92% where all three were the same, another 7% with 2-1
(required review), less than 1% needed either manual cleanup before re-Turking
or just manually entering some of the more gnarly cases. We considered it a
truly massive success, price efficient and absolutely unbelievable quick.

~~~
sametmax
You outsourced personal data to mechanical turk ? Can I have the name of your
company ?

~~~
robryan
Is address data without a name/email or phone number to link it to really
personal data?

~~~
sametmax
They extracted it, so it means it was in a context. Address + context is very
much personal.

Also I note that the comment author prefers not to answer.

~~~
icebraining
_They extracted it, so it means it was in a context. Address + context is very
much personal._

That doesn't mean it was available to the MT workers. Why would it be?

 _Also I note that the comment author prefers not to answer._

They haven't made any new post since that one, how do you know if they even
saw yours?

~~~
dx034
Even if it's just an ID, if you know the company you work for is a specialised
online shop and you see your neighbor's address on there (assume rural area),
you know they ordered with them. Depending on the kind of store that can
already be critical.

------
ehead
It's an interesting question, whether or not the data is representative (or
what demographic is the data representative of).

Are there any studies done of the demographic distribution of Turkers?

~~~
archgoon
Several of the referenced papers from the article seek to address this
question:

"Evaluating Online Labor Markets for Experimental Research: Amazon.com's
Mechanical Turk"

[https://www.jstor.org/stable/23260322?mag=amazons-
mechanical...](https://www.jstor.org/stable/23260322?mag=amazons-mechanical-
turk-has-reinvented-research&seq=1#page_scan_tab_contents)

"Socially Mediated Internet Surveys: Recruiting Participants for Online
Experiments"

[https://www.jstor.org/stable/43284764?mag=amazons-
mechanical...](https://www.jstor.org/stable/43284764?mag=amazons-mechanical-
turk-has-reinvented-research&seq=5#page_scan_tab_contents)

------
jankotek
Please fix misleading title, it should be: Reinvented _Survey_ Research.

> _thirty to forty minute survey ... paid $1.10_

What sort of results do you expect from this "research"? Do you really expect
people to read the questions and answer truthfully?

You can prove pretty much anything, by reordering fields or manipulating the
questions.

~~~
mbym
Prolific ([http://prolific.ac/](http://prolific.ac/)) is designed specifically
for science research, but I've used used it for two different use cases: 1\.
Product research, quickly testing appetite for an idea. 2\. Micro tasks (a
"survey" with at least one participant, where basically the only question was
"have you done the task (on external page)?" The task in question was editing
a public transcript, so confidentiality wasn't even a concern.

Each time I was impressed with the quality of the responses. In the optional
open-ended text fields for the research survey, in particular, I was impressed
with how people took time to free-write their thoughts, rather than just
rushing on to the next question / to complete the survey. Point is I do think
you get what you pay for...

From my experience, it seems Prolific's ethical approach (e.g., insisting on
decent minimum payment rates) leads to overall higher quality of participants
as well as responses (vs. race-to-the-bottom micro-task platforms like MTurk).
[Disclaimer: I'm friends with the founders -- so I'm quite familiar with the
amount of resources devoted to ensuring truthful/quality responses.]

------
myro
I've been turking once, made a $100 worth of Amazon credits and bought a first
version of Kindle :) good old days

~~~
dx034
How many hours? Was it 30-50 hours as some reports suggest or a decent hourly
wage?

------
jonbarker
To paraphrase The Simpsons: "In the future, computer programs will be built by
labeled data sets. And our job will be to build and maintain those labeled
data sets."

------
Nasrudith
I can't help but imagine unscrupulous bot programmers going through every
possible survey and answering them quickly with garbage. Or if they monitor
expected times doing parallel instances while waiting long enough to look like
a human. High volume low value junk ruining things has a long history with the
internet.

~~~
benp84
Sadly there is disincentive to report bad workers on MTurk. You can block them
through the MTurk system which rightfully puts their account at risk for
termination. BUT Amazon sends them an email identifying you as the blocker,
which causes them to go write terrible reviews of you on worker sites, which
reduces the supply of workers for your HITs. So the ideal way to handle it is
block them from your HIT server so at least they can't do YOUR HITs again and
never report them.

------
knguyen0105
Have anyone used Mechanical Turk for tasks that involve non-English languages
like Chinese or Vietnamese?

~~~
titanix2
I tried a batch for some data typing involving Chinese characters. Nobody did
it. Maybe it wasn't paid enough, maybe there was no skilled people willing to
do it. I'm also interested in other people experience involving to two
languages.

~~~
jdietrich
That's not too surprising - most of Amazon's IP ranges are blocked by the
Great Firewall.

------
ada1981
My friend has used this extensively for all kinds of interesting things.

I interviewed him about his work combining mturk & AI to help Trump / Clinton
supporters better understand each other.

[https://huffpost.com/us/entry/us_581a4825e4b0f1c7d77c9555](https://huffpost.com/us/entry/us_581a4825e4b0f1c7d77c9555)

With regard to pay, it seems reasonable to adopt a standard where the average
per hour rate is disclosed in the research papers. This alone may provide
social pressure for academics to adopt payment inline with local norms.

Also this brings up some questions for me. Is it unethical for a researcher in
a locale with low wages to post on mturk looking for work at comparable rates?
Should posters be posting rates comparable to their own countries minimum wage
laws? Is there another standard? Could requiring some researchers (from richer
countries) to increase their rates result in resesrchers from other countries
being priced out or having their research deprioritized by Turks?

------
8bitsrule
'Mechanical Turk' is a sweatship run by a notorious skinflint.

------
ccnafr
Do people who participate in MK get any money for their time?

~~~
iagovar
Not outside the US, so I won't be so confident for non english data.

~~~
ccnafr
Why not? Are they afraid of users from low-income countries flooding the site?

~~~
mberger
Yes, and that you get a lot of tasks that you can't prosecute their origin.
For example (off the top of my head) solving captchas, cracking passwords,
doxing, hacking offers etc. I think there was also the spectre of organized
crime enslaving people to do the tasks and taking the profits.

