

Mechanical Turk Workers Are Not Anonymous - anandkulkarni
http://crowdresearch.org/blog/?p=5177

======
nmcfarl
This ID is pretty much the only thing employers get to ID the workers, and
thus track the quality of their work. (Which is particularly important on
Turk, as anonymity seems to encourage fraud.) Everything else is push: web
service calls to contact a worker, or comment on an assignment. That stuff is
all intermediated by turk, meaning it’s all anonymized fairly effectively -
and quite constrained as to content and format.

These days Turk goes so far as to handle your 1099s for you - so you don’t
know the names of even the workers that have done more than $600 worth of work
for you…

Of course most employers in it for the long haul build up reputation databases
built on these ids, and many encourage their workers to fill out profiles,
direct on the employers website, allowing for more direct contact, and less
constrained messaging.

Which is why this is a big deal for the employers: names, pictures, locations,
and review helpfulness, can all be shoved into the reputation algorithm or
used by survey companies for demographics. So it will be harvested.

And that makes turk a lot less anonymous. Which seems like a breech of trust
to the workers, because although they aren’t guaranteed anonymity in the TOS,
that has been the bargain so far.

------
anandkulkarni
This was a curious design choice by Amazon.

By choosing to use Amazon IDs as worker identifiers, Mechanical Turk gets all
of the disadvantages of having anonymous strangers do jobs on the web -- lack
of accountability, diminished incentives, malicious workers -- but none of the
advantages of protecting workers, since at the end of the day, they're not
actually anonymous anyway.

This is a non-issue in MobileWorks (and for that matter, oDesk and Freelancer
as well). In these systems, everyone's explicitly working under a real name,
which improves incentives to perform well, and makes it a whole lot easier to
get good results through trust and honesty.

~~~
jheitzeb
Well said and 100% agree. I pressed them on that a few times and never got a
good answer as to why _they_ think anonymity is a good design choice. Many
others I have spoken to who've used mturk significantly have come to the same
conclusion you did.

~~~
tellarin
Well, I know this is not the typical use case, but the need I had for it was
to perform some experiments on the quality of recommender systems suggestions.

In this case, as the experiments involved people and the intend was to publish
the results, the "workers" had to be anonymous.

------
LogicX
> "For academic researchers: worker IDs may have to be treated as personally
> identifiable information. For example, publishing worker IDs online in
> public data sets may be a violation of worker privacy, and counter to the
> requirements of the researcher’s institutional review board."

This is why we created <http://SocialSci.com> for use by academic researchers:
We have better control over PII outside the Amazon ecosystem, and access to
more data we can take our own steps to protect or de-identify, enabling higher
quality data collection.

In general academic research done on Mechanical Turk has a ton of issues, from
response quality to verifiability of responses. Frankly people have no
incentive to remain authentic across surveys, and we've seen cases of people
just taking on identities or just faking answers, which isn't reliably
detectable. The only way we've been able to overcome this is by tracking
users' response consistency over time, and keeping them anonymous in the
process. It's proved to be a much more reliable method, and at this point
based on our experience we can't recommend MTurk for valid academic research.

~~~
1337biz
You will always have these issues. If people want to cheat the system they can
create fake accounts, administrate profiles for family members and so on.
Isolated surveys, no matter if qualitative or quantitative questions, are
quite limited in their implications as long as there are incentives to give
dishonest responses and no other ways to evaluate the data in the context of
practice exist.

~~~
calinet6
Developer at SocialSci here. You're absolutely correct, the incentive for
gaming the system is always there. The difference is that we also incentivize
consistency and are very proactive about tracking down fake accounts through
various means. You're right that it's a difficult problem, but while Amazon
turns a blind eye to such abuse, our system is designed from the start to both
disincentivize and prevent it, and in the meantime encourage good consistent
high-quality responses over time.

We don't claim to entirely prevent fraud, but we're very confident that we
raise the barrier and reduce the rate significantly.

------
smackfu
Are they supposed to be anonymous, or were people just assuming that?

Edit: The closest thing I found was that you can't create tasks that involve
"collecting personal identifiable information". I would think that is aimed
more at preventing tasks that are essentially selling your info for the price.

------
firloop
Shocking that something with privacy implications like this wouldn't have been
already caught by Amazon. It almost seems like common sense to not reuse (at
least, publicly) the same user ID between a user's public page and a users
more anonymous MTurk profile but I guess it slipped through the cracks or
something.

------
obituary_latte
It appears crowdresearch.org has yet to research how to handle a crowd.

~~~
raverbashing
Also, do they need a whole paper to say that the Worker ID is the same userid
used in different parts of Amazon?

------
runarb
Cache:
[http://webcache.googleusercontent.com/search?q=cache:crowdre...](http://webcache.googleusercontent.com/search?q=cache:crowdresearch.org/blog/?p=5177)

~~~
mrtimo
here is a text only version that worked better for me:
[http://webcache.googleusercontent.com/search?q=cache:crowdre...](http://webcache.googleusercontent.com/search?q=cache:crowdresearch.org/blog/%3Fp%3D5177&hl=en&strip=1)

------
niggler
Where in the privacy policy does it state that Mechanical Turk program
wouldn't share the ID with people using the service? I don't see any item in
the privacy notice <https://www.mturk.com/mturk/privacynotice> or the
Participation Agreement <https://www.mturk.com/mturk/conditionsofuse>

------
krilnon
Static page: [http://groups.csail.mit.edu/uid/crowdresearch/mechanical-
tur...](http://groups.csail.mit.edu/uid/crowdresearch/mechanical-turk-workers-
are-not-anonymous/)

------
lnanek2
Not particularly a new finding. Heck we've had posts where people had the
turks post pictures and details of themselves too, and many times turks have
to create a profile on site x as part of their job...

------
noagendamarket
Amazon should use bitcoin for mechanical turk.

------
est
site's gone... any cache?

