
A Bot Panic Hits Amazon Mechanical Turk - yarapavan
https://www.wired.com/story/amazon-mechanical-turk-bot-panic
======
yarapavan
The blog post mentioned in the article -
[https://www.maxhuibai.com/blog/evidence-that-responses-
from-...](https://www.maxhuibai.com/blog/evidence-that-responses-from-
repeating-gps-are-random)

In the past day or two, I discovered that a large number of responses in my
latest Mturk survey appear to be random responses. I detect that a large
portion of these random responses have repeating GPS locations. So far, a
relatively large number of social psychological researchers also seem to have
noticed a drop of data quality in their Mturk data and detected concerning
patterns using the GPS method (see the related discussion threads on Facebook
PsychMAP).

------
herogreen
I find this article interesting because I did not know that MTurk was used by
psychologist to survey people. The only usage I had heard of was data labeling
for machine learning. This surveying usage seems very strange because I expect
the data to be very biased: the (human) respondents will be located in places
where the average salary is close to what is paid by MTurk, will be abe to use
a computer, etc...

~~~
snapspans
If you're interested, there is more on that (linked from the article) here:

[http://journal.sjdm.org/10/10630a/jdm10630a.html](http://journal.sjdm.org/10/10630a/jdm10630a.html)

Effectively, without the bots, the paper contends that the Turkers are at
least as representative a sample as undergraduate students.

~~~
sireat
One thing that the paper does not address is MTurkers becoming professional
survey takers who recognize typical psychology surveys thus skewing the
responses.

Personal anecdata: I took many MTurk surveys for about a month in 2012. It is
mind crushing work.

There were countless near identical surveys presumably by psychology
undergrads.

Pretty much anything in Kahneman's Thinking Fast and Slow was there.

Very quickly you recognize the typical priming experiment and then is your
response any valid?

If you know the idea behind the baseball bat and ball pricing question then is
your correct answer made in a split second helping?

Most survey givers wanted MTurkers with high ratings.

If you think about it, those MTurkers with high rating would have likely seen
all those survey questions already.

Thus experienced MTurkers would really distort the answers compared to a
random sample of undergraduates.

Maybe you can compensate for that somehow.

------
georgeecollins
Isn't it ironic that a service named after a person pretending to be a machine
may be subverted by machines pretending to be people?

