
My Half Workday as a Turker - alceufc
http://www.cs.cmu.edu/~jbigham/posts/2014/half-workday-as-turker.html
======
zackmorris
I think this is a perfect example of how money today is worth more or less
than it actually is. Another way of saying that is that money is nonlinear or
even non-constant.

For example, say you are broke and wandering the streets of a big city and
really want to buy a bus ticket to another city for $20 or whatever. No matter
how much your net worth is, you can't get that $20 easily. You could try
asking various businesses to do some work for cash, you could try a song and
dance routine, etc, but your choices all amount to various forms of begging.

Meanwhile you can be sitting in an office somewhere and make that money in
less than 3 hours even at minimum wage, even if you do little or no work. You
can even sell something on craigslist if you’re home. I suppose in desperation
you could sell blood plasma, but that’s one of very few lifelines.

Nobody thinks $20 is worth much, but when you don’t have it, it’s very
expensive indeed.

To me, the root of the problem is whether you are resource rich or resource
poor. So the web, by virtue of being intangible, is almost quintessentially
resourceless. The trend seems to be lower and lower wages for increasingly
onerous labor. In other words, if you have money, you get a real world return
greater than the value of your money. But if you don’t have money, then
getting it requires an expenditure of resources and effort larger than the
value of the money itself.

At some point in the near future, acquiring money will be so expensive from a
labor standpoint that it will be cheaper to simply do things yourself and live
outside of mainstream society by bartering goods and services. This really
bothers me, because that shouldn’t be the goal of progress. The paradox is
that even though every new fiverr and mturk create more jobs, they lower the
value of work. Right now this is affecting developing countries by creating a
race-to-the-bottom economy, but the futurist in me looks all the way to the
end and sees how so many jobs today (especially the non-production ones like
administrative/clerical work) will eventually be automated by technology and
thereby make the purchase of capital through labor even more expensive.

Does anyone see a way out of this? Did I miss something fundamental?

~~~
presidentender
In the aftermath of the Bubonic plague in Europe, the shortage of labor led to
higher wages for serfs, eventually ending any semblance of what we'd call
'feudalism.'

I'm not advocating a mass die-off of human beings, but any other means of
limiting the supply of low-quality labor would have the same effect. If some
sort of back-to-the-land subsistence thing became popular among the youth,
those who didn't partake might find that a McJob paid more by virtue of there
being fewer people willing to do it. Sort of a "shrug" by the lower-class
Atlas instead of the elite.

~~~
scythe
>If some sort of back-to-the-land subsistence thing became popular among the
youth

This won't become popular because it means accepting a drastic decline in
living standards: you lose Internet access, public transportation, access to a
variety of goods, the rural area you move to might have a corrupt government
or an underfunded police department -- why do you think gun ownership is so
popular in small towns? -- you experience a lot of social isolation, etc.

This idea that people should move "back to the land" can only originate from
the deepest misanthropy and ignorance of the plight of the lower classes -- or
perhaps the most starry-eyed technocratic utopianism, if you think that
society can somehow provide poor people in disparate areas with a modern
lifestyle -- and I'm not holding my breath. Of course, we _could_ just kick
poor people to the curb like this; it seems to be our policy already, city
councils won't approve new housing for them, they won't approve the
construction of a Wal-Mart, expansion of transit services is always opposed by
a chorus of veiled racists, etc.

~~~
presidentender
I am a gun owner from a small town. What do I know?

~~~
dctoedt
> _I am a gun owner from a small town._

That comment plus your handle ('presidentender) might get you a visit from the
Secret Service ....

------
crazypyro
I used MTurk off and on through my teens[0] (15-18) and found that the
majority of my earnings came in through the 50 word or less summaries. You get
to a point where you've read enough PR department press releases that you can
write the entire summary (at least to the required standards, which in my
experience, is "barely comprehensible English") by looking at the headline. I
was able to finish 50 word summaries in under 30 seconds. They values range
from .25-.50 and they seem to come from a lot of submitters.

I've had a theory that out in the world somewhere is someone selling a guide
to using MTurk to create article summaries and the guide suggests the price
range of $.25-.50 because for the entire time that I've been on MTurk (many
years now), every time I come back the price range is the same with small
fluctuation.

Past the relatively low amount of summaries that are posted daily, money can
be made by watching the right forums where people post high paying HITs (the
name of the most popular forum eludes me at the moment) that are typically
5-15 minute surveys that pay .5-3.00 each.

By far, the biggest barrier to making a decent wage for me was the number of
available HITs. The ones that pay over pennies each are few while the ones
that are horrible time invests are everywhere. If I had an unlimited number of
summary HITs, I could have made well over minimum wage.

[0] Made about $450 over the time, but this was sporadic, random working
periods.

~~~
ski
I'm curious, have you found the ability to write 50 word summaries in under 30
seconds to be a useful skill later in life?

It sounds like good practice in concise communication.

~~~
Natsu
I'm surprised someone hasn't automated jobs like that, honestly.

~~~
andrewchoi
Isn't that what Summly was supposed to be? The startup founded by the 17 year
old that got acquired by Yahoo for a chunk of change?

------
incision
I dabble on MTurk every now and then.

It might sound strange, but I find certain kinds of survey type simple tasks
incredibly relaxing and some of the HITs clearly connect to what seems to be
interesting research.

Pretty much everything the author says here is spot-on and has been for a long
time.

MTurk is positively flooded with HITs for generating fake reviews for products
and fake content for websites. The best paying HITs by far are translation
tasks - transcribing Arabic and Farsi seems to pay the best ($0.30 or more per
minute of transcription).

I aim for academic projects with preference for things which require
qualification (simple math / reading comprehension tests usually) as they
generally pay quite a bit more.

I've earned an average of $0.27 per HIT and have never had one rejected.

Some of the more interesting / unusual tasks I've seen:

* Pinpoint various joints (shoulder, elbow, knee) in each frame of a clip showing a baseball player swinging a bat.

* Drop a pin on an estimate map location based on scenery shown in a short video clip (This was long before MapCrunch/GeoGuessr).

* Manipulate the camera in a scene of flat-shaded objects to bring them into 'correct' perspective.

* Choose 'preferred' structure designs (little houses) which appear to have been created by some sort of genetic algorithm.

~~~
VLM
"Some of the more interesting tasks I've seen"

I get distracted by categorization jobs, like look at this blueprint and if
there is a blueprint number in the title box, enter it. Well, I end up
admiring the print. Oh look a giant gear for some kind of mining thing. Why,
what an interesting looking architectural drawing.

Similar thing happens with categorize document numbers. Apparently some
municipality decided to turn millions of paper scanned pages of contract bids
into piles sorted by contract, so I'd end up sitting there reading all about
bids to put in some drainage culvert. Who would ever have guessed theres so
many steps and processes and backfill and compaction are critical and theres
so many inspectors... I mean you can half A a drainage culvert in like one
paragraph but if you want it done right its 50 pages of PDF.

These jobs are probably impossible without an excellent short term memory or
dual (or more) monitors.

The ones that annoyed me the most were "leave blank if no number" so I'd scour
the document for what seemed like forever just to make sure. Inevitably
leading me to becoming fascinated by drainage culvert design.

Its very D+D like. Whats behind this dungeon door? Oh, I see, a blueprint of a
giant gear from some mining equipment. Fascinating. Like archeology but I'm
getting paid for it. Well, not much pay.

------
spikels
Jargon Alert - HITs (Human Intellegence Tasks) are simply the individual tasks
(or units of work) that the worker is paid for completing.

~~~
tripzilch
Thank you! I had to ctrl-F down 75% of this thread because it wasn't explained
in the article either.

------
geronimox
"At the time, I was hoping Dmitry would reject my HIT so I could go on a
tirade about how unfair it was, and try to get him banned from MTurk or
whatever."

I found this quite amusing. You know why this HIT was approved? Because
requesters are also punished for rejecting too many HITs! He also would have
been banned had Amazon discovered the scheme he was pulling.

~~~
compare
This is a serious problem that requesters face. Often the workers who spam
everyone with fake work will report anyone who rejects their work as
retribution.

------
ChuckMcM
This was a great read. I found it particularly interesting that the author
came into each encounter 'cold', which is to say without much in the way of
additional tooling. They did install a word counter extension but much of
their tools were ad-hoc or non-existent.

That suggested that there may be an interesting 'toolware' market for Turkers.
For example, document editing; Lets say we build a simple document editing
platform (copy editing not typesetting) which includes components that are
dropbox like (shared storage), Google docs like etc. And design a workflow
around that toolset. So pick a HIT, get a document id, it shows up in your
shared storage, use the included tool to edit it, when you're done, click
'done', and have the whole thing resubmit back for evaluation. That might
allow you to focus on editing and not get hung up on a bunch of getting it
done issues.

Not sure how practical that is, but if there are common Turk workflows it
might be useful to build some common tooling for them.

~~~
_delirium
> That suggested that there may be an interesting 'toolware' market for
> Turkers.

In the early days of Mechanical Turk (2005-06), when almost all the HITs were
seeded by Amazon themselves and were of a few common types that kept being
replenished, there were browser extensions to greatly improve the UI for the
common tasks (the default UI is very bad). One of the more profitable tasks
was fixing the location of addresses on the new-defunct A9 BlockView (a
product similar to what Google would later introduce as Street View). They'd
present you about 10 photos of a streetfront that were supposed to be near an
address, and you were supposed to pick the one centered around the address
(e.g. the store or house entrance). Or else indicate none of the above, if the
alignment was seriously off so that the correct address was out of the frame.
If you used the regular website UI, you could make maybe $3-4/hr, but most of
this was tedious clicking: accept HIT, click a tiny radio button, scroll down,
submit HIT, request next HIT, repeat. The browser extension implemented the
obvious UI improvement: use the images instead of tiny radio buttons as click
targets, and then auto-submit and request the next one on click. I think in
one version you could also just hit a number 0-9 on the numpad instead of
clicking. With that extension I was making $20-30/hr for a little bit.

------
abakker
I've used MTurk to have interviews transcribed several times, and almost
always had flawless results. (sometimes jargon or names trips them up.) I've
always been incredibly impressed at the speed they get done, which is
substantially better than I've ever achieved doing them myself. I usually aim
to pay $10 for a 10 minute segment, which should amount to more than $10/hour
for a good typist.

------
pessimizer
An journal article I submitted a few years ago:

 _Amazon Mechanical Turk: Gold Mine or Coal Mine?_

[http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057](http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057)

------
danielweber
Last time I looked at MTurk, it was offering money if I would complete some
simple task, and, oh, by the way, download this crapware onto your computer
and run it.

~~~
crazypyro
To be fair, those types of HITs are forbidden by the terms (often removed
quickly) and, in my experience, tend to be rare.

------
sparkzilla
I checked Mechanical Turk as an example of how to _not_ to price for
Newslines, my crowdsourced content site. It seems that most publishers use
crowdsourcing as a way to save money, and while that may work for some tasks,
you will not get consistent work done.

Rather than crowdsourcing being a way to find cheap writers, content business
should see crowdsourcing as a way to find a diverse pool of writers who have
different interests or knowledge, who can dip in and out of work, and then pay
them fair money for their time.

I pay $1 per post for writers to write 50-100 word posts that follow a very
specific format. We provide feedback to the writer to try to make them make
money faster. The way I figure it is, if the writer makes money then I can
make money. If they stick with you, you don't have to train new people. People
won't leave after half a day.

We have already paid out thousands of dollars and our site is growing fast.
Writer satisfaction is high. We currently have a waiting list of hundreds of
writers wanting to sign up.

[http://newslines.org/newslines-rewards/](http://newslines.org/newslines-
rewards/)

~~~
e12e
Initially I found $1/[edit: item] strangely low as an example of "high pay".
Having a look at what newslines is actually about, I still think it is low,
but not as low as I intially thought (I'd assumed you wanted contribution from
domain experts, who'd I guess you'd price at at least 100/hour).

I'd assume anyone that can write well, should be able to make at the very
minimum 30/hour writing -- That leaves on average 2 minutes/newsline. I
suppose the lesson is that there are many skilled poor people out there.

(Note, this isn't meant as negative criticism, just some observations)

~~~
cm2012
You are dead wrong if you think any skilled writer can make $30 per hour. Only
copywriters and technical writers can do that, and those are not easy jobs to
get. Journalism, as an example, is one of the lowest paying professions.
Writing jobs on Craigslist are almost always unpaid internships.

~~~
e12e
I suppose I'm rather out of touch with the not-cs hourly wages in the US.
According to:

[http://www.forbes.com/sites/susanadams/2012/04/26/is-
journal...](http://www.forbes.com/sites/susanadams/2012/04/26/is-journalism-
the-best-job-or-the-worst-job-heres-new-evidence/)

Journalists earn (for ~2000 hours/year) between 15 and 28 (without a graduate
degree), while lower level CS started at around 25/hour. So I was clearly
wrong, but it would also appear journalism isn't especially low paying either?

[http://www.poynter.org/latest-news/mediawire/202644/gov-
stat...](http://www.poynter.org/latest-news/mediawire/202644/gov-stats-median-
salary-for-reporters-35k-52k-for-editors/)

Seem to mostly agree.

------
ccb_
Like Jeff, I'm a professor who works on crowdsourcing. For one of my current
projects, I am working on a tool to make the Amazon Mechanical Turk
marketplace fairer for workers. I have developed a Chrome plugin that tracks
the length of time it takes to complete a task, and a web site that aggregates
the information across many workers. Crowd-Workers.com allows workers to
discovery higher paying work by sorting tasks by estimated hourly rate.

Read more about the tool: [http://www.cis.upenn.edu/~ccb/publications/crowd-
workers-pos...](http://www.cis.upenn.edu/~ccb/publications/crowd-workers-
poster.pdf)

Try out our Chrome plugin and our web site: [http://crowd-
workers.com/landing](http://crowd-workers.com/landing)

------
lilsunnybee
It's been over a year and a half since i've spent any serious time on
Mechanical Turk trying to supplement low income, so some of my experience
might be dated. With a bit of a learning curve it's plenty doable to find
decently paying jobs in transcription, writing, etc. on MT, especially if you
have a strong grasp of the English language. But even with higher-skilled,
higher-paying jobs on that platform, the overall experience of working on MT
can often end up being plenty demoralizing and unfair in many many ways, such
as qualifying for skilled work and also being compensated for work performed.

The most frustrating jobs i've ever performed on the platform were usually
approx 5-10 minute tasks that seemed to be decently paying and
straightforward, asking you to categorize items based on some arbitrary but
seemingly simple system, but later all 5 or more submissions you might have
made could be blanket denied / payment refused. In that case not only do you
not get payed, but having HIT submissions rejected really seriously hurts your
reputation and what jobs you're allowed to apply for on MT; often many more
high paying jobs on the system use high acceptance rate and flawless work
history as the lowest bar for entry, and so MT workers can be screwed over
incredibly by having work rejected through no fault of their own.

To not get seriously screwed newcomers are somehow supposed to know which
types of HIT's to definitely avoid that could kill their rating, but without
enough information to actually make that judgement. A task might seem easy and
give the illusion that it's okay to send 5 or 10 submissions in a row, only to
later have them all rejected and serious reputation damage done.

I think it's interesting but also really sad how this mirrors a lot of the
unfairness and power imbalance in the larger working world as well. Reputation
is everything, but if you just happen to get unlucky to start out your career
working for clueless bosses / clients with unrealistic expectations, there's a
good chance future job and career prospects are going to be seriously
hindered, if not completely derailed.

Transcription work was usually very well paying i found, but doing one long
transcription task for a $30 payout is a huge risk the way things were set up.
Never mind that you worked for 4 or 5 hours on it; whoever assigned the job
can reject it for whatever random reason they want and you get nothing. As
nice as it can be to earn a nice wage for more high-skilled tasks, the
possibility of being denied any payment for a days work with little recourse
is really frustrating and demoralizing; the way Mechanical Turk is structured
if anyone's taking a negative financial hit for work done, it's almost without
fail going to be the worker.

Though this CMU professor obviously put some work into this write-up, i don't
see how somebody can possibly get an accurate picture of what it's like trying
to earn an income on one of these platforms by just devoting 4 hours and
extrapolating based on that really limited experience. It is really easy to
miss all but the most obvious and glaring problems that way, just from not
sticking around long enough to even run across them.

There are plenty of MT workers who have spent much more substantial amounts of
time on the platform, that are much better able to communicate benefits and
pitfalls for the everyday worker. They might not be professors at prestigious
universities like CMU, but having a decent amount of hands-on experience
should really be the low bar for discussing pros/cons of the platform
seriously; just as one example assumptions about longer-term take-home pay in
the write-up were pretty naive: many better paying tasks (especially web
browsing tasks and surveys paying more than a pittance) are often in limited
supply and cannot be repeated either at all, or only a few times per day /
week, etc. Even the best employers providing the most decent fairly-
compensated work come and go, making a steady healthy income sometimes very
difficult to achieve even in the best of circumstances.

Mechanical Turk has experienced a lot of success from the start largely
because they created a service that makes it so easy for virtually anybody
with internet access and a basic computer, to work from home and perform
simple tasks / other types of work when and wherever they want, and actually
get paid for it.

By any decent standards of developed-world countries the pay is shit though,
and worker protections and benefits are nonexistent; workers can be denied
payment and even have their reputation ruined for no good reason, with little
in the way of an appeals process. But even now services like MT are still a
pretty new thing. Maybe in 10 or even 20 more years most of the bugs will get
worked out, and workers of every skill level can be a on more equal footing
when negotiating with employers, settling disputes etc.

~~~
pjc50
If there's no real appeal process, what reason is there for any MT employer to
make any payouts at all? Honesty and not realising this about the system?

~~~
mpthrapp
There's an unofficial ratings system for those who post the jobs[1] that most
people who use MT with any sort of regularity (including myself back in the
day) use. If a requester consistently fails to pay out, they'll stop getting
as many workers who are willing to work on their tasks.

[1]turkopticon.ucsd.edu

~~~
lilsunnybee
That's a great point! Though still really hoping the default experience has
improved in the past year or so as well. Not every new or even semi-
experienced user is going to be savvy and aware of external rating systems, or
other peripheral tools to help improve the worker experience. I know i wasn't
(first time hearing about turkopticon, sadly -_-), but that doesn't reflect at
all on how qualified someone is to do the mostly very simple tasks available
to work on.

------
aceperry
Kind of reminds me of the saying, "pay peanuts, get monkeys."

------
bitlord_219
What a dehumanizing experience.

~~~
runj__
Yes, working: how awful.

I hope you're trying to be on the workers side but either way this is a
terrible thing to say. The way you look down upon people trying their very
best to make a living is not helping them at all.

~~~
andreasvc
It's dehumanizing to work and get paid little or nothing at all. You seem to
be reading much more in the comment you're replying to than was actually said.

~~~
iLoch
Depends on the sort of tasks you're doing. Someone above mentioned receiving
~$0.27/min of translations. This is a relatively high figure in comparison to
a lot of tasks, but if the task you're doing isn't entirely mind numbing then
being a Turker isn't as bad as it sounds. $0.27/minute is something like
~$16/hr - if you're working at a gas station with a computer handy, you'd be
able to more than double your wage. Are the tasks undervalued? Not sure it's
fair for us to say. The HITs keep coming in and they continue to be completed,
so something is working.

~~~
tormeh
Depends what sort of gas station you're at, I guess. My sister worked at a
gass station one summer. She was mostly alone on the job, kept an eye on
everyone who filled gas, made fast food, made sure the shelves were well
stocked with products and cleaned. Not much room for going to the toilet much
less sitting at the computer, but then her wage was something like 30 USD/h.

------
pc86
For those of you like me getting a white screen with no content, the direct
link is
[https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxv...](https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxvz18Lv-
ej8U9QXEgPlsg/pub?embedded=true)

~~~
robertfw
FYI, Privacy badger is what caused my browser to whitescreen, as it was
blocking the pass off to google docs

------
ggchappell
Interesting article. One hears about MT, but not so much from this
perspective.

It's also interesting that the linked page has two scroll bars (tried both
Chrome & FF).

~~~
stopbits
This page is just a container for a google doc in an iframe. Interesting CMS
strategy.

~~~
krilnon
It seems to work reasonably transparently most of the time. I was in one of
the author's classes in ~2010 and I think the class schedule was put online in
the same way.

------
mkhpalm
I wonder what pays more? Panhandling on the street corner or working on
mechanical turk?

------
alexchamberlain
Very interesting. Has anyone used Mechanical Turk to get work done?

~~~
jimmar
I've collected experimental data using Mechanical Turker. In my experience,
you have to do a lot of validation to make sure the data you are getting is
reasonable. You have to kick people out who aren't paying any attention at all
and just want to click through your survey to get paid. (Similar techniques
also have to be employed for undergraduate participants who just want to get
done with your survey so they can earn their 10 points of extra credit).

If nothing else, Turkers are good at making sure your data collection
instruments are working the way that they are supposed to.

~~~
pavel_lishin
> _In my experience, you have to do a lot of validation to make sure the data
> you are getting is reasonable._

Do you ask any 'trick' questions that are the cognitive equivalent of a
captcha?

~~~
jimmar
I haven't found the need to use "tricks." Basic questions work pretty well.
I've done a couple of things. 1) Ask, "Do you have any problems reading
English?" Some people with poor English skills, or people being honest, or
people not reading carefully enough will click "Yes" thinking that I'm asking
them if they speak English. 2) If sound is required, I've played an audio clip
and asked people what was playing. They have to be able to identify what was
happening from a list of words. This makes sure that they have headphones and
can speak English. 3) I sometimes will ask people an obvious question about
something that was presented on the previous page to make sure people pay
attention. For example, if they have to read an essay on baseball, and the
word "baseball" is mentioned 20+ times in a paragraph, I might ask them what
sport the essay was about. A surprising number of people miss questions like
that. 4) As mentioned by another poster, tell them to select a specific choice
(e.g. "Strongly disagree"). I try not to do anything too nitpicky out of
respect for people's time and because people aren't perfect.

------
pgl
Can anyone suggest any alternatives to Mechanical Turk that offer a better
experience for the worker?

~~~
garretthunyadi
Check out CrowdFlower. The workers are mturk workers, but I believe that
there's more attention placed on the worker.

