

Human Workers, Managed by an Algorithm  - prayag
http://www.technologyreview.com/news/428440/when-crowdsourcing-gets-outsourced/

======
huahaiy
It's all about how much you pay the turkers. I have done many studies on mturk
and the quality of results have been very good so far. My secret? Pay people
equitable wages.

I always do a small pilot study to find out how long on average does my task
take people to do. Then price the HIT so that people can get at least $6 an
hour, sometimes much more.

This way, I get high quality results very fast. For example, I posted 100
assignments at 11am today, each took about 10 minutes to do, and now it is
1:40pm, and I have already gotten all the results I need.

Compared with having to pay $20 for people to come to my lab for an hour,
mturk is heaven.

~~~
anandkulkarni
That's a terrific strategy, and that's one of the strategies we apply
internally here at MobileWorks.

Treating folks fairly is the right way to get good results. We tie price to
task time explicitly, to prevent folks from underpaying MobileWorkers. This
makes sure results come back right.

You can read our IEEE paper for details:
[http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-
mobilewor...](http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-mobileworks-
ieee2012.pdf)

~~~
technotony
That is great. When I was working in the Philippines for a microfinance bank I
had some of my staff work on mTurk during a quiet period and they were barely
earning local minimum wage from the work. Croudsourcing work like this offers
great potential to get income into poor, under-employed regions of the world
but we can't let it become the new sweatshop.

------
kghose
Full circle.

Remember that the original 'computers' were people who performed computations.

If you recall Feynman's self indulgent book 'Surely you are joking' he has a
chapter where he describes how they would write algorithms which were then run
by humans operating adding machines while working on atomic bomb related
problems.

~~~
danso
Is this related to the claim that Feynman was one of the original forefathers
of parallel computing?

~~~
kghose
Hmm, perhaps: there is a passage somewhere about how they set up a scheme
where instead of the one person doing all the different computations they set
up sequences where each person would only do one computation but for different
problems which would circulate through the pool in a staggered fashion of some
such.

------
orangethirty
Whilst waiting for my turn to order on a local subs restaurant, I saw
something familiar. I was watching Big O live and in action. There were two
cashiers. Each one greeted the customer differently. The lady on the right was
faster, because she would ask yes or no questions. Then she would ask for
payment type up front and prepare the ATM/credit card terminal in advance if
the client said ATM/CC. The other cashier would just ask random questions in
no particular order, and would wait until the last minute to prepare
everything for the payment part of the transaction. She never had the pen at
hand. Ever.

Two people. Same training ( I assume ), and two vastly different algorithms
operating for the same goal. Imagine if a computer would take in the order. It
would closely match the lady on the left POS terminal. She was as close as a
computer anyway.

Her routine went a bit like this:

    
    
        (start transaction
            (greeting
                (choose sub type
                     (type of bread (white or wheat)
                          (any deviation from the STD ingredients)
                               (? combo meal))
                (? change chips for cookie)
                (? change soda for bottled water)
            (type of payment
                 (prepare pen & adjust terminal)
                (? ATM 
                   (get card ( swipe ( punch in numbers ( ? accepted || declined (goto start <-- ) give customer card back )))))
         (end transaction)))
                   
    

_I just realized how lispy this came out to be. :)_

------
rdl
Interesting that MIT Tech Review wrote an article on this, since a lot of
great research (at various universities, including MIT) is now a lot easier
due to easily available human task workers. Hopefully this will make research
easier, maybe reduce grad-student drudgery, and speed up scientific progress
for everyone. (Task work is useful for business, too, but it seems
particularly useful in the same ways cloud computing is useful -- the ability
to rapidly spin up a short-term task to handle load, without persistent
staffing costs.)

~~~
dchichkov
Unlike turkers, grad students produce much cleaner data and also see the data
first hand. Which is important. I consider mturk/offshore as a necessary evil,
rather than a panacea that will "make research easier and speed it up".

If data quality matters and you can avoid mturk - avoid it.

~~~
aantix
Seems like you're painting with a broad brush. Design of HITs on Mechanical
Turk is crucial. Implement gold standard questions per assigment to ensure
that you're not receiving spam from a bot.

~~~
anandkulkarni
It turns out this is one of the core problems with Mechanical Turk that drives
folks to MobileWorks instead.

In Turk, users spend so much time implementing protection against malicious
workers and bots that they need to develop real expertise in crowdsourcing in
order to be able to outsource their work effectively.

We've filtered workers more effectively up front and reduced the incentive to
cheat substantially, so things largely just work.

------
EvilTerran
I'm immediately reminded of Manna:

<http://marshallbrain.com/manna1.htm>

~~~
anandkulkarni
Let's hope it doesn't end up that way! The article only briefly mentions it,
but we've worked hard to make sure that even if tasks are algorithmically
generated for MobileWorkers, they have the opportunity to collaborate on task
execution in a way that's much more similar to a virtual office than to being
a cog (persistent worker-to-worker chat, managers to support and explain
tasks, and more).

This makes a material difference in the quality of the final results, lets
people enjoy their work more, and lets individuals take ownership over what
they're doing.

~~~
EvilTerran
Oh, I'm not saying you're heralding some kind of dystopian future, it was the
headline more than the article that reminded me of the story. It does sound
like you're doing good work.

Incidentally, much as that story does drag on a little in places, it does
eventually present a more utopian part of that hypothetical future world --
one where post-scarcity has been embraced, and the technology is used to
increase freedom instead of restricting it.

------
nobbis
Add a couple more layers and you get... algorithms managing humans who control
robots to clean for humans:

[http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1](http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1)

------
patdennis
_According to company cofounder Anand Kulkarni, the aim is to get the crowd of
workers to "behave much more like an automatic resource than like individual
and unreliable human beings."_

Rationally, this makes sense. Emotionally, I find it repulsive. I'm not sure
which impulse is stronger.

I feel like if I were involved in this company, I'd have to focus to some
extent on providing a leg up to people in developing countries. Turning people
into (potentially miserable) cogs in a machine would weigh on me after a
while.

~~~
patio11
_I'd have to focus to some extent on providing a leg up to people in
developing countries._

Get them paying jobs which involve nothing worse than sitting at a computer
screen and you're already more effective than, oh, approximately all charity
and foreign aid. After you have the jobs you can even try to make them fun
jobs, but un-fun jobs beat 3rd world poverty by _quite a lot_.

~~~
patdennis
I don't think fun jobs are the goal. I think merit based opportunity is the
goal.

------
DenisM
Frederick Taylor would be proud:

<http://en.wikipedia.org/wiki/Scientific_management>

------
stfu
Interesting article, but somehow the comment section turned into a
MobileWorkers commercial?

