

TC50: CrowdFlower Crowdsources Mundane Labor To The Cloud - ujjwalg
http://www.techcrunch.com/2009/09/15/tc50-crowdflower-crowdsources-mundane-labor-to-the-cloud/

======
snewe
Direct link:

<http://crowdflower.com/>

Their quality control could eliminate the quasi-quality control that I now
perform with MTurk. I set up a task like "get bio from webpage" and send it to
3 unique workers. I then run the results through a script to check whether
they agree. If they do, accept all (pick random one). If they don't agree hand
check (!) and possibly reject one of the jobs.

~~~
guywithabike
This is the sort of thing that CrowdFlower does automatically, except it takes
the quality control aspect to a new level. It takes into account the history
of the worker (were they accurate workers in the past? etc.) and also handles
automatically rejecting workers who fail too many gold questions.

Speaking of, that's one of the cool bits about CrowdFlower, too, is that you
can define "gold" questions. These are questions to which you know the answer.
Workers will see these as if they were any other task. We then give them
feedback about how well they're doing. If they keep missing the gold
questions, then we mark all of their work for your job as "tainted" and their
answers don't go into the aggregate calculations, which tell you what the
users agreed on the most.

So, for instance, in a job where we pull in tweets about, say, #kanyewest, and
ask workers to judge the sentiment of the tweet (positive, negative, etc.), we
could take some obvious ones, like "Kanye West is a big jerk!" and mark it as
gold with an answer of "Negative". Anyone who says "Positive" is probably
yanking your chain. So we trust them less.

Of course, gold questions don't work too well for subjective questions, like
writing short paragraphs.

I hope this gives you an idea of what CrowdFlower does in terms of Quality
Control. I'm only the front-end guy, so I'm not too knowledgeable about all
the math behind the scenes. All I know is that there's lots of it. :)

(I'm a front-end developer for CrowdFlower. I don't represent the company's
views. Except in regards to bacon. I represent those.)

~~~
snewe
Sounds like a very good refinement of the reputation score in MTurk. For
example, I usually only request workers with 95%+ acceptance rates.

The "Gold questions" sound like another refinement of the tests you can give
Turks: if you have an image processing task, you can require that a worker
pass a series of tests and effectively become certified before attempting your
job. Are you guys trying to replace that system or complement it?

------
howsta
IMHO a lot of tasks that they talked about seem to be be repetitive, large-
scale tasks that already can be broken into small pieces. For instance the
sentiment analysis demo (each tweet is a task) or image moderation (each image
is a task).

------
byoung2
_CrowdFlower will price the task based on the amount of time it takes to
perform the assignment per unit (the user does the sample task). The startup
will break the task into units that can be performed by a single person and
price the task accordingly._

How do they break tasks down into individual units? That seems like a tough
thing to automate, so they may have a bottleneck there when they try to scale.
On the other hand, they could have an upsell opportunity if they can sell
people by handling more complex tasks that a "virtual project manager" would
break down for them.

~~~
ujjwalg
I dont think it will be too difficult to automate for mundane jobs. They will
need a database to calibrate and then use that calibration to automate the
process. There will always be deviations but the deviations can improve with
more datapoints.

------
ujjwalg
I think its a brilliant concept. I think there is a huge market for mundane
jobs and no specialized place to go for. Guru, elance, odesk are all for
project work. Amazon turks is too spread out. They actually have found out a
niche, now it is just going to be about implementing it. I will definitely be
giving these guys a try.

~~~
auston
I agree with you, I am currently paying my friend to complete some mundane
tasks for me & will be looking into replacing him with this.. lol

~~~
guywithabike
Ha! Man, I'd love to crowdsource my laundry. :)

(I'm a front-end developer for CrowdFlower. I don't represent the company's
views. Except in regards to bacon. I represent those.)

~~~
auston
Ha!

No I meant, I have him calling people & asking questions (& putting it into a
spreadsheet) - Crowdflower can handle this, right?

------
ig1
I'd use them, mturk is a pain to use directly if you're outside the US.

------
rabidsnail
How is this different from amazon mechanical turk?

~~~
znbailey
There was an explanation of this during the TC50 presentation and in the Q&A
afterward.

From my simple understanding, this is a management application layered _on
top_ of the Mechanical Turk platform which is able to provide more
robust/advanced management and analytics in terms of tracking the quality and
speed of repeat turkers, as well as being able to integrate with other
commoditized/outsourced labor systems. Due to being able to maintain more
state concerning tasks and turkers, they are able to provide valuable
intelligence about how trustworthy and accurate the results are.

It was very interesting during the Q&A when Bradley Horowitz asked the company
rep if he thought that Amazon might simply do this themselves - but the
presenter stated firmly that they had a pretty good personal relationship with
the people at Amazon and they took pride on the turk service as really being a
black box type of platform.

Regardless, the main problem this business faces (which was handily pointed
out by the experts) is the issue of gaining traction - how do you make big
businesses aware of this as a possible solution to their labor issues? Even at
MySpace where they pay 300 people to moderate photos for obscenity they have
not realized they could much more effectively and cheaply be leveraging MT for
this task.

~~~
sachinag
MySpace can't use MT for the task because every picture sent to MT would be a
potential "OMG PORN ON MYSPACE FILM AT 11" story. They have to keep their
screeners in-house to deal with potential publicity issues. Trust me, the
MySpace people (as would any moderated community) would _love_ to use MT for
this stuff; moderation is no one's idea of a core competency.

~~~
guywithabike
Content moderation is one of the things that CrowdFlower does really well --
and yes, you can have workers search for adult content (At least on Mechanical
Turk. I'm not sure about our other labor pools. In any case, you can
definitely use CrowdFlower for filtering out adult content.)

You'll probably want to make sure your job description mentions that, though!

(I'm a front-end developer for CrowdFlower. I don't represent the company's
views. Except in regards to bacon. I represent those.)

