
How Clara Labs (YC S14) Is Using Humans to Build AI - marannelson
https://blog.claralabs.com/cooperative-intelligence-5c59960c2d95#.9fvitxue8
======
PameVls
This is SO interesting! I am surprised by how companies are trying to solve AI
problems... Machines normally take you 80-90% there, and using humans for that
10-20% left is an amazing idea. It's also a great monetization model for many
companies.!!! Also, Clara is awesome :)

~~~
jasonlaska
We think it's a good idea too ;) Machines and humans truly have different
talents: machines are great at memory, keeping track of state, distributing
information while people are great at understanding subtle nuance in natural
language.

~~~
visarga
And in another 3-5 years, AI will start understanding subtle nuances too
(probably) judging by the number of papers in computer language understanding.

~~~
jasonlaska
Even modern methods today (deep recurrent networks, etc.) can do pretty well
with these kinds of tasks (a very large ontology for instance) if you have
enough annotations of the nuance!

~~~
YeGoblynQueenne
>> Even modern methods today (deep recurrent networks, etc.) can do pretty
well with these kinds of tasks (a very large ontology for instance) if you
have enough annotations of the nuance!

Where do you get that from? NLP, with neural networks or not, stays as safely
away from meaning as is humanely possible while working in an area very
closely connected to it.

Also, ontologies? Very few people are interested in that nowadays, though that
includes the team that made Watson. The push is instead to do away with all
that and rely on statistical approximation.

------
kandalf
To me this seems like the right approach - either you start with some massive
amount of data that's not quite adapted to the problem (think Google), you
start with full automation and have to basically write the decision tree
yourself, or you generate the appropriate labeled training data like this.

However, it seems like there are some scale issues if you start upmarket like
Clara Labs has been. I wonder if there's benefit in having a cheaper more
mass-market version as well that can be used to generate larger amounts of
data and test algorithms better?

~~~
jasonlaska
It's certainly possible. One advantage of our setup is that rather than
getting ok -to- noisy labels from customers, our CRAs understand the end-goal
of the application and generate pretty great data. We are also able to
incentivize them to produce fewer errors.

------
jasonlaska
We've posted a follow up on the humans behind Clara here:
[https://news.ycombinator.com/item?id=12074657](https://news.ycombinator.com/item?id=12074657)

------
jasonlaska
Hi! I'm the author of the post and happy to answer any specific questions
here.

~~~
kayhi
How much human tagging time does it take to process 1000 meeting requests or
more general how does the annotation side work?

Do you have people overlap items since there can be a fatigue issue as you
mentioned in the post?

~~~
jasonlaska
Clara has a 1-hr SLA for the processing of an incoming message. While I cannot
give numbers on the speed of annotators (or volume), I can say that our
platform is designed to enable quick and accurate work via incentive
mechanisms. We avoid fatigue in part by making it easy for CRAs to navigate
and work with data. Wrt to overlapping annotator schemes, these are known to
be effective. We'll be writing more about how our human backend works in
future posts.

~~~
kayhi
Thanks, in the future post would also be interested in the trade offs between
building internal system vs. using a 3rd party (maybe you do?) like mechanical
turk and the incentive structure.

~~~
jasonlaska
Our follow up on the humans behind is here:
[https://news.ycombinator.com/item?id=12074657](https://news.ycombinator.com/item?id=12074657)

------
maxhm10
This is AWESOME. Human+Machine is always going to be better and stronger than
machine alone. Keep it up! I wonder how this can impact the amount of people
that can participate in crowdsourced projects.

------
intenex
Would love to integrate the ML and context aspects of Clara into our existing
assistant team. Curious if that's something that's doable, or if it's just a
package deal of their trained operators + AI.

But yeah, basically awesome and everything we'd need. Being able to go even
more in-house and specialize our personal assistant and equip them with
augmented capabilities here would be even more awesome.

------
jgalt212
Very similar to x.ai, although x.ai seems to be currently constrained to one
problem domain (meeting scheduling). However, I'd bet they're looking to move
be beyond just this.

[https://x.ai/](https://x.ai/)

------
harshnisar
Relevant: [https://techcrunch.com/2016/06/07/crowdflower-
series-d/](https://techcrunch.com/2016/06/07/crowdflower-series-d/)

------
dreeves
Related: [http://messymatters.com/ai-plus-ui/](http://messymatters.com/ai-
plus-ui/)

About how AI often needs a good UI to handle the last few percent of cases.

------
giardini
Navigating to

[https://claralabs.com/](https://claralabs.com/)

freezes my Linux system - a reboot is necessary. Is this a SPAM blog post?

------
kespindler
Reminds me of the writeup a few days ago about Google's machine emotional
intelligence for email suggested replys - except Clara is doing it (and more)
with a team the fraction of the size.

The business solution here is very clever. As an engineer, my instinct is to
fully automate any system. Clara is a great example where the most effective
answer is getting the 90% solution, then going the last mile with humans to
bring it up to 99% (making that up) accuracy, with most of the same savings.
Imagine similar solutions for customer support (especially over website chat
bars. Some systems like this already exist.)

------
Hydraulix989
Contemporary ML is really (and by that I mean human-level) good for some
things: classifying images, guiding a car along prominently visible lane lines
in perfect conditions, predicting timeseries that have latent patterns.

It's NOT good at the general "bot" problem (or should I say "true AI"
problem), and some "features" of human assistants like EQ, generalized task
completion, and complex planning are decades away from getting solved.

I would even argue that a mechanized/automated Clara would HAVE to pass the
Turing test in order to work.

~~~
jasonlaska
Your last comment on the seemingly circular relationship between Clara and a
"turing-test-passing bot" is especially salient. This reference (found in the
second footnote in the post)
[http://www.ijcai.org/Proceedings/95-1/Papers/125.pdf](http://www.ijcai.org/Proceedings/95-1/Papers/125.pdf)
has a lot of interesting perspectives on the role of machine learning vs.
human intelligence.

~~~
Hydraulix989
I am going to read that actually, thanks.

On the note of bots and your denouncement of the Turing Test as a metric, I do
remember an article on here a couple of days ago hinting that one of the
biggest frustrations of customer service calls is dealing with scripted
responses, bots if you will, instead of real humans.

I do wonder whether alleviating this user frustration will amount to passing
some "similar" but possibly weaker test.

~~~
jasonlaska
"The customer service test" \-- I like it.

------
yeukhon
Side note: your website's Javascript is broken. "i" is undefined.

