
Why We Do All-Hands Support at Olark - bcx
http://www.olark.com/customers/why-we-do-all-hands-support
======
fishtoaster
I worked at a SaaS startup that did this, and found pretty similar benefit:
quicker fixes, and a better understanding of real customer needs.

Eventually we made it optional though, because we found that a minority of our
developers _really_ disliked it. Which I can understand- not everyone enjoys
interacting with customers and answering dumb questions like "can you reset my
password?"

I personally found it really valuable, though. After the hundredth password
reset request, I was pretty motivated to make our password reset function
easier to find. :)

Some pitfalls to be aware of, though:

1\. This only works when the people doing support feel empowered to improve
things, either because they can do it themselves or because they feel they are
listened to by whoever can. For example, I can say to our founder over lunch:
"People have been complaining about X a bunch this morning. How about we do Y
to address that?" Then we'd discuss it, and I could build it that afternoon.
For smaller fixes, it was often just posting on our group chat: "Hey, I'm
fixing A by changing B" and merging it if no one objected.

Without this ability, though, it could easily get frustrating. If you're being
forced to deal with repeated customer issues you have no power to address, it
can feel more and more like an unproductive burden.

2\. It can have trouble scaling with the complexity of your product. We were
targeting small businesses when I started, and I was able to answer most
questions about our product by myself. As we got bigger, though, and started
introducing more complicated features aimed at the needs of larger companies,
there were more and more features that I didn't have a solid grasp on. I
probably could have kept the whole product in my head a lot longer if I were a
full-time support worker, but as a dev I only did 5-10 emails per day,a nd
didn't have the time to devote to learning some of the more esoteric aspects
of our product.

3\. Not everyone is cut out for customer service. We tried to hire good
developers with personalities we wanted to work with. That often, but not
always, overlapped with the type of communication skills necessary to be an
effective support person. We had a few developers who lacked the patience to
deal with slower customers, or weren't really good at explaining things
clearly to them.

~~~
djloche
>" If you're being forced to deal with repeated customer issues you have no
power to address, it can feel more and more like an unproductive burden."

You have summed the standard customer support experience so distinctly.

~~~
fishtoaster
Yep. I worked tech support part time in college- I remember. :)

When implementing an "everyone does tech support" system, though, remember
that at least some of your employees won't have much trouble leaving if they
find themselves hating the experience. When they have to deal with customer
complaints, but can't do anything to fix them, they may well move to a company
where that's not the case.

------
kyrra
National Instruments does something interesting with their customer support.
NI is one of those companies that hires mostly college grads then tries to
keep people around forever. If you are a college grad hired into NI, will will
spend 6-12 months in their customer support division. This applies to almost
everyone regardless of degree.

They do this for a few reasons from what I understand. It helps indoctrinate
newhires to the NI culture. It also helps the people figure out what
department they will enjoy and be a good fit into, as they will get exposure
to a large part of the companies product portfolio.

~~~
Serow225
The MathWorks (makers of MATLAB & Simulink) do this too. For their new hires
coming from an MSc/PhD, you spend the first 12-24 months in the Tech Support
group as a Tier I with a rotating schedule of 1 week of phone/email support
and the next week spent working on a project of your choice with one of the
many Dev/QA/BaT/doc/etc groups. The idea is that once you've completed a
handful of different projects, you've had a chance to figure out what you like
to work on, what group you'd like to work with, and the groups get a chance to
see what it's like to work with you for several months on a real project. Once
you've been there for 12-24 months you typically apply for a job somewhere
else in the org, although some people find that they enjoy doing support and
move up into Tier II/III. It's not like doing support for Comcast, you get to
talk to a lot of smart people with interesting/challenging questions and
issues :)

~~~
_pmf_
As Mathworks customer, I've come to hate this, since this involves talking to
clueless people with superficial information. Every question that involves
something slightly more complex is dispatched to deeper support levels and
regurgitated in a more or less screwed up way to me by the tier one guy/gal.

~~~
Serow225
I'm sorry your experiences have been less than positive. When I worked there I
followed the path that I described above, and the other Tier I folks I worked
with were nearly all smart and helpful. And whenever the customer had complex
support questions and it was clear they were clued-in themselves, we usually
were able to hand them over to the Tier IIIs to work things out directly
between them.

------
brianpgordon
This is in pretty stark contrast to yesterday's article about devops:

> Somewhere along the way, however, we tricked ourselves into thinking that
> because, at any one time, a start-up developer had to take on different
> roles he or she should actually be all those things at once.

> What began as an experiment aimed at increasing software quality has become
> a farce, where the most talented employees are overworked (while doing less,
> less useful work) and lower-level positions simply don't exist.

> Large companies love this, as it means they can hire far fewer people to do
> the same amount of work. In the process, though, actual development becomes
> a vanishingly small part of a developer's job.

[http://jeffknupp.com/blog/2014/04/15/how-devops-is-
killing-t...](http://jeffknupp.com/blog/2014/04/15/how-devops-is-killing-the-
developer/)

I certainly wouldn't be happy with a developer position that made me do tier 1
customer support. That stinks more of "absurdly cheap" than "connecting with
customers."

~~~
sunir
My experience says that if you're responsible for building a product, it's way
cheaper to be directly in touch with the end customer so you can fix what
you're building. It's better than having a legion of product managers, sales
people, business owners in between you and the user.

If you're a developer who is oriented around building products that sell and
are loved by customers, it's pretty addictive to talk to customers to find out
what they think of what you built.

~~~
quanticle
I disagree quite strongly. In theory it works like you suggest - developers
see problems, and they're motivated to fix them in a permanent fashion so they
won't ever have to deal with them again. In practice, there are multiple
equilibria. The good equilibrium is what you describe and what I've summarized
above. The bad equilibrium is that the team is short staffed and developers
are too busy putting out fires to actually implement the architectural changes
required to prevent the fires from starting in the first place. If you're
building an organization where developers are expected to do operations work,
you _have_ to think about the latter case. If you don't, you can very easily
fall into a trap where your core infrastructure stagnates because developers
are too busy coming up with the next hack to fix the ongoing outage to step
back and think about broader changes that will make the entire system less
painful in the future.

In fact, this is one of the things that Steve Yegge cites in his famous essay
[0] about how Google, organizationally, is better than Amazon:

    
    
        And their operations are a mess; they don't really have SREs and they make 
        engineers pretty much do everything, which leaves almost no time for coding - 
        though again this varies by group, so it's luck of the draw.
    

At a small company like Olark, they may be able to get away with this because
their product is small, their team is small, and so by necessity they have to
do everything. But once their product grows, or their customer volume grows,
it's very easy to get sucked into a very painful trap by having your engineers
be your ops and your customer support.

[0]
[https://plus.google.com/+RipRowan/posts/eVeouesvaVX](https://plus.google.com/+RipRowan/posts/eVeouesvaVX)

~~~
sunir
As they say, there are an infinite ways to fail but only a few paths to
success. If your engineering is so out of control you're constantly in crisis
mode, you have more important problems than customer empathy. I think that's
an orthogonal issue.

It's not about asking engineers to do support because you don't have any
support capacity. At Olark, we also have a dedicated customer support team.
Same thing at FreshBooks where I used to work that also does all hands
support. These teams are professionals, and are responsible for ensuring we
have world class support practices.

So, we don't ask the engineers (or marketing or design) to be on support
because no one else will do it. It's just the way we keep everyone close to
the customer.

The Jeff Bezos story in Steve Yegge's post is exactly the reason why I like
working at companies that keep engineers close to the customer. It cuts down
the BS of opinionated product people thinking they know what customers want
when they don't have real experience with customers.

------
bcx
It's pretty interesting to think about how to scale rotations of All Hands
Support. It's one thing when you are like 4 people, but as you build a company
there are a lot of resource constraints to think about. What's the biggest
company you guys have seen that still does rotations?

~~~
jorts
I believe Zappos makes all new employees do a one time two-week rotation on
the support team, regardless of their position.

~~~
bcx
I've seen it done at on-boarding with companies like Freshbooks and Zappos,
but I am not sure how often that experience is renewed. (i.e. the rotation
aspect being more than one time)

Engineers at Zappos do support for just a few hours a year doing peak season
after that onboarding. With a program called "Holiday Helpers" \-
[http://blogs.zappos.com/blogs/zappos-
family/2011/12/20/holid...](http://blogs.zappos.com/blogs/zappos-
family/2011/12/20/holiday-helpers)

------
iamleppert
This is just a note but you guys really need to work on making your chat
thingy less annoying. I'm reading this on an iOS device and there was no way
to dismiss the bouncy-puppy chatbox that was begging me to talk to someone. It
was so distracting that I rage quit your blog post.

------
digitalabyss
I’m a IT department of one so I get a lot of those "reset password" requests
and have to do a lot of hand holding I feel I should not have to do. Its 2014
and a certain level of computer literacy should be expected of you if you want
to function in corporate America. (or whatever country you are from). I have a
Masters of Science in IT Security, a B.S. in electronic engineering and my
RHCE with 10 years experience and feel that being pulled away from "real work"
to reset a password is extremely annoying. However its a lot less annoying
when the client services team is not available and the receptionist passes the
call directly to me. In most cases the client tends to be very friendly and
thankful and makes you want to help them.

The problem with this is geeks are generally not business people and do not
know if the client is up-to-date on their billing or if they are notorious for
trying to get free work and need to be kept in check. When I do take personal
interest in solving a clients issue I risk promising things I should not be
promising them or putting way more time and effort into a client who is
already behind on billing than I should be.

I also have a very hard time quoting a client an appropriate amount. I work
for a interactive and design agency and when you work with us you have a team
of individuals working for you including but not limited to an account
manager, producer, designers, art directors, user experience, strategy, front
end developers, back end developers, system administration in some cases.
Having a team of 5-10 specialists working for you is expensive. Just having a
1hr meeting with everybody is probably going to get billed at around $1000. I
feel really weird giving clients numbers like this and responding to the
sticker shock.

------
weixiyen
Highly doubt this scales past 30 employees unless your customer base is really
small. Customer service should really be summarizing concerns, and you should
verify the concerns by looking at data to see if it's really a concern worth
alleviating.

~~~
wdewind
Etsy does this (or did while I was still there) at 500+ employees. While they
have dedicated CS reps, they also rotate everyone in the company. My
experience with it was (literally) mixed.

-As an eng I was frustrated by it because I felt it got in the way of "real work" (I don't know if this was right or wrong - it just felt frustrating at the time).

-As a manager, I was frustrated by it because my engineers were constantly disappearing for customer support rotations.

But then I watched one of my friends literally eliminate 400+ hours a month of
customer support time (as well as the less measurable corresponding customer
frustration) by writing a small tool in an hour only because he had that first
person experience with the issue.

~~~
ziziyO
I had an inverted experience like this at my job. My job title is Technical
Support. I do internal and external support for our company. We use
Salesforce. By sheer luck I convinced my boss to allot me full system admin
access to our Salesforce instance. I've gone to town improving our customer
support workflow for our 100+ phone and e-mail reps by creating small tools
and tweaks using Apex/VF Pages among other things.

I had no prior experience with Salesforce when I first requested access. I'm
no Salesforce Architect, but I have a great grasp on the internals of it now
all because my boss had a little faith in me.

------
jorts
That's a great idea. I'd be really interested to see how this works out as you
scale. How do you continue to consolidate and relay feedback from customers to
the rest of the team that isn't on the support rotation?

~~~
bcx
Consolidating feedback is a problem for any team at scale regardless of who is
doing the support. Dedicated customer service teams have a more complete view
of what's going on with customers, than the average rotated employee, but at
the same time both need a way to bubble up that feedback to the product team.

More importantly, it's essential to understand the difference between the
problem you head about today, verses the problem that customers have been
facing for a long time. (I do think that support rotations cause a bit of a
recency affect in prioritization for better or worse)

In the long run my argument is the closer you are to your customers the better
insight you'll have in general. Because we all make a ton of decisions that
aren't clearly specified, and we want everyone on the team to have the best
intuition for the customer when they make those decisions.

------
sunir
I love this talk by Kevin Hale about their all hands support at Wufoo.
[https://www.youtube.com/watch?v=fAnl8ZGHVXU](https://www.youtube.com/watch?v=fAnl8ZGHVXU)
(alternate.
[https://www.youtube.com/watch?v=wYIH2qTizUU](https://www.youtube.com/watch?v=wYIH2qTizUU))

The most interesting thing is that they gave every engineer time to fix the
$#@! they heard on support. 30% of their effort was on internal tools.

Disclaimers. I also work at Olark. I also think Kevin Hale is (almost)
perfect.

~~~
emilsundberg
I love the talk. It's too sad to see Wufoo these days. All their great ideas
about support disappeared after selling it to SurveyMonkey.

------
JoeAltmaier
It means you hire folks who can do customer support. That's definitely not an
introverted engineer. Sounds like the benefits might even be worth the extra
hiring filter.

~~~
rcfox
As an introverted engineer, I beg to differ. If you can't spend 10 minutes in
a very directed conversation with a stranger who wants to use your product,
that's an attitude problem, not a introversion/extroversion problem.

Introverted doesn't mean anti-social or shy.

------
jefftchan
I think doing all-hands support in a way that's efficient (not taking too much
out of engineer's time) and scalable requires a well-designed system to
support it. Quizlet, where I've worked before, does a good job of this. See:
[http://quizlet.com/inside-quizlet/quizlets-incredible-
feedba...](http://quizlet.com/inside-quizlet/quizlets-incredible-feedback-
center)

------
MisterBastahrd
That's neat, but when you've already optimized your password reset as well as
you possibly can, now you're just throwing time and money out the window
because of a dumb company policy and probably pissing off your most talented
developers too.

Some customers are just incredibly stupid. That's who Tier 1 support is
precisely for: the kind of people who would suffocate in a wet paper bag. I
walked into my manager's office one day and said enough. I developed 10
products used by over 250k users and I was the only developer at the company.
Never again was I going to reset another password. I followed best practices
and made it as easy as possible. Any further difficulty on the part of a user
was not my problem.

~~~
lsiebert
Some customers need hand holding.

One thing to consider is an inverse of customer support. Have your dedicated
customer service rep handle only repeatable issues with a script.

Then you just say, "I'm going to escalate your issue to our Solutions
Department, who are responsible for that type of issue. They will be able to
walk you through the process."

But really, whatever you do, if you do email/chat support I'd be feeding every
chat session/email exchange into an expert system, and trying to generate
responses when possible.

------
elliotz
It's a great policy. Stripe was doing this 2 years ago:
[http://blog.alexmaccaw.com/stripes-
culture](http://blog.alexmaccaw.com/stripes-culture) Anyone know if they still
do it, even after growing?

------
mfish
my company did this (out of necessity) back when there were ~10 of us. As a
young developer, I didn't know any better, but I dreaded it. I was SO glad
when we moved to a dedicated support team. Also, in my experience, if you are
hiring good people, a dedicated customer support person is going to give your
customer a MUCH better experience.

~~~
vvvVVVvvv
Obviously, but being exposed to real issues first hand will certainly give you
a better insight on why it should be fixed.

Some kind of reality check.

------
tphan
We considered using Olark for a client's website one time but, at the time,
their embed code didn't work properly with Google Tag Manager. Whilst we ended
up using another service, I did receive an email from Olark a few months later
saying that they had addressed the issue, which I appreciate.

------
batmansbelt
Not having to deal with customers is a huge perk for a developer.

~~~
fernandotakai
well, depends. i like having to deal with customers since that makes me able
to build a better product -- dealing with customers makes you realize what
they want.

(disclaimer: i work at olark)

~~~
imsofuture
It's also awesome to interact with people who are legitimately happy about the
stuff that you build.

------
paulhauggis
This sounds like it works well for Olark.

However, I've seen companies use this to stretch the responsibilities of the
development team to include customer support..and not hire anyone for customer
support.

It was a disaster and I hated it...and eventually quit because of it.

