
A Review of Airbnb's SRE Hiring Process, from someone who just interviewed there - philiph
http://redbluemagenta.com/2012/08/15/review-of-airbnbs-interview-process/
======
drewcrawford
I'm going to say something potentially very stupid hoping someone will correct
me: I don't understand why technical interviews exist in the general case.

I mean, I understand why companies like AirBnB would like them to exist. What
I don't understand is why engineers participate in them.

I totally get selling yourself to the potential employer. Spending an hour on
the phone talking through a technical problem. Having IRL conversations where
you see if you "gel" with the team in a bunch of hard-to-quantify ways.

But why would any engineer in today's market take two days to respond to
asinine pet/trick questions that random engineers lob at them in order for a
_shot_ to spend additional time for people higher up the food chain to lob
hypothetically less-asinine questions in a way that makes them directly
comparable with the other 19 people jockeying for the spot?

Do companies like AirBnB believe that there are top-tier people with nothing
better to do? Or do they believe they have some perk that is different than
the perks of every other startup currently hiring that would motivate top-tier
people to create such time?

Maybe my perception of market demand leads me to overestimate it. I get so
many "please work for us" emails (from actual founders, even, let alone
recruiters) that my inbox is out of control. I've gotten a job offer while
writing this comment. I can't even reply at that scale. Perhaps one offer a
month is lucky enough to receive an actual e-mailed decline; I never even
consider doing a phone interview. Am I a precious snowflake, is my experience
non-representative?

~~~
borski
I don't write negative comments, but this was impossible to not respond to.
Honestly, you sound less like you're precious and more like you're
pretentious.

While I'm very happy for you receiving job offers all the time
(congratulations!), there are definitely reasons for an engineer to take an
interview; in particular, falling in love with what a company stands for. The
idea that all engineers are mercenaries ready to work for anyone and sell
themselves to the highest bidder is an aged one, and one I had hoped was no
longer true. If I fell in love with a company (as has happened in the past),
I'd certainly be willing to jump through some hoops to "prove" myself to them
- if that means going through an interview process so they can fall in love
with me as much as I have with them, so be it.

When I see a woman I like, I don't go up and tell her about all the other
women that want to sleep with me. Instead, I flirt, I jump through some hoops,
make her jump through some of mine, and ultimately establish a strong
connection. Or not; that happens too. Same goes for finding work.

~~~
drewcrawford
> in particular, falling in love with what a company stands for

People can purport to stand for just about anything, but unless there is a
track record there of standing up for [principle] in the face of [adversity],
it's just a politically expedient claim. There are wonderful exceptions, but
most ordinary people and most ordinary companies don't deduce their day-to-day
behavior from first-order principles, and at any rate, companies the age of
AirBnB don't have the track record to enable an observer to distinguish
principles from marketing claims.

> The idea that all engineers are mercenaries ready to work for anyone and
> sell themselves to the highest bidder is an aged one, and one I had hoped
> was no longer true.

This is not my view; nobody is saying take the highest offer, and I have said
absolutely nothing about money at all.

My question is: why do engineers participate in technical interviews? Insofar
as I understand it, your response explains why an engineer might (I would say
_improperly_ ) create a post-hoc rationalization and "fall in love" with the
company after-the-fact, but under your own logic it doesn't explain why the
engineer takes two days off to go in the first place. Assuming that "falling
in love" is even something you can do in two days, the candidate hasn't done
it at the point this decision is made.

~~~
borski
Thanks for clarifying.

I suppose my point boils down to the following:

1) If you already love a company pre-interview (which can happen, by the
actions of the company), then it makes perfect sense to take an interview.
Even in the short lifespan of most startups, there can be enough action and
culture shown that you can truly desire to work there / "fall in love."

2) Sometimes it's worth taking an interview to find out more information on
both sides - a phone interview is useful, but as you say, IRL conversations
are far more useful for the things you can't see on paper or hear over the
phone. I'm not saying AirBnB did a great job with this - they clearly didn't -
but that doesn't make all interviews useless. Without them, a lot of "bad fit"
hires would be made.

~~~
throwa
It is okay to love a company but note that the company never falls in love
with you as an employee. They always do what is in their best interest,
interpret as what is in the interest of the major share holders and not
employees or minor share holders.

That is why even when you kill your self for a product launch or their overall
success, they still don't hesitate to give you the sack if it is required for
something like cutting cost not to talk of when major adversity comes.

My point is be pragmatic and not too emotional with loving a company
especially if you are not a founder or major share holder. Many people have
been burnt by loving someone or an organization that does not have capacity to
return that love.

------
relation
_There was too much emphasis on tool choice/religion, Linux trivia, and
getting the ‘right answer’ versus solving engineering problems, where multiple
tools might have to be glued together to come up with one of many solutions._

airbnb seems to be missing the mark. tools come and go. the 'right answer'
doesn't matter as much as the reasoning.

the ability for one to assimilate him/herself quickly and productively into
the environment is what they should be looking out for.

------
joering2
Remains me of a recent almost interview I almost went through. I have 15 years
of experience in IT and basically over past 7 years I built two large
healthcare-related CMS systems. The company found me through one of clients
that uses my product for 5 years and is very happy with the value 2 cost
ratio.

So I go in and first question I hear. "can you describe how is TCP/IP built".
This is for web development position, you know code igniter, php, jquery, that
kind of stuff. I looked at them for about 45 seconds, got up and said "thank
you for your time" and left before they had a chance to say anything.

Dude, wtf, are you serious? Its not like i couldnt remaind something about
tcpip from the times of my b.d. 10 years ago, but seriously unless we are
going to invent new transportation protocol for the whole human civilization
to use, why on earth would you ask me that, when you hiring me to be a web
developer. What a joke!

~~~
ryanpers
leaky abstractions. It's important, but there is also a cargo cultism of
asking questions in this certain way that is imitated in form only from how
people envision google demands their engineers.

------
philwelch
I'm going to go ahead and say a few things, slightly in defense of Airbnb from
the perspective of a fan, an observer, and an occasional customer, and also to
get these points out of the way.

It's really hard to know the right way to hire SRE's/sysadmins/systems
engineers, from a startup's perspective. If your founders are highly
technical, they're more likely to be developers, which is a similar profession
but frankly different enough to note. In some cases they will be designers, or
non-technical people entirely. As your company grows, you need to hire
developers first, so at best, you find a (decent) solution to that problem. As
we already know, though, nearly every company that interviews and hires
developers still does a shitty job at it--not because it's their fault but
because it's a genuinely hard problem. It's only after you've grown awhile and
your developers get tired of doing all the ops shit that you think to hire
SRE's. But at this point, your culture and hiring practices are already pretty
set, and set by people who aren't actually SRE's. It can be fantastically
difficult to rollback this kind of momentum even if you do manage to get a
top-notch SRE or three onboard.

There's another side of this, though. It's both a root cause for the problem
and good news for Airbnb: Airbnb's competitive advantage is not going to be
having the best SRE's. They're not AWS, Heroku, or Google. So it's not really
a problem if they're bad at hiring SRE's, because _most_ companies get away
with being bad at things that aren't central to their business. What Airbnb
gets right, they get fantastically right. This isn't to say they shouldn't do
a better job hiring SRE's. They should. But they're still a great business
despite this.

~~~
noomerikal
I agree and would only add that of the many purposes of raising some $120
million is to put appropriately experienced individuals in a leadership
capacity. Appropriately experienced meaning management that understands how to
maintain a happy medium without bias towards development or operations.

------
georgebarnett
I'm not in a position to comment on Airbnb's process here simply because I
haven't experienced it, so this comment isn't aimed at anybody in particular.

I find it remarkable how many skilled engineers reach for the toolchain first
and then find problems to solve with whatever tool they have chosen. Even
worse, they often reshape the problem to fit a particular tool and then solve
it badly.

One of the downsides of the Github/BitBucket open repository movement is that
there are now zillions of tools for these engineers use to go looking for
problems.

Kind of reminds me of the database as a hammer story:

 _When I first wanted to spend some time learning about relational databases,
my then boss told me a story about database people, which I’ve found to be
remarkably true. The story is, RDB people have found the most beautiful,
wonderful, perfect hammer in the whole world. It’s perfectly balanced – not
too heavy, not too light, and swings just right to pound in a nail just right
every time. The grip is custom-made, fitted to the shape of the owners hand,
so that they can use it all day without getting any blisters. It’s also
beautifully decorated – encrusted with gemstones and gold filigree – but only
in places that won’t detract from how well it works as a hammer. It really is
the greatest hammer ever. Relational database guys love their hammer. It’s
just such a wonderful tool! And when they make something with it, it really
comes out great. In fact, they like it so much that they think it’s the only
tool they need. If you give them a screw, they’ll just pound it in like it’s a
nail. And when you point out to them that dammit, it’s a screw, not a nail,
they’ll say “I know that. But you can’t expect me to use a crappy little
screwdriver when I have a magnificent hammer like this!”_

    
    
      http://scienceblogs.com/goodmath/2008/01/22/databases-are-hammers-mapreduc/

~~~
dfc
_"Github/BitBucket open repository movement"_

Did you mean to say "open source"? I am not familiar with the github movement.

~~~
georgebarnett
What I mean is any piece of code that's open for re-use on GH/BB.

I've seen many repo's that aren't open source in the GPL/BSD sense but are
open source in the "here's the code now its your problem" sense. Sorry if this
has resulted in confusion.

~~~
dfc
That's a movement?

------
learc83
I really feel like most of these companies are just trying to emulate Google's
hiring process rather than finding something that actually works.

How many people have actually been hired after going to one of these? Every
job I've gotten has been through networking, back channels, and a token
interview.

2+ days of anyone's time is too much to ask without compensation.

------
MichaelGG
_I was asked how I could coordinate actions between many machines, and I told
them that I could probably use Chef to do a lot of that, which was met with a
huge red “no” response. I then had to play bingo and landed on the “correct
answer” of “Zookeeper.”_

I'm curious about this. I'm only vaguely aware of either system, but I thought
they solved rather different problems. Chef handles system configuration (like
things in /etc, or "configure memcached to listen on this IP and port").
Zookeeper is more of an API to deal with distributed locking/sync issues for
an application itself.

Was this interview question less of a "bingo" and more of a "one-side really
misunderstood something"? Or how do Zookeeper and Chef compete?

~~~
cparedes
Author here. They do, though for a certain set of problems, you can get away
with using things like Chef for coordinating lots of information between a lot
of machines.

My point, however, was that tools should _never_ be the focal point of solving
a problem - if you recognize the problem first as a distributed locking
problem, or as a configuration problem, then you can start deriving a solution
(and maybe that solution _might_ have Zookeeper in it, maybe you'll write a
new piece of software, maybe you'll use Chef, etc.)

~~~
ryanpers
One of the biggest problems as I noted in a different comment is that chef
doesnt solve _software_ deployment. Configuration management and "best effort
file pushes" is not really appropriate for running a website. Sure you could
do something with chef, but waiting 20 minutes for a rollback would be pretty
lame-o.

The issue here is software push and service orchestration. How do you do a
rolling restart of a cluster in an automated fashion?

As for 'configuration management', I feel like puppet, chef, etc arent catch
all solutions for 'i want my config to be the same on all machines' for the
above mentioned 20 minute cycle.

------
cglee
This breaks the golden rule of relationships: don't kiss and tell.
Interviewing is the first step in forming a relationship, and it looks like
Airbnb dodged a huge bullet by not engaging further.

~~~
troels
Yes, it did seem a bit out of line. He could have sent the entire post as an
email to one of the founders instead.

~~~
philwelch
Forgive me for being a little naive here, but I have a genuine question. Is
there any actual harm that could come of this, aside from people looking down
on the author for writing and publicly posting it?

~~~
troels
I don't know. It's probably in a gray area zone.

What conclusions would you draw yourself, if you were about to hire this
person and that post came to your attention?

If I was hiring and I googled the persons name and found this, I would perhaps
be slightly concerned. Not alarmed enough that it on its own would sway me
either way, but I would probably pay a bit more attention to the persons
attitude in general. To me it suggest either a lack of courtesy or an
overblown ego.

~~~
philwelch
Is there an unspoken understanding that job interviews are some secret process
that we shouldn't openly discuss? I'm honestly mystified by all of this--I
understand that it's considered a faux pas by some, and would never post
something like this myself, but this just seems like something that's bad
judgment only because everyone thinks it's bad judgment.

~~~
learc83
I think it's kind of like discussing your salary. It's considered wrong
because in general management doesn't want everyone to know just how arbitrary
compensation is.

If you really dig into it, you'll find huge variations in pay for similar
skilled, experienced, and hard working employees just based on who did the
hiring, how much was asked for, and when they were hired.

When I was in college I had a job where I made about 50% more than most people
I was working with--I didn't really work harder (everyone worked hard) nor was
I much more skilled. I just negotiated a bit better, and was hired by the
right manager.

------
dfc
Can anyone with hiring experience comment on the relative merits of publising
a post-interview write up like this? (Not just from airbnb's perspective, also
future employers) My initial reaction is that nothing good can come from this
but I do not have a lot of experience in hiring or recruiting.

------
ryanpers
bad interview process... but they are correct in the sense that chef and
puppet solve problems, but that problem is NOT software and clustered software
deployment. You could hack it on top of it sure, but it'd be more straight
forward to basically start from scratch. Hell bash for ssh loops would work
better.

Perhaps they really were looking for someone who already thinks they way they
do, and that is valid. they probably dont need you in their office for a few
weeks saying "i could do this in chef if..." until they fired you for "poor
fit".

Also dont forget that tech/aspy folks just are... blunt in a way that most
people interpret as "insensitive". Maybe they're just being efficient? You
write a blog, seem pretty eloquent, so I'm guessing you are NT, so perhaps not
a good fit after all.

~~~
cparedes
Author here.

NT? (edit: you mean non-technical? I'm definitely not non-technical - I work
as a systems engineer by trade.)

And, yes, you don't need to use Chef or Puppet to solve the software
deployment problem. You use it to ensure system state, so you _know_ you can
throw software on top of it without any problems. (this can go on into a
debate about golden images vs. configuration management, but I think this is a
different discussion.)

That's probably true as well, though... I've been in shops without Chef or
Puppet, and it's really not a big deal for me, either. The thing that stood
out was the _justification_ for not choosing Chef or Puppet, not really the
fact that they weren't using it.

~~~
ryanpers
NT = neurotypical, standard lingo in "aspy" forums. You're probably pretty
normal. Expect eye contact and all.

On a technical note, it's not clear to me how you KNOW that chef or puppet has
solved your system state problem. Now if its limited to doing super basic
stuff, ok great, but then you really arent adding enough value to justify the
huge cost of these systems.

I'd also like to hear what you have to say about "throw software on top of
it"... for example, I want all 40 webservers to have the same build revision
of our codebase. But I need positive feedback - and a gradual rollout
possibly. Eventually run puppet clients really dont tell us this. Oh wait, I
mean I want to rollback to the previous deployed version, because it's broken
horribly.

These practical software deployment issues I find just are not addressed by
the system configuration management tools like chef, puppet.

And that is why they are so passionate about it, because a million people have
told them 'just use chef' and yet chef doesnt solve their problem at all, nor
does it solve any fraction of their problem either.

~~~
cparedes
I can't know for sure, but it's at least corroborated with quite a bit of
research:
[https://www.google.com/search?q=usenix+configuration+managem...](https://www.google.com/search?q=usenix+configuration+management&sugexp=chrome,mod=14&sourceid=chrome&ie=UTF-8)

I'm a fan of both of those systems, because it's easier to express system
state with them (and clearer, too), versus using shell scripts or what have
you. I'm definitely not advocating using it as a way to deploy software, but
it's definitely possible to do so (I'd likely build system packages, throw
them in an internal repo, and use yum/apt to ensure software packages are up
to date, or at a certain version.)

It's not the only solution, though. I've worked at shops where there have been
_exceptional_ software deployment pipelines that are absolutely not tied to a
configuration management system. I'm also not a fan of using Chef or Puppet
for doing a git/svn repository checkout of software.

Again, I want to reiterate my point: it's still not about the tools, but about
identifying the problem you need to solve and discussing possible solutions to
the problem. There's no right answer.

~~~
ryanpers
This is a pretty typical answer I've gotten from various people. First off, no
one should ever (a) be using SVN EVER and (b) be checking out in production as
a way to get code there.

So you said "use yum/apt to ensure software packages are up to date, or at a
certain version."

So how do you make a 100 node cluster get to the desired installed state? Do
it by hand? Ssh loop? cssh or similar tools that attempt to do parallel ssh?
This is the part where things really break down and then I realize there
actually is no principled way to do it, and in fact it's done by hand, and the
sysadmins are just creating more work for themselves ensuring job security.

~~~
cparedes
Kick off Puppet/Chef runs? It's not too hard to automate these tasks, either
(or even put a fancy interface around it, though of course this takes a bit
more time.)

It's also a bit insulting when you say that we 'in fact [do it] by hand',
because sysadmins, as a whole, strive to _not_ do that. Fine, I can see myself
logging into a box and typing 'chef-client' or 'puppet' by hand - but this is
_too_ much work to be done by hand. That's why we automate. I don't know why
you have this idea that we simply want to make things harder just for job
security.

~~~
ryanpers
I have run into sysadmins who prefer to be manual, and avoid automation.

I have a destination in mind, and I'm always looking for new travelers. But I
havent seen many lately.

------
johnx123-up
_At the tail end of the interview loop, I was asked how I would improve
Airbnb’s site._

May refer these guys <http://labs.agriya.com/burrow>

