
The anatomy of the perfect technical interview from a former Amazon VP - jnazario
http://firstround.com/article/The-anatomy-of-the-perfect-technical-interview-from-a-former-Amazon-VP
======
evanjacobs
In my 10 years at Amazon, I interviewed lots of engineers and I think Amazon's
interview process is fatally flawed. Of course Amazon is still able to hire
good candidates but I believe it is in spite of this process and not because
of it.

Here were the most frustrating aspects of interviewing for me:

1\. Bar raisers don't ask questions that they themselves don't know the answer
to. I believe this is what Roseman is referring to (as a desirable?!?
attribute) when he says "One of the things that really pisses me off is people
asking questions that they don’t even have a good handle on themselves". What
happens in practice is that you miss hiring people who may not be able to
answer a question _to which every interviewer already knows the answer_ but
whom might be uniquely qualified in some other area.

2\. Interviewers still ask candidates to code answers to theoretical questions
on the whiteboard. There is no room in the Amazon interview process to "try
out" a candidate for a week or two in a real world setting to see how they
perform.

3\. Amazon considers all SDEs to be fungible but internal politics determines
which group gets to interview a new candidate. Have an advanced degree and
strong interest in machine learning? Too bad, US Retail needs someone to work
on front-end development and they'll ask you lots of questions about
javascript.

4\. This isn't really part of the Amazon interview process but Amazon's job
descriptions are (for the most part) written so generically and absent of
personality that I think they miss out on lots of great candidates for whom
startups simply seem more fun & interesting.

~~~
bane
I remember my own interview process with Amazon. I wasn't interviewing for an
SDE position but a TPM role. I made it through the gauntlet and it all fell
down due to Amazon's grossly unrealistic understanding of what the going rates
for experienced PMs are.

I mostly had fun with it, but I was also frustrated.

The phone screens were the hardest in a purely technical sense, lots of
algorithm questions that really didn't have anything to do with the job at
all, I had to reach into my way-back machine to dig up some of the runtimes.
But I made it through three of phone calls like that and had a bit of fun.
After the phone calls and a chat with a senior person in the department, I was
pretty sure I had a feel for where I was going to end up and what kind of
projects I'd be managing. So I boned up on management theory and algorithms
and a few other odds and ends.

Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-
to-face interview was wildly unfocused, I was interviewing in subjects the
interviewer often didn't have any experience in and completely didn't reflect
the nature of the phone interviews or the statements I had been given by the
senior guy I spoke with. In one interview I'm designing Java APIs for movie
tickets, in another I'm specifying contingency plans for overseas warehouse
fulfillment issues. In another I was asked by two _very_ junior people who
frowned in confusion the entire time as I discussed complex employee and co-
worker conflict resolution cases I've experienced over the last 18 years. In
fact I was never asked a question about the first 15 years of my work
experience, most of which would have been more relevant. But I was asked tons
of questions about my side-startup, which was just me and my wife hacking
around on the side and getting passive ad revenue and had no relevancy at all
to the job.

The "lunch chat" was with somebody who couldn't have made small talk if he had
a gun to his head. He just sat in silence and ate his soup the entire time
while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've
ever encountered.

I definitely _didn 't_ nail the face-to-face, but that's because I was playing
trivial pursuit with the interviewers and the answers were arbitrary, instead
of knuckling down on the job, the role, the department, the expectations of
performance and the responsibilities of the position.

But I did something right because I was called in the next day to go over
compensation and was blown away to find out the absolute max pay cap for the
entire company was almost a 20% pay cut from what I'd been making in my
previous job (a job that I was leaving because I was 10-15% under my peers and
had been in a pay freeze for 3 years while the company worked towards
profitability -- and yes it was from a metro area with a similar cost of
living to Seattle).

Despite that, Amazon would have been good to work at, but even my absolute pay
floor was 10% over. I even asked to make up the compensation difference with a
mix of stocks or other methods, but eventually we had to agree that there was
no way I could live the life I had become accustomed to with what Amazon was
willing to pay.

~~~
AsymetricCom
My understanding is when they fly you out, you're pretty much a shoe-in and
now they're letting their devs have a shot at trying to interview, while
getting a feel how you work with others. i.e. are you sane or will you eat on
of our disposable dev's face?

~~~
crucifiction
Its generally the opposite, the phone interviews are the sanity check.

~~~
AsymetricCom
"sanity check" in this context is a bit different than in software.

------
johnhess
I'm always amazed by how confidently people can propose a hiring "rule" or
even guideline.

The article pays lip service to the fact "There's no way to A/B test hiring
decisions" but then goes on to describe a handful of rules to "build an
effective organization, regardless of size or resources." The thing is,
there's no evidence that these are effective and definitely none to suggest
they're generalizable to organizations of every size. There's an appeal to the
author's authority as an Amazon VP (admittedly a hell of a credential), but
that's not evidence of effectiveness.

The fact is, hiring is tough to learn. It's made especially difficult because
your feedback comes at least 6 months after you make the decision. I've had
the good fortune to be in an organization which grew from ~10 employees this
time last year to ~80 now, so I've gotten to see how a few dozen hires panned
out in a short period of time. I learned a lot through that. Even so, as a
professional data geek, I couldn't say with a straight face that I "know"
anything about hiring. I do my best day in and day out, but saying you've got
hiring all figured out is the professional equivalent of knowing "What (wo)men
want." It's a sign of confidence, but not necessarily knowledge.

Don't get me wrong, in the absence of quality data, I'll take the wise words
of someone who has been there and done that. But I'd much rather some
empirical proof or at least some "YMMV" humility from those who purport to
know.

~~~
vonmoltke
This topic has started taking on religious tones in the past year or so. It
seems to be a long stream of anecdotes and opinion pieces presented with
appeals to authority either of the author or someone the author is connected
to.

------
jpalioto
I appreciate the notion of having the whole team prepared for the interview
process. I think this is a best practice. But, when I interview, I am looking
for completely different things than the author.

I really don't care about what you did at your last company or if you raised
this by 50% or that by 30%. Rather, I want three things in a candidate: smart,
talented and hungry. If I get someone who is smart, talented and hungry and I
drop them into a situation where there is unlimited upside, my job then
becomes to guide them to greatness.

I also strongly disagree with the notion of not asking questions you do not
know the answer to. Anyone can read an obscure section of the C++ language
spec or a chapter on some data structure 5 minutes before an interview and
make themselves feel superior to the candidate. What does this prove? Nothing.
I often ask questions I don't know the answer to in order to determine whether
or not the candidate can teach me something. I love having people on my team
that have the ability to teach others.

The author's point is very well taken -- interviewing is an important process
and it should be conducted in a way that results in the best possible outcome.
But, he's looking at the situation far too rigidly for my taste.

~~~
umsm
I disagree with the hungry part (if you are referring to someone that needs
the money). Being "hungry" just means that they will be stuck with a company
but will not love what they do.

~~~
jpalioto
Less about needing money and more about being ambitious. Hungry for success I
should say.

~~~
Terretta
Whether hungry for money or hungry for success, you're then missing hiring
candidates who aren't "hungry" because they have a proven track record at
having done very well financially from a series of successes.

At this point, they don't need money or validation, they just figure they can
help you solve things and are talking to you because they _enjoy_ solving new
things.

Hungry is overrated when what you need is focus on solving the business'
problems. This is one reason VC like to see founders take "shelter money" off
the table early.

~~~
nilkn
Someone who has sufficient self-made wealth to retire, or nearly retire, but
who is still looking to work for you as a developer I would certainly describe
as "hungry" in this context. But I'm mostly interpreting that term to mean
"passionate" about the intellectual and creative aspects of the job.

------
hobb0001
> [he] believes every phase of the process needs to be meticulously designed
> to drill deep into skill sets, ... and leadership potential

Ugh. I hate this. For a technical interview, why is leadership potential a
priority? Speaking for myself, I've done the leadership thing and don't care
for it. People tell me I'm good at it, but I still prefer dealing with
algorithms over humans, and I actively decline offers for "promotion". (I
already have my very own batch of little humans of to deal with, thank you
very much.)

Unfortunately, no matter where you go, there's always the deep-seated cultural
mindset that manager > worker. In terms of the corporate efficiency
calculation of effectiveness/hour, I suppose that's still true. But for me,
personally, that's not the equation that I care to maximize.

~~~
dragonwriter
> For a technical interview, why is leadership potential a priority?

Probably because in many modern ways of organizing knowledge work,
distributing leadership roles is an important part of the way organizations
operate.

> Unfortunately, no matter where you go, there's always the deep-seated
> cultural mindset that manager > worker.

The places where that is least true are also the places where evaluating
leadership potential in technical workers is the most relevant to immediate
job performance.

~~~
rhizome
_Probably because in many modern ways of organizing knowledge work,
distributing leadership roles is an important part of the way organizations
operate_

Does it work, by which does it produce the desired results or benefits? It
seems somewhat likely that this would have a side-effect of pushing employees
to asserting a dominating personality.

------
afandian
"No matter how many times they say it, most still make decisions based on gut
feel, basic credentials, GPAs, ivy league educations, flashy company names -
even SAT scores. Roseman objects."

What's a Roseman object? Is it some psychological classification of the means
by which we just things? Of course, I mis-read it. Shame, that could have been
an interesting topic.

~~~
lsb
If it's confusing to parse, let me expand it: Neil _Roseman_ strenuously
_objects_ to that assertion.

~~~
afandian
I understood it perfectly about half a second after I misunderstood it. I just
thought I'd share my experience, as I found it amusing.

------
freyr
_" The anatomy of the perfect technical interview from a former Amazon VP"_

And more recently, a former Zynga VP.

Surprising that such an expert at evaluating people chose to work for Zynga,
of all places.

~~~
hyperbovine
Money is a hell of a drug.

~~~
umsm
That's how you get the best talent. There is no secret question you can ask to
find great talent.

I've interviewed persons that did very very well during the interview process,
but failed miserably on a day-to-day basis.

~~~
seunosewa
So money is the answer? Have you never observed a well paid developer who
"failed miserably on a day to day basis"?

------
pdovy
There seems to be a lot of criticism of the suggestions here, but I think they
are good ones. It seems to pretty much boil down to:

(1) Spend a lot of time discussing accomplishments at their current / previous
employers. If you drill down on particular projects they should be able to
explain, in great detail, exactly what their contribution was and how they
worked with the rest of the team to get that done.

(2) Pick a technical question that has a lot of depth so that you avoid false
positives because people have just looked up the answer on the internet. I
would also add that in my view the best questions have no tricks and require
no "aha!" moments. It should be something the candidate can be reasoned
through step by step so you can see their thought process. When they come up
with an initial solution, then you can dive deeper by expanding the problem or
adding additional constraints.

~~~
mturmon
I'm also disappointed that so many comments above do not engage with the
information in the article. I, too, found this to be a thoughtful and helpful
piece. Along the parts I liked --

Using "calibrated" questions that you have used before and know how far people
get with.

Focusing on a specific accomplishment or area of strength of the _candidate_
to probe. (As opposed to choosing something from the resume that, say, the
interviewer was most comfortable with, or that the interviewer found
interesting.)

Paying attention to preparing the _interviewers_ , so they know why the
candidate is being considered, they know why they are on the interview list,
and they are experienced enough with interviewing to give a useful opinion.

------
awakeasleep
The one element no one ever accounts for is the interviewers. You're never
going to have a perfect technical interview because you're an imperfect human.
You're incapable of judging whether someone is a perfect cultural fit, and
someone who doesn't fit the culture might help the company more than someone
who does.

That isn't even getting into the the conflict between whats good for the
company and whats good for you.

~~~
Matt_Mickiewicz
When Google pushed out their interview-results data this year, one of the big
findings was that there is NO SUCH THING as a good interviewer.

To quote the NY Times interview with Laszlo Black of Google - "Years ago, we
did a study to determine whether anyone at Google is particularly good at
hiring. We looked at tens of thousands of interviews, and everyone who had
done the interviews and what they scored the candidate, and how that person
ultimately performed in their job. We found zero relationship. It’s a complete
random mess, except for one guy who was highly predictive because he only
interviewed people for a very specialized area, where he happened to be the
world’s leading expert."

That being said, there are a lot of very disorganized interviewers out there,
especially among seed stage companies. I've been amazed about how many people
grumble about the "talent shortage", yet regularly miss interviews with "hot"
candidates because of meetings, forget them altogether, or just lack a sense
of urgency & speed when it comes to running an effective hiring process.

A large part of having a hiring culture is about decisiveness. Loosing a
fantastic Engineer because you spent 4 weeks making up your mind, or you
didn't want to pay an extra $10K/year is just silly when it means that you
won't hit your product roadmap goals.

~~~
RogerL
I think that is perhaps telling. Google doesn't interview you on your
experience, but whether you can can solve network/graph algorithms taken from
Knuth exercises. I would expect a random mess from that. It sounds like this
person actually drilled down into what the candidate actually knew.

I think it is far more useful to ask somebody about what they actually did.
Are they excited and engaged by it, did they come up with novel thoughts about
it, do they understand the CS literature in relation to it, and so on. Program
a hash table on the whiteboard when you have been just fine using the built in
one in your favorite library or language? I'm not sure what that predicts
about my on job performance. If I _had_ to do that I'd reread the literature
and then whip one up, easy peasy. I can state that with confidence because I
read the literature on other topics and do that every day. Have me explain the
new numerical filtering technique I just implemented and I'll give you good
interview. Ask me to implement a smart pointer? Not so much.

~~~
rhizome
_Google doesn 't interview you on your experience, but whether you can can
solve network/graph algorithms taken from Knuth exercises._

I believe this is a backdoor method to requiring a college degree for
positions that do not require one within labor law.

~~~
eropple
This doesn't make sense to me on first read. I'm unaware of a labor law that
requires you to interview people without college degrees. Care to elaborate?

~~~
rhizome
I may have worded it wrong. A large swath of programming jobs do not require a
college degree, and in those cases it's illegal to require one. This is why
job ads will say, "Desired: College degree _or equivalent experience_." They
don't describe it as a requirement for a reason.

~~~
eropple
It's _illegal_ to require one? Citation?

~~~
rhizome
HN's linker requires you to manually add the period at the end:

[http://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co](http://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co).

------
c0deporn
I disagree with "In most cases, the “best and the brightest” already have
jobs, so you’re really just on the lookout for the best available.". Just
because one is employed does not mean one is taken. If you want the best you
have to work at it. Appeal to their desires and they'll come to you. Looking
for the "best available" is no better than settling. In my opinion you'll end
up doing a disservice to both your company and the new hire.

~~~
walshemj
Lets be honest Amazon is an online shop the best and brightest are either
going to be working at tier 1 RnD or on wall street or the city.

~~~
kenster07
Re Wall Street: It's unfortunate that some of our brightest are tempted into
playing multi-million dollar poker, rather than focusing their energy on
producing any significant long-term value.

I believe software engineering wages will continue to rise over time, and will
eventually reflect the true value the industry produces.

~~~
dylangs1030
I have no shame in admitting that I would rather use high-level mathematics
and programming to play "multi-million dollar poker" on Wall Street than join
half of the startups and big tech companies and have a boring job churning out
insignificant apps at a desk.

In fact, the way you phrased it, being a quant sounds really fun :) you don't
want to play multi-million dollar poker?

I mean, sure, meaning and existentialism and all that, but I bet it's at least
somewhat exciting.

~~~
kenster07
That's the fundamental problem. The function of the finance system in a
healthy economy is to ensure efficient allocation of capital. Playing poker
has a neutral to negative effect on the system.

~~~
dylangs1030
Well you're the one who used the word poker, so I guess it depends on your
definition here...but the financial system somewhat revolves around poker.

Valuations of companies are only important to investors because of the
inherent gambling. If you purchase capital at a single valuation that never
rises or falls, what's the point? Even an index fund pursues gradual increases
over time.

...Or am I building a strawman here? Is this not where you were going?

------
coin
Not related to the content, but why do sites like this disable pinchzoom? What
value could it possibly add?

~~~
freyr
Yesterday I needed to look at source code on GitHub using my iPhone.
Horizontal scrolling AND pinch-to-zoom were disabled, making it impossible to
read lines of code wider than my phone screen.

 _Github_.

Why is this so hard?

------
atopuzov
You pass the interview process and then end up doing shit that monkey can be
trained to do.

------
tomjohnson3
one of the most important things - something that is difficult to suss out in
an interview - is how well you (or your team) would work with a person. (i
think this is perhaps the most important quality - definitely more important
than being able to solve brain teasers.)

i've wondered lately if it'd be better, during an interview, to choose a
problem that both you _and_ the candidate don't know...and try to solve it
together...

------
cell303
Moving away from "the best available" towards "not the worst possible" (who
could potentially destroy our company) would be a good idea.

------
laichzeit0
What's notably lacking from this perfect interview is mention of referrals.
I've had the best of luck with referrals as the managers I've worked with have
never felt the need to "burn" me and could give an accurate assessment of the
roles and responsibilities I have in my current job.

------
realspeech
without any data and careful analysis of that data, his opinion on how to hire
is worthless.

it's somewhat disturbing that he likes to use the STAR framework, which
normally leads candidates into artificial responses. is there any reason to
believe that this framework leads to better results?

