
Lessons from Interviewing 400 Engineers Over Three Startups - mooreds
http://firstround.com/review/my-lessons-from-interviewing-400-engineers-over-three-startups/
======
sho
> One year, I hired 50 engineers. I couldn’t have done that in a high quality
> way without a system that let me interview about 500 candidates

How is this even physically possible? I don't mind interviewing usually but 2
or more a day, every single day, sounds like pure hell. Not to mention that
after not touching any code for a whole year I'd be questioning your ability
to be doing technical hires in the first place.

I'm very skeptical also that this ultra-standardised approach is getting
anything like what I would call "top 10%" programmers. Top 10% of people who
happen to do well in this particular interview setup, morelike. Teams I like
to work for have a healthy mix of abilities, passions and experience. This
sounds like a recipe for a room full of dull clones, none of whom have any
knowledge outside a strictly specified little box. What next, psychometric
screening?

I do agree with one point though - 3 people in the room is probably the right
number.

~~~
caio1982
> What next, psychometric screening?

I did one about 10 years ago, and let me share a funny story. The head of HR
was sitting next to me and right after the psychometric test she gave me a
blank sheet of paper and told me to draw a human being. I promptly asked
(without even considering my question, I was ready to ask the person's gender
right next, looking for requirements): with clothes on or naked? She laughed
so hard that I thought I had failed and laughed at it too. I got the job,
though.

~~~
paulie_a
That would have been an absolute perfect opportunity to to draw a middle
finger and walk out. That sounds like an absolutely idotic application
process.

I might have even laughed at the HR person and asked the question "are you
fucking serious?"

~~~
bobdole12345
Sounds like you've got a fair bit of experience limiting your own career.

~~~
paulie_a
Not really, a company that's employing those tactics is probably one not worth
working for. An interview is a two way street, they are interviewing me and I
am interviewing them. This would be a huge red flag.

~~~
bobdole12345
The best companies I've worked for have been the ones who put a lot of effort
into making sure they understand who they're hiring. That usually means a lot
of weird questions about topics that aren't related to the technical aspects
of the job.

My last round of interviews I got the tech side without any issues, but I was
subjected to three rounds of the soft skills side because they were concerned
my quiet affect might not fit into the team. I did what I was asked and was
rewarded with working on a fantastic team that someone had put significant
effort into ensuring the personalities meshed well.

Burning a bridge because you thought someone cared too much about the wrong
thing is a great way to never get another callback from everyone involved in
the process. Recruiters and HR people move around a lot, and have long
memories.

I'd rather someone cared too much, than not enough.

~~~
paulie_a
I'd happily burn a bridge with someone in HR that expected me to jump through
hoops and draw pictures. Those people need to hear the truth, they are bad at
their jobs.

------
FLUX-YOU
This article feels like it dances around the point of technical questions and
the assessment that comes with it, which is what most of us worry about when
interviewing.

Soft skills are often treated as if they're there or they aren't. There's not
as clear a distinction between 'junior' and 'senior' soft skills. I don't
think I've ever gotten a salary bump in the offer for being a cool guy to work
with (according to the interviewers). Usually it means I get the job or I
don't. This isn't really that interesting to me.

The group stuff is spot on though.

\- One on one is often dry. You have to have an affinity right off the bat.
This is the equivalent to showing up at a bar and having a fun conversation
with a stranger.

\- More than two or three interviewers, and it can quickly turn into a company
circle-jerk (I've had 15 interviewers once from 4 separate teams in one
interview -- that fucking sucked).

~~~
rhizome
_One on one is often dry. You have to have an affinity right off the bat._

If I follow you on soft skills, are you implying that a company that
subscribes to this logic is selecting for attributes that their current
employees lack, even though they're participating in interviews that will be
used to judge same?

~~~
FLUX-YOU
I mean that it's more boring, not that it's intentionally set up to be that
way to find certain people.

~~~
rhizome
Well yeah, because the interviewer knows the other people! The candidate,
however.

My point was that if they're actually hiring for soft skills, why does it not
hold sway in one-on-ones? It would seem to symbolize a weak bullpen, so to
speak.

------
rightbyte
1 in 10 interviews is a hire? How can his methods be described as sucsessfull.

Speaking of my own experience I'd say it takes 6 months to find out if a
programmer is good or really good (bad is easy to spot) and being able to tell
that in 4h is just self-deception.

I wonder how hireing managers compares to coin tosses after some 15min
telephone offsite initial screening and just filtering of resumés.

~~~
tchaffee
1 in 10 fitting the company culture and being a good technical fit sounds
pretty good to me. What number would sound good to you and why?

As someone who has done a lot of interviews, I think his methods sound great.
Through trial and error, I've ended up with an approach that is pretty similar
to what's described in the article, although there are some new ideas from the
article I'd like to try out.

> I'd say it takes 6 months to find out if a programmer is good or really good

I'm much more concerned about cultural fit. After that I'm happy with both
good and really good programmers. A good programmer can be trained up to be
really good. Most companies need a mix a junior, intermediate, and senior
programmers.

------
chrisbennet
Aside from the numbers, this has got to be one of the more enlightened
approaches I've heard of for doing interviews.

Compare it to some places that think it is _good_ thing to make candidates
uncomfortable so they can "see how they work under pressure".

------
pjc50
> interviewing is a team priority: engineers on his team who are seasoned
> interviewers can conduct 12-16 interviews per month.

That is a _lot_ of interviews. What does that imply about the rejection rate -
90%, 95%?

> Every engineer must be equally skillful with code and colleagues.

I agree with this but feel it might be controversial here.

~~~
rhizome
_engineers on his team who are seasoned interviewers can conduct 12-16
interviews per month_

By my back of the conventional-wisdom calculations, each engineer is losing
about a day's worth of productivity every week.

~~~
BeetleB
That's 20% for recruiting and hiring.

What is the cost of a bad hire, though?

~~~
ryandrake
I’d love to see a company actually quantify this, instead of just assuming it
is large and letting the question paralyze them into being excessively
conservative and picky.

I’ve been in a situation where we needed somebody _right now_ to backfill
someone who left, and the company agonized over who to hire, taking over a
month and blowing through dozens of candidates, ensuring that the product
schedule slipped and losing us an untold amount of revenue.

~~~
mycelium
And to pile on the anecdotes, I've been in the exact opposite situation where
we needed somebody _right now_ to handle a new requirement, and we hired
someone "pretty good" after already taking longer than the rest of management
was happy with to hire.

They ended up being pretty weak, having trouble shipping, eating large amounts
of onboarding and management attention, as well as other engineers patching up
their work to make it serviceable. They were just good enough to make it seem
worth the slog too, so they sucked a bunch of extra labor out of everyone else
for the better part of a year before they left.

In retrospect, I made the right decision with what I knew in the moment — but
with 20/20 hindsight I should have waited.

~~~
rhizome
So it sounds like they were "fine," and the cost was not really that high.

~~~
mycelium
No, it sucked.

------
mooreds
I saw Marco speak at Railsconf 2017 and he was wonderful at challenging some
of the social norms that I had around development as a career.

Highly recommended:
[https://www.youtube.com/watch?v=g6fBRwi6cqc](https://www.youtube.com/watch?v=g6fBRwi6cqc)

~~~
kirubakaran
What social norms in particular? Could you please elaborate?

~~~
mooreds
His feelings about being black in tech, how he felt he had to act when he was
in that unique position and the feelings he had about being an example.

------
joemag
In interviewing, like in many other subjective measurements an important trade
off is precision (percentage of hires that worked out) vs recall (good hires
that got rejected).

Experienced interviewers, or an innovative process can optimize both values,
but I do not believe one can reach perfection. In fact, I don’t think most
companies are even anywhere close to optimums in either of those values.

Yet what’s rarely considered during interviewing is which of the two values
the organization should optimize for. An early stage startup, hiring a senior
engineer may choose to optimize for precision. A mature organization that
needs to hire quickly to start a new initiative may choose to minimize recall.

Another way to put it - hiring is risky. The amount of risk you are willing to
take should depend on circumstances and your ability to absorb failures.

------
tchaffee
Having learned this myself over a few decades through both trial and error and
also researching best practice, I have to say this guy gets a ton of stuff
right. I also read a few new things I now want to try.

Just to reiterate one of the more important points from the article in my own
words: resumes (CVs) have almost zero correlation to the quality of
candidates. Pre-screening calls can really only screen out people who are
completely unqualified due to not actually having the skills listed on their
resume. So I will pre-screen with some very basic tech questions. Or more
likely, I'll just use a recruiter to pre-screen for me.

I'm going to finish up with a copy paste from a past comment I made to a
different article with some tips that have helped me over the years:

1\. Unless you've been trained to interview people, you are probably really
bad at it. Think about how bad someone is at coding who never did it before
and has no training. Interviewing people is no different: it is a skill that
takes time and effort to learn. Get trained up and practice. As much training
as you have time for. Educate yourself in your free time on a regular basis
with articles about best practice. Adding more words won't make this sink in
more, but I will end by saying I can't stress this point enough. You are bad
at interviewing. You need to fix that before anything else if you want to hire
good devs.

2\. How are you measuring success as an interviewer? This is a really
difficult thing to measure because how can you find out that your interview
process is rejecting candidates who would make great employees? I'm not going
to try to answer this here, but it's an excellent question to think about. One
thing that might help is the next point.

3\. Whatever your first impression is of the candidate, spend a good part of
the interview trying to prove yourself wrong. Candidate seems like a perfect
fit for the team and a great coder? Instead of relaxing because you're sure
you already found the right person, see if you can figure out why they might
be a bad fit or if their coding skills aren't as good as first impressions.
Same for someone who seems weak at coding. Maybe they are just nervous. Can
you make them relaxed enough so their awesome skills can come through? Maybe
in both cases your first impressions will be proven correct. But if not you
might have just found a diamond in the rough that other companies are skipping
over.

~~~
richardking
Regarding your first point - any recommendations on articles to read on best
practice?

~~~
tchaffee
I just went through my bookmarks and quite a few of the links are dead... I've
been doing this for decades so that's not a surprise. And not really a big
deal. Most of the progress I've seen has been made in the last five or so
years. Technical interviews might be horrible now, but they used to be way
worse!

A LOT of the articles I've read over the years I have not bookmarked because
my idea of a good technical interview is the opposite of what those articles
recommended.

I normally read everything interview related I come across in HN or Reddit
programming subreddits but I'll also do a search every six months or so and
see if anything interesting comes up.

Although I have cherry-picked advice from reading many articles over the
years, more importantly I've tried really hard to come up with an interview
process I myself would enjoy as a candidate. That factor is very important to
me because I think the interviewing process sucks at many companies. It's
painful for everyone involved _and_ ineffective. I would tolerate painful and
effective. But I think it can be somewhat painless and effective.

I remember getting at least a few valuable ideas from these:

[http://firstround.com/review/The-anatomy-of-the-perfect-
tech...](http://firstround.com/review/The-anatomy-of-the-perfect-technical-
interview-from-a-former-Amazon-VP/)

[https://www.wired.com/2015/04/hire-like-
google/](https://www.wired.com/2015/04/hire-like-google/)

[https://medium.freecodecamp.org/we-analyzed-thousands-of-
cod...](https://medium.freecodecamp.org/we-analyzed-thousands-of-coding-
interviews-heres-what-we-learned-99384b1fda50)

Also ask someone in HR what soft skills you need to improve related to
interviewing. And then take classes based on that.

Most importantly, and as the article suggested, interview candidates with
another person who has more experience interviewing. With a variety of people.
You'll learn the most by doing interviews alongside people who are better at
it than you. Practice really is what makes perfect. Especially if someone can
coach you.

One final thought, recruiters aren't going to give away the store and teach
you how to be a great interviewer, but you can pick their brain with
questions. If your company will pay for a recruiter then always go back to
them with doubts about your process after every interview. They want you to
hire their candidates so they will help out a little with advice. I've heard
some great advice from recruiters themselves.

~~~
richardking
I haven’t had a chance to look through all the links yet but thanks for
following up! Will definitely review.

------
gumboshoes
Lesson one: interview the people who aren't engineers.

------
mchannon
I'd be embarrassed to admit to these metrics. This guy has wasted not only his
own time by bringing 5x the candidates needed onsite, but his team's, his
employers', and the candidates themselves'. A quality hiring process would
know with at least a 50% confidence level that the person is worth hiring
before bringing them in for the onsite. A phone call or one-hour Skype, or
even 3 supplemental ones, is a lot cheaper than the onsite.

It says a lot about the industry that not only is it common practice to have a
"concave" funnel where you don't winnow people out early enough and end up
making your engineers spend half their workday conducting pointless interviews
instead of doing actual work, but also that management writ large is too
clueless to be embarrassed.

If anything your engineers did, anything, had a 20% success rate, would you
consider them "only the best"? Even if their appraisal of the top 10% of
candidates was accurate and reproducible (it's not even close), I'd still
expect them to achieve their results with fewer than 3, and preferably fewer
than 2 onsites per hire.

Sure, the guy can point to his numbers as some sort of merit badge, but if I
told you I washed my hands 400 times a day, you wouldn't consider me some kind
of expert on handwashing, would you? You'd rightly think I'm some kind of
crazy person.

~~~
dahart
> A quality hiring process would know with at least a 50% confidence level
> that the person is worth hiring before bringing them in for the onsite.

After having done a decent amount of hiring interviews and managing in my
life, I don't think it's possible to have 50% confidence before you meet
someone. The way you gain that much confidence is by 1- talking and listening
to them, and 2- comparing them to other candidates.

> I'd still expect them to achieve their results with fewer than 3, and
> preferably fewer than 2 onsites per hire.

This seems hyper-focused on the minor efficiencies of interviewing, at the
expense of employment efficiency; it's feels like an engineering approach that
might not have all the requirements at hand.

The employee is (probably) going to be paid a six figure salary, and will have
a large impact on the team. Great candidates can improve the efficiency of the
people around them for years and years. Bad candidates can slow entire teams
down. I've watched some excellent and prolific and opinionated engineers do
major productivity damage -- millions of dollars worth, and I've watched some
young CS graduates outlearn and outperform some veterans. Resumes & github
accounts simply don't have enough information to know whether a candidate can
help your company.

In my experience, it has been worth the time to let the hiring process go slow
enough to be able to compare people's personalities, and not just their on-
paper engineering credentials.

I think far too many engineers here on HN and out in the workforce
dramatically underestimate how important attitude and people skills are in the
hiring process, and dramatically over-estimate how important specific math &
programming skills are.

~~~
tchaffee
As another person with some experience with hiring, I agree with you fully.

To put one of your points in my own words, your engineers are going to be
spending the majority of their waking hours with this new person five days a
week. Take the time to make sure it's a good fit.

~~~
mchannon
As yet another person with some experience with hiring, let's not get theory
and practice confused.

Taking the time to make sure it's a good fit is a worthy goal, but the
industry is operating under some kind of delusion that there's a Ballmer Peak
([http://xkcd.com/323/](http://xkcd.com/323/)) to interviewing people
somewhere between the 4th and 5th hour of the onsite interview.

Returns start diminishing very quickly after the first five minutes. This is
why multiple people earlier in the process are necessary.

~~~
tchaffee
> Returns start diminishing very quickly after the first five minutes.

That's the opposite of a scientific approach. One of the best things I ever
learned from an experienced interviewer is the spend the rest of the interview
trying to disprove your first impression. Anything else is an exercise in
confirmation bias. I'm not saying that I've never ended an interview early
when it's obvious there is a bad fit. My time is important and so is other
people's time.

~~~
mchannon
A scientific approach is to pose a theory (this candidate would make a good
hire) and perform a measurement (whoa, this guy's good) and then perhaps
perform the measurement again, without bias, to see if the measurement is
consistent and reproducible.

What you're suggesting is to spend the next several hours performing
additional measurements in the hopes of finding one contrary to your first
measurement. _That_ is precisely the opposite of science. It's also how
casinos stay in business.

~~~
tchaffee
> and then perhaps perform the measurement again

Not perhaps. Definitely measure again.

And that's exactly what I said. Don't trust your initial impression. Measure
again. Get as many data points as you can.

> What you're suggesting is to spend the next several hours

I didn't suggest that and I'm not sure where you got that idea from. My
interviews are usually less than an hour and then I hand the candidate over to
the next interviewer. If they are a strong no from me, I'll even talk to the
next interviewer to explain why and see if they even want to continue with the
candidate. They usually don't want to continue with a "strong no" candidate,
but sometimes the next interviewer might not share my concerns and they do
want to continue with the candidate. Could be the person is a good fit for the
other interviewer's team.

------
rubbingalcohol
Tried to read but the site is hot garbage on mobile.

~~~
chatmasta
Yep, absolutely horrendous and totally unreadable on iOS safari. First a modal
pops up with the “no thanks” beyond the fold (and weird scrolling behavior).
Once I finally figure out how to close it, I’m at the bottom of the page. If I
scroll up, I see the first few sentences and then an email form. No thanks.

~~~
ghettoimp
Even on Desktop, "No thanks", then "No thanks" again before even getting to
article. Hit back without reading.

------
ainar-g
>So if you come on site with us, you're going to have four sessions. That’s
our ‘right’ number, because we don’t want to ask too much time from candidates
and be grueling, but we also need to have enough data points. The four
sessions include three evaluations sessions with the team, each an hour long.
Then there’s a session with the hiring manager, usually me, so I can answer
any questions about how the team works

Did I read this correctly? Four hours? Is this over one day or several?

Most people I know won't even touch a job listing with more than two
interviews.

~~~
ctvo
When you say most people, who are you referencing? Day long loops are the norm
for my social circle. As in, get in at 9am, leave at 1-2pm. That includes
lunch for an informal round.

~~~
ainar-g
I guess I needed to be clearer there.

I meant experienced programmers in Moscow, Russia. For the most part it's
either one Skype call (<= 30m) and one personal interview (1h - 1h30) or two
personal interviews. There are exceptions though. I've heard Yandex requiring
at least four interviews, but my colleagues all tell me that they wouldn't
agree to proceed on such terms.

Apparently, looking for a job is a much more tedious process in the US.

~~~
pkaye
I once interviewed at Apple many years back. Personal interviews with 19
people over 2 days! Well a few of them were in pairs (I presumed they were
training their engineers to interview also.) I didn't make it through however
not for technical reasons but cultural fit (maybe I didn't act fanatical
enough about Apple products?)

