
The interviewer skills ladder for high growth companies - leeny
https://medium.com/@alexallain/what-ive-learned-interviewing-500-people-the-interviewer-skills-ladder-for-high-growth-software-37778d2aae85
======
choppaface
This article has some interesting tips and useful encouragement but the system
it outlines has a glaring hole: there is no objective feedback loop based upon
_outcomes_ back to the interviewer. The author appears to encourage that
interviewers seek feedback, but the message is not to look for data but
_sentiment_ :

“One way you can tell you’re proficient is that recruiters and hiring managers
try to find ways to include you in their interview loops because they know
you’ll both get the candidate excited and be able to assess their work fairly,
even if they take an unusual approach.”

If recruiters and hiring managers want you on a panel, it signals that you’re
unlikely to disrupt their pipelines by rejecting candidates. This is
especially true for high-growth companies (e.g. Twitter where ICs would do 8
onsites a week; I did 5 per week for years). They like you not because you’re
proficient but because you’re aligned.

Is it possible to be aligned and “wrong” ? Of course! Invariably, gluttonous
hiring practices result in the need for “bar raisers” to restore consistency,
entire teams with no real focus, and an explosion of communication channels.
Hypergrowth is cancerous. And if everybody is in it together, nobody will be
hold another accountable.

This article outlines very little about proficiency and a lot about how to
brainwash new grads into believing alignment itself with a recruiting process
is a very special skill. Seek evidence about real outcomes instead. Do your
own critical thinking.

~~~
bsder
> If recruiters and hiring managers want you on a panel, it signals that
> you’re unlikely to disrupt their pipelines by rejecting candidates. This is
> especially true for high-growth companies (e.g. Twitter where ICs would do 8
> onsites a week; I did 5 per week for years). They like you not because
> you’re proficient but because you’re aligned.

Even recruiters will cut you a lot of slack if your feedback is occasionally
something like: "Well, we passed on that candidate. However, his knowledge of
X, Y and Z were quite outstanding. Try and funnel him to a place that needs
that--maybe company H?" This shows that you, as an interviewer, really did
dedicate the time and effort to interview the candidate properly. Good
recruiters consider this to be _gold_.

And other hiring managers have never begrudged my negative judgments.
Especially because I generally try to find positive points in a candidate and
give some details--"Well, I liked X, Y, and Q, however, I am deeply concerned
about lack of Z. If anybody else has counter-evidence that shows he _does_
know Z, I'm open for it." Generally, the result is a couple other managers
also saying "No, that lack of Z is real, and there are some other holes we
discovered."

Occasionally, you get a surprise. Someone will pipe up: "Huh? We talked about
Z for almost 40 minutes and he was phenomenal. You must have had a
communication misfire." It happens--then I will defer on that.

I _have_ had managers probe my consistency by sending me "ringers" that they
were obviously going to hire just to see if I was being overly harsh. One
manager decided to quit doing that after I threatened to hire the candidate
for my own group.

~~~
choppaface
How do you push back when the manager sending the "ringer" is the VP of
Engineering? And they're wrong about the candidate? I've had this happen twice
with VPs at different companies. VP makes a joke panel, candidate gets hired,
then fired within 3 weeks. Then at least 1-2 people want out of the fired
person's team the following month.

There are definitely recruiters and other players who are thoughtful when they
get thoughtful feedback. Yes, if you help a recruiter tune their funnel,
you'll help their numbers, so of course they're going to listen. Those
incentives are clearly aligned. But in a high growth setting, especially
around new grad job fair time, the head count quota is what dominates. It's
easy for that quota to be unrealistic, and for hiring efforts to become
misaligned, or even at odds, with the long-term interests of the engineering
team.

The core problem is that recruiters, hiring managers, executives, etc.,
deprive ICs of the contextual data (e.g. headcount goals / internal rankings)
and outcome data (especially compensation details) that are critical for ICs
drawing their own informed conclusions. We must cease accepting the norm where
ICs have to dig for glimpses of this data, and share it surreptitiously (e.g.
Blind), when the whole picture is readily available to those owning the final
decisions.

~~~
bsder
> How do you push back when the manager sending the "ringer" is the VP of
> Engineering?

Work someplace better?

Seriously, anyone I have known who is senior in the engineering chain _always_
made sure to tell people "Look, I'm recommending this person, but _you_ are
the hiring manager whom he is going to be reporting to. If you and your team
don't agree, then he doesn't get hired. So be it."

And, they meant it. VP recommendations only have about a 50% success rate when
I'm involved. If the VP isn't a dolt, normally it's simple misfit. The tougher
discussions go along the lines of "This person is really good at Z and we
really need that. The problem is that for him to be useful at this company
after this project he needs background in X and Y, and he simply doesn't have
that." If you have a good VP, he recognizes that the solution is "short term
contract consultant".

------
emtel
I don't think "assessing talent" during interviewing is a skill you can learn.
The skill you're learning is assessing how well the candidate performed during
the interviewing process. The interviewing process itself is what is assessing
the talent.

Put another way, I would rather have a novice interviewer use a good process
than have an experienced interviewer use a bad or mediocre process.

In my experience, FAANG-style whiteboard interviews are a bad process. They
don't work for assessing developers, and the skill of the interviewer can't
compensate for that. The only thing I have found that works consistently is
some form of work-sample interview, where the candidate produces working code
in their preferred development environment.

~~~
braythwayt
"I would rather have a novice interviewer use a good process than have an
experienced interviewer use a bad or mediocre process."

This provokes me to ask, is that also true of software development? Would we
rather have a novice programmer using a good process than an experienced
programmer using a bad or mediocre process?

My first thought is that the experienced programmer produces good results
despite the bad process because they write a lot of tests, and so forth, but
that suggest that they are using their own good process inside a bad process.

An experienced person using a bad or mediocre software development process
really ought to be understood to be eschewing source code control and tests.
They are patching production directly. They aren't putting thought into their
names or software organization.

And yet they get some good results. But given this false dichotomy of novice
with good process versus experienced with bad process... Which do we really
prefer, and why?

~~~
cbanek
I would say each developer also has their own process, which may be more or
less or different than the organizational process. I find that good devs have
a good personal process.

I think being an expert sometimes means you know which rules should be
followed, and which rules can be bent for certain circumstances, possibly even
producing a better result. Novice programmers may not know all the
consequences of their actions, so it is better for them to follow a strong
rigorous process.

Good results is also subjective. Does good results mean fast? Or that you can
look at the code while not having to clean the vomit off your keyboard? Bug
count? Getting it right the first time? Good UX?

Knowing what you're going for and what counts is where the process should be
pointing, but sometimes they are just rules for rules sake.

------
pkteison
Can anyone recommend a good book on becoming a better technical interviewer?
I’ve found plenty of books on being a better interview candidate, and a few
for hiring ceos or sales people, but I haven’t been able to find anything with
useful suggestions for the hiring side of the table in a tech interview.

~~~
jsjohnst
While I can’t recommend a specific book off hand, I’d suggest looking into
books in the realm of profiling. Technics I’ve read in that area made the
biggest improvement in my overall ability to interview engineering candidates.

~~~
Ozzie_osman
Me and over a dozen other contributors are working on a Guide [1] to Technical
Hiring and Recruiting that's due out this fall (published by Holloway). Alex
Allain (author of this blog post) is actually a contributing author and wrote
our section on interviewing. If you're curious enough, and willing to spend
time reviewing an early draft and giving feedback, please use the contact us
link on Holloway.com.

[1] [https://www.holloway.com/g/technical-recruiting-
hiring/about](https://www.holloway.com/g/technical-recruiting-hiring/about)

------
tignaj
Engineer / engineering manager here.

Nicely articulated article. There is one observation that I respectfully
disagree with - the one that says interviews are monotonous and boring.

I have done hundreds of interviews in my career and never found them to be
boring. Perhaps one the reasons is I don't do scripted interviews. It is
always an exploration for me. I will start with simple questions and if I get
good answers I will go deeper and deeper into the subject matter or into the
adjacent area. If I don't get good answers I will retract and probe another
direction. This is sort of a depth-first traversal of candidate's skills with
a heuristic to stop traversing a particular branch if I see it is not a
promising direction.

My goal is to discover the area where the candidate is the strongest
(obviously within the areas I am interested in for the particular role I am
hiring for). It is very fun and almost never boring. The experience is always
different because people are never the same. Yes, some candidates are just not
good, but even bad ones are bad in different ways.

I know common wisdom for using scripts is to have repeatable and comparable
evaluations so that you can choose the best candidate. I understand the desire
but I prefer to find the best in each candidate and if it means I have to use
a lot of gut feeling and intuition to make the final decision then so be it, I
believe it still results in better overall results. The idea to choose the
person I want to hire based on how many answers to my standard questionnaire
they got right was never appealing to me.

I can totally see how if you do scripted interviews it can become boring, so
don't. Interviews can be fun and it can be a creative process.

Another way interviews can be fun is if you find someone who has stronger
skills than you do in a particular area. I love it when a candidate teaches me
something new. Find where their strength is and let them shine.

~~~
Ozzie_osman
I agree that scripted interviews can be boring... but it's not just common
wisdom that using structured interviews leads to better assessments, there's
plenty of research* that shows that too. Of course, you should always take
research in this field with a grain of salt, since things are different from
industry to industry and company to company. But I think it's safe to assume
that unstructured interviews are more prone to unconscious bias. Research also
shows that most people think they are better at assessing candidates in an
unstructured environment than they actually are. If you're really experienced,
you might be calibrated enough to be immune to that to a large extent, but
it's still hard.

But yeah, I agree that it can be boring and a poor experience for both
candidate and interviewer if interviews are too scripted. It also has other
failures, e.g. it can fail to capture different talents that candidates might
have. I personally think the best way to handle this is to mix both structured
and unstructured interviews.

There are plenty of other ways to make interviewing rewarding. Think of it not
as a burden, but as a way to help both the company and the candidate make a
possibly life-changing decision as accurately as possible. Empathy can go a
long way towards humanizing the process. Honestly, one of the most rewarding
things I felt as an interviewer (and then a manager) was seeing someone I
interviewed (or hired) later thriving in a role. Being a part of making that
happen is really fulfilling.

* [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172.1733&rep=rep1&type=pdf)

~~~
tignaj
Thanks for the link, I haven't seen this before, going to read it.

~~~
Ozzie_osman
Google has a few links as well: [https://rework.withgoogle.com/guides/hiring-
use-structured-i...](https://rework.withgoogle.com/guides/hiring-use-
structured-interviewing/steps/introduction/)

There are also some showing that many people choose unstructured interviews
over structured ones because they are overconfident in their ability to
assess, and I think one or two that show that unstructured and structured if
done properly and together, are better than either method alone. But again,
these sorts of studies and metanalyses are very general, so up to you on how
seriously to take them. I at the very least found it made me much more
cautious of my own assessments in unstructured settings.

It's also true that interviewing isn't just about assessing, it's about giving
the candidate information and a good experience. So for instance some studies
show that small talk at the beginning of an interview can bias the
interviewer. But I still do it because I want interviewee to be as comfortable
as possible.

------
sixtypoundhound
My reaction to this article - we've utterly failed when it comes to training
effective mid-level software engineering leaders. It feel like we are
substituting rule by committee for what should ultimately be a leadership
decision. If you cannot select engineers, you should not be leading them.

Employee Input is certainly welcome. We don't want to hire jerks or idiots.
But the manager is responsible and accountable for the quality of hires and
employee performance.

The other half of my life is in manufacturing / distribution. We have
engineers too, managing great big piles of machinery. We RUN on engineers and
mechanics. Generally led by engineers. Hell, my last CEO was a former engineer
and my current COO (who runs the plants) is an engineer himself. It's
engineers, all the way up.

They don't need to shop candidates all over the building to figure out if they
can handle the job. They know. And can make an appropriate decision.

It starts with setting the expectation that engineering centered functions are
to be led by engineers. And the price of leading those functions is knowing
your stuff...

No Excuses.

------
the_arun
I found this is a great summary of non-technical side of the interviews.
Thanks for sharing!

------
neilv
The article names "making the candidate want to work with you" as the first of
the two critical skillsets, and goes on to reinforce that this is important.

I've thought for a while that this might be one reason why some large dotcoms
(which reportedly study their interview efficacy) put even very experienced
candidates through one-way leetcode whiteboard batteries. It seems similar to
"negging"[1]. If there's an attraction intent behind it, that would explain a
lot.

If so, then perhaps we have countless other companies mimicing the big
dotcoms, or at least being influenced by the practice, without being in on the
dark pattern side to it.

[1] [https://xkcd.com/1027/](https://xkcd.com/1027/)

------
draw_down
I noticed that the people who interview a lot don’t really get anything in
return; in our quarterly engineering org meetings or whatever there’s a slide
thanking the people who interviewed most. I’ve never gotten a thanks when I
interviewed more than other times, nor any other indication that this is
something my manager values. (And what your manager values is kind of
everything, in my experience.)

So I almost never volunteer when they need someone to cover, don’t try to put
more effort into it than necessary, and have blocks in my calendar so they
don’t try to schedule when I don’t want them. It may be crucial for the
company (and ethically, you owe it to candidates to treat them fairly) but
it’s not treated like something they value.

For what it’s worth, I somehow slipped through the cracks and didn’t have to
interview for quite a long time after I started. No one complained or made a
note of it at all.

~~~
kinkrtyavimoodh
You touch on a very important thing. It feels like no one cares about
interviewing UNTIL they can use it like a cudgel to hit you in the head when
you ask for a promotion. In the normal course of affairs, interviewing gets
you nothing. But if you don't interview, they get a convenient excuse to not
promote you.

~~~
draw_down
I’m sure you’re right, but honestly they will always find a reason if they’re
looking to find one.

