
The Technical Interview Is Dead  - victorhn
http://techcrunch.com/2013/06/22/the-technical-interview-is-dead/
======
tptacek
The technical interview is overwhelmingly the primary screening method of
choice at the world's most competent software development companies. The team
that builds the match engine for the electronic exchange, the team that builds
the boot ROM for your smart phone, the team that builds the Google crawler and
indexing, the teams building kernel SAN storage code --- these teams are hired
by technical interviews.

The "audition project" is a trend story with, from what I can see, very little
empirical evidence to back it up. When the most notable example of a company
routinely employing audition is Treehouse --- all due respect to Treehouse,
but still --- that's a red flag.

The bigger red flag is that the paid audition project method has obvious
flaws. The top 20% of software developers are almost uniformly employed.
Prospective new employers for these people court them actively; in fact, the
problem of companies luring top developers out of other companies is so
challenging that large SFBA tech companies have entered into illegal compacts
to stifle the practice. Companies are so hard-up for talent that they'll "buy"
startups just to get access to their teams.

In this environment, why would a top developer, who has their choice among
tens of different high-paying interesting jobs, _moonlight_ for a prospective
new employer just so they can make sure the relationship's going to work?

Most technical interviews are terribly flawed. They aren't standardized and
they aren't rigorous so you can't compare candidates on any apples-apples
basis and you can't correlate them to job performance to make them more
predictive. Most of the developers tasked with conducting them suck at
interviewing; many use interviews as a sort of hazing ritual, and most use
them as opportunity to project their own subjective views about how software
should be built onto candidates. Many technical interviews are trivia games
punctuated by awkward attempts at working through code on a whiteboard or a
piece of paper in a high-pressure environment.

The solution to this problem is to improve technical interviews. It's not to
pretend that the whole market for devs has suddenly embraced temp- to- perm
hiring. It hasn't. It's getting _harder_ to find good developers, not easier,
and the notion that companies have the luxury of inflicting "audition
projects" on candidates is counter to reality.

~~~
sethbannon
As mentioned in the article, after Google examined the data, they found that
technical interviews were worthless at determining employee quality. Instead,
they say, "what works well are structured behavioral interviews." This seems
pretty damning for the technical interview. Would you just say that Google was
doing technical interviews wrong?

~~~
michaelt
It's funny - when I hear people in the press discussing technical interviews
[1] I hear questions like this: "How many golf balls could you fit into a
school bus?"

When I attend technical interviews I hear questions like this: "How would you
implement an arbitrary-precision decimal number class, like Java's BigDecimal?
Can you write out the add function on that whiteboard?"

When Google's Laszlo Bock [2] says "On the hiring side, we found that
brainteasers are a complete waste of time. How many golf balls can you fit
into an airplane? How many gas stations in Manhattan? A complete waste of
time. They don’t predict anything." it seems fairly clear to me the idea he's
dismissing is brainteasers, not whiteboard coding.

That's consistent with other reports [3,4] from a few years ago saying brain
teasers were not used.

The author of the TechCrunch article seems to interpret Bock as saying
whiteboard coding is an ineffective practice. I'm not convinced the Bock
interview said that.

[1]
[http://online.wsj.com/article/SB1000142405297020455230457711...](http://online.wsj.com/article/SB10001424052970204552304577112522982505222.html)
[2] [https://www.nytimes.com/2013/06/20/business/in-head-
hunting-...](https://www.nytimes.com/2013/06/20/business/in-head-hunting-big-
data-may-not-be-such-a-big-deal.html?_r=0&pagewanted=all) [3]
[http://www.technologywoman.com/2010/05/17/debunking-the-
goog...](http://www.technologywoman.com/2010/05/17/debunking-the-google-
interview-myth/) [4] [http://www.guardian.co.uk/technology/us-news-
blog/2012/oct/0...](http://www.guardian.co.uk/technology/us-news-
blog/2012/oct/03/google-interview-brain-teaser-myth)

~~~
dyno12345
I'm suspicious of whiteboard coding as well. Why not give the person a
computer and 20 minutes? Why make it less realistic by taking all reference
material, editor and ability to test-run the program?

~~~
raverbashing
This

It's the same stupid behaviour with "scrum post-its" when there are tools
available.

Yes, you can't interview a surgeon by giving him a scalpel and a patient, but
this is not the case with software development interviews

------
JamesVI
Variations on the theme of "have them do a paid project for you" surface here
every so often. The problem is that a large number of job seekers currently
have a job (everyone keeps telling them it is easier to get a job while you
have a job). Even qualified, competent programmers will find it hard to accept
the risk of being dropped at the conclusion of the short project (and if you
no-one ever gets dropped, what is the point of the project besides giving you
a warm and fuzzy?)

Fact is, you have to do everything up to that 'mini project' then just go with
your gut and hire/no hire. If they underperform in the role you need to either
change the role, try and train them up or let them go.

This trial period is just a way of pushing the risk of a bad hiring decision
back onto the candidate and allowing the manager to avoid the discomfort of
actually firing someone (although telling the candidate that the project
didn't work out is functionally the same thing).

~~~
dwc
Yeah, I think it's a nice solution but it only works in very narrow
circumstances.

Once I was given a much smaller, unpaid task as part of the interview process.
It wasn't a puzzle or trick, but something actually useful. All told it took
me a couple of hours one evening. You might be thinking, "How much can you
learn about someone from such a small task?" It depends on the task, of
course. Mine was typical network code, with a couple of non-obvious but
absolutely real-world edge cases. I spent time making sure my comments were
great, and then wrote a man page. I think they learned some important things
about me, and on my end it was an easily paid investment.

~~~
CoolGuySteve
We've been using something like this but centered around concurrency rather
than networking. It works out great, you get to compare across many candidates
how they handle the tricky parts, if they understand the standard vocabulary
of concurrency, what their coding style is like and if they understand basic
C++.

One thing we noticed was that there is small handful of mistakes that many
candidates tend to make. The ones that note the existence of a mistake or work
around them skillfully are typically the best.

------
balloot
I don't know what it is about technical interview-style code puzzles, but I
have always been terrible at them. Every time I go through a job hunt, I end
up with at least one fabulous flameout where I just can't do something that
would normally be very simple for me. It sucks. I have fantastic experience,
good references, a degree from a top university, but you put me in a room with
someone looking over my shoulder as I write code and I just turn into mush
often enough that it makes passing a many round technical interview a
statistical improbability.

The good news is that more and more companies (and especially the small ones
that I like to work for) do the coding sample/test code thing. And every time
I get one of those I absolutely crush it. I've never once been denied an offer
when asked to write an offline code sample - and the offers have been from a
number of the top startups in the game.

What does any of this mean? Well, in the last two companies I worked with I
ended up rising to be the top technical person on my team. I've never had any
kind of negative review of any sort. And in the last round this happened after
I was denied an offer from two different companies because I couldn't code.

So I guess I'm that guy. I've been rejected from Google 3 different times
because they keep begging me to interview based off my experience/internal
recommendations, and then I interview and their interview/hazing process weeds
me out. It would be nice if they woke up and realized that process isn't an
effective filter, but I'm not holding my breath. In the meantime there's
plenty of companies who will happily hire me after they see me do work that's
actually relevant to the job. And when I interview people for my team I steer
clear of trying to stump them under intense pressure, and instead ask for a
mini project. I guess it doesn't feed the ego quite like breaking someone with
a super awesome whiteboard question, but screening candidates more effectively
makes up for that.

~~~
brianmcc
Agreed, surprised this whole aspect isn't discussed much, much more.

There's obviously a massive difference between "can code proficiently" and
"can code proficiently in an interview situation" and I would guess that a
significant amount of even fizz-buzz fails are due to this, let alone complex
algorithmic stuff.

When you're an interviewer doing many sessions it soon becomes very routine,
to the point of tedium. Being an interviewee is on the other hand hugely
stressful for a lot of people, and that just isn't conducive to good "on the
spot" analytical thinking. And interviewers in their comfort zones completely
forget this and don't factor it into performance (done so myself).

------
benjamincburns
This annoys me. Articles like this co-opt a term (technical interviews) and
redefine it against a subset of things that qualify as being that term
(brainteaser/quiz heavy technical interviews).

There is simply no way to reliably hire skilled people without at least
attempting to assess their skills. For technical people this means technical
interviews. And an interview process that includes a trial project... is
_still_ a technical interview.

This is distracting from the real issue. Most companies are really bad at
scaling their interview policies/processes. Quiz/Brainteaser interviews are
born from an effort to standardize evaluations across multiple interviewers
and multiple candidates. So rather than focusing on developing an interview
process that's focused on finding people that are a good fit, you're focusing
on developing an interview process that's first easy for anyone on your side
of the table to execute. Structure and process are good when they benefit the
prior case, and stifling when they benefit the latter. In the end you wind up
with a very expensive and completely repeatable noise generator that's of no
benefit to anyone.

------
ig1
Giving someone a real project to do which they get paid for is a really bad
idea. If the person is working for another company at the time then they
likely have something their contract that specifically stops them doing work
for someone else while still under employment. It's ground for a lawsuit
against the employee.

It also means that as the interviewing company that you don't legally own the
IP rights for the code you just put into production, which opens you up to a
lawsuit as well.

And that's just for when you're hiring someone from another random company, if
you're hiring someone from a competitor you've got a whole different layer of
complexity in terms of NDA, non-compete, etc. violations.

And then you get into employment law and accounting complications. Who's
declaring tax on the payment ? - if the candidate isn't setup for contracting
work they'll need to do a bunch of work to sort out the taxes, if the company
takes care of taxes than that might be taken as evidence of employment.

It's one of those ideas that might seem nice superficially but which will get
you into a world of pain in the long term.

~~~
carterschonwald
You can't bar moonlighting in a work agreement in the US.

~~~
munin
AIUI, both legally and practically, you can!

Legally: a lot of graduate student employment contracts bar any other paid
employment with clauses like your graduate student tuition deferment being
lifted if it is discovered that you have a side job.

Practically: So, you moonlight for a competitor for a week and if your boss
finds out that you added a shipping feature into someone elses product, and
you didn't get the job..?

~~~
carterschonwald
those aren't legal statutes, those are contracts.

Just because a contract says something doesn't mean its enforceable. Many grad
students moonlight with consulting work in computer science and other
technical disciplines.

~~~
munin
the law being on your side is a cold comfort when you're fired!

------
k8si
I haven't started applying for jobs yet (I'm still in undergrad) but I know
for a fact that I would choke on those "brain teaser" questions, turn bright
red and everything. Admittedly, there's just too much pressure (I hate looking
stupid, especially in front of strangers, especially in front of male
strangers--don't we all though?).

However, I'm pretty sure that I CAN do the following (even under pressure):
Work through a problem rationally, work constructively with others, pay
attention to detail without losing sight of the big picture, write code. And I
would much rather focus on side projects than studying variations on Towers of
Hanoi.

So this whole interviewing trend makes me want to cry out of relief.

~~~
munin
> (I hate looking stupid, especially in front of strangers, especially in
> front of male strangers--don't we all though?).

You should get over that. It is probably very important for your career (and
life) that you not care about looking stupid in front of other people. It's
going to happen (a LOT) in your professional career, dealing with it
gracefully is one of _the_ skills to have.

~~~
lelandbatey
For many people, that's rather like asking them to "get over" their fear of
heights; it's not very reasonable. It's the kind of thing you struggle through
over the course of your lifetime, probably never totally solving it.

~~~
munin
owch yeah. re-reading my words, they could be pretty harsh. I didn't mean for
the glibness of my reply to imply that it would be easy - it's very hard! but
from personal experience, it's very worth it.

------
sp_
Paid moonlighting projects are illegal for all us work visa holders but
sometimes we wanna change jobs too. :(

------
pandaman
I think this person misses the point of technical interview. Technical
interview is to assert the candidate actually has the skills and experience
claimed in the resume.

When somebody is saying he or she has 20 years of C++ experience and 150
projects shipped yet he or she does not know what is virtual function - this
means he or she lied on the resume. If you had not conducted a technical
interview - you'd learned about this much later, at much greater cost. Chances
are that somebody who lies on the resume is also good at BSing about "past
projects" and "greatest achievements".

FizzBuzz or a general screen test won't help because people have different
skills. If you are hiring somebody who writes Perl it's stupid to ask about
C++ and vice versa.

Writing a test project - probably the best way to hire recent graduates.
Probably the worst way to hire somebody who already has a job and you are
desperate to poach, not really good at picking people who have several offers
to choose from either.

~~~
alok-g
The purpose of an interview is to see if the person is a good fit for the
position at hand; looking at the "future". The past experience is a natural
measure of future performance, provided the future projects at the new
position are similar to the past projects. My experience [1] is that the past
projects can often be significantly different from the future projects, and
yet the candidate may be a great fit.

What does carry along with the person is still something about the way that
person thinks, handles problems, basic aptitude, knowledge and understanding,
etc. Fizzbuzz is a good measure of basic programming aptitude [2]. I have
constantly used simple equivalents of Fizzbuzz for electrical engineering and
have hired some great performers.

[1] I have conducted over 100 interviews. [2] C++ vs. Perl should not matter
much since the question is about the basic algorithm.

~~~
pandaman
>The purpose of an interview is to see if the person is a good fit for the
position at hand

This depends on the position. If the position requires an MS degree and 5
years of experience you'd need to spend couple of weeks testing for the whole
corpus of knowledge, that these requirements demand. It's quite expensive and
few candidates are willing to spend this much time. The only practical way is
to go with the credentials the person already has.

If you are looking for somebody to write websites using frameworks then,
indeed, you can test all required skills in a day of technical interviews.

The handling of problems would be nice to know but, unfortunately, is hard to
test. Somebody who just flew into the interview and is confronted by a bunch
of strangers is not in the best shape to handle problems. If he can even in
these conditions - good, now you know that this person can work under stress.
Successfully solving simple puzzles though does not show anything more than
this - you are likely looking for somebody to solve much harder problems that
require weeks and months of research. Neither does a negative result mean much
- this is most likely a result of stress.

So the brain teasers and "how many sorting algos you can write on a white
board in 30 mins" type of questions, in my opinion, are useless. Asking
concrete questions in the domain of the candidates knowledge on the other
hand, lets you quickly verify much wider set of skills.

~~~
alok-g
I am largely in agreement on what you wrote here.

------
EliRivers
Whilst I'd love to have them do a paid project for me that will actually ship,
I work in a part of the programming world where a project easily runs a decade
from first draft of requirements to entering maintenance. This sort of
suggestion always seems to come from the world of web apps or SaaS or whatever
else is hip at the moment.

I also can't imagine what HR would say if I told them I wanted to hire someone
for a week just to see how they get on. "Well, then your HR system is broken
and you should totally hack it to make it better." I'll get right on that,
just as soon as I become Vice-President of HR. They'd tell me to do my job and
hire people who can do what they're hired for.

Where this is going is the ripost that this kind of suggestion always comes
from the small part of the industry where it might be practical; the are are a
great many programming jobs in companies where it really, really isn't.

~~~
jerrya
I guess I am such a terrible developer, that I too can't imagine a project
that is useful that can _ship_ after just a week's work (of probably part time
while working at my real job).

I did an online coding interview a few months back, and after working with
some dude for 20 minutes on some program he asked me if it was ready to ship.
And I was "uh, no, of course not" since we had done no testing of any sort
apart from mentally running through the code. "Well, make it ready to ship!" I
don't think he understood. The two of us certainly didn't agree on what
software engineering means.

------
jacques_chester
I see that hyperbolic headlines are going strong, however.

~~~
saosebastiao
This is from the same organization that proclaims your startup dead if it
hasn't raised any new venture capital in the last 12 months. Hyperbolic is all
they do.

------
incision
I interviewed with one of the big Internet companies not too long ago and
found the process pretty refreshing and quite unlike what I'd so often read
about.

It opened up with a sort of multi-faceted FizzBuzz followed by long
"behavioral" discussion like one of the references refers to [0].

The audition process doesn't make any sense to me. It's essentially a short
duration temp-to-hire. Great people tend to be employed or at least not idle,
unemployed people are looking for jobs, not 1 week gigs.

0: [http://www.nytimes.com/2013/06/20/business/in-head-
hunting-b...](http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-
data-may-not-be-such-a-big-deal.html?_r=0)

------
brenfrow
I think whats always been hard for me is dealing with the problem I'm trying
to solve and all the anxiety happening at the same time. I feel totally
comfortable writing code, but when 80% of brain is dedicated to flight-or-
fight response it makes it extremely hard to answer technical questions and
pass the interview.

My favorite interview that I took part in and got hired was when I was given 2
hours to pair program a project with the interviewer. So I really like the
concepts in this post. Anything to get the developer to calm down and feel at
home.

------
eliben
Two common kinds of reactions by people who fail tech interviews at Google,
Facebook, etc:

1\. Think how to improve and practice for the next interview.

2\. Bitch about how technical interviews are unfair, biased, miss the best
people and ... OMG - DEAD!!!

~~~
balloot
3\. Get a job at another company that doesn't hire via silly hazing rituals.
Preferably one that isn't past its heyday like the two you mentioned.

------
mahyarm
What is so bloody hard about just putting a laptop in front of someone, ask
them to do a bunch of basic but real programming tasks, including internet
access and see what happens? The overall jist I get out of people have been
pretty accurate the months and years I have worked with them.

I dropped the ball on one employer because the audition project took too long.

~~~
abraham_s
Isn't there some real logistical problems with that. For example I am used to
an particular keyboard and typing on a laptop keyboard is big issue for me.

~~~
NegativeK
If you simply type slower (or with a higher error rate) on a different
keyboard, I suspect no one would care. If it's a medical condition, I suspect
a reasonable company that wishes to avoid probably valid lawsuits would let
you bring in your own keyboard.

~~~
abraham_s
It not about others caring, rather it will a major distraction for me and we
are back with a uneasy interviewee (which was the problem with whiteboard
coding).

------
krob
I've done interviews before. I think that there are still many older people,
and young people who do not have accounts on sites like github, they have
nothing to show. But may be reasonable candidates to hire because of their
technical talent. I think it's appropriate to ask questions to an interviewee
based on the domain knowledge you'd expect them to have during an interview.
These involve little things that are very typical of day-to-day work. These
are the type of things I'd ask over the phone or email. Once you determine
they are not blowing smoke up your rear, ask them to come in for a more formal
person-to-person interview.

I then ask them a fairly broad open-ended set of questions to see how they
describe the process of solving said problem.

This usually involves talking about tools, their process, their libraries of
choice(something more generic here for other fields) and generally helps to
assess their personable skills.

If I like them, I will ask them if they have any personal projects they want
to share or something they are enthusiastic about in their domain of
knowledge.

I think the biggest thing is that a lot of companies doing technical
interviewing only reserve a small period of time to determine their new
candidate. For people long in the industry, their resume should speak for
itself. For younger people, it's kind of like, would you sleep with a woman
you just met? How does that experience usually turn out? It's a 50/50 or worse
chance you end up with a nightmare. Spend some time to interview your
candidates, or end up with the one-night-stand.

------
nether
This is funny because they've adopted the same interviewing style that large
bureaucratic organizations have been using for at least a decade, even for
technical positions. I went through a behavioral interview at Boeing, and then
another at a large federal agency (engineering job). Technical aptitude is
usually clobbered by the inability to work with people.

------
freework
This article doesn't mention the best interview method of them all: the
presentation interview. Have the interviewee prepare a 15 minute presentation
of a technical topic of the interviewee's choice. At the end of that
presentation the team should have a general idea of the person's thinking
speed/experience/communications skills. Basically all you'd ever need to know
about the person.

In my version of a perfect world, each company would have a "speaking hour"
designated once per week. Each speaking hour would feature two or three
speakers, each with 15 minutes to speak about something interesting they've
worked on. The idea being if you wanted to work for a company, you'd apply to
speak at their next "speaking hour". If you show up and give your talk and the
audience approves your talk, then its understood you're hired.

------
kenster07
I like the idea of using the interview as a screening process, just to make
sure the candidate didn't lie on their resume, and that they are a socially
competent person who can survive in a team environment, or better yet, improve
the team.

But, you should also carefully examine the past code that the candidate has
written (that he can legally disclose) and also design a programming test for
candidates to complete.

------
agsamek
One good observation from the article is that more simple questions is better
than a few harder ones. Not much besides this.

My observation is that recruitment is only one part of the problem. The other
part is that companies do very poor job at assessing skills during employment
and loose/fire randomly. This seems to be much more important problem for top
companies leading to slow degradation of skills over time.

------
lowglow
We talk a little about how each one of us at Techendo does our hiring. It
might offer a bit of insight on what works and what doesn't.
[http://www.youtube.com/watch?v=xHunZeWOrjc](http://www.youtube.com/watch?v=xHunZeWOrjc)

~~~
yason
How about explaining your insights in ten sentences or so in this thread
instead of just linking to some video that only few people would actually go
watch just like that?

------
biot
Reword the title as "Is the technical interview dead?" and you have your
answer: it's always "No".

------
dschiptsov
So, finally, "show us your code/poetry/papers" won?)

