
The Terrible Technical Interview - mtviewdave
http://techcrunch.com/2015/03/21/the-terrible-technical-interview/
======
StevePerkins
I do almost all of the technical phone screens for my company. The process
starts with a "where-do-you-see-yourself-in-5-years" personality screen, with
an H.R. drone or outside recruiter. It moves on me, and then on to a brief
"homework" coding exercise that we ask people to write and submit. If we like
their code sample, then they come in for the panel of "whiteboard-exercise"
people who conduct the face to face round. At my stage, I seldom bother with
certification style questions about the programming language, or with highly
abstract thought exercises, because other people will do those things in the
face to face stage.

Typically, I ask questions that give me an idea about the candidate's depth of
experience and awareness. For example, " _I see that you 've spent X years
using Subversion for source control. What are your opinions on trunk-first
development vs. branch-first development?_" I ask questions that speak to
practical experience and design ability, without getting into too much depth.
For example, " _Pretend that you 're using an OO language to build an
application for <insert purpose here>. Just off the top of your head, what are
some classes that you would expect to see in your class diagram?_". Etc.

I find that this level of questioning is a much better screening tool than
trick questions about programming language quirks or the minutiae of
frameworks, or cliche puzzles about how many golf balls fit on an airplane.
However, I groan and roll my eyes when I hear people challenge the need for
technical interviews at all. __Yes __, they are necessary.

Having performed a thousand interviews by now, I am _awestruck_ by how poor
the software development talent pool is. I am aware that the Bay Area is
overrepresented in HN's readership, and that crowd tends to take for granted
the talent level found in the technical equivalent of Mecca. However, I assure
you that the rest of the planet is dominated by sleepy line-of-business
developers... who have all the passion beaten out of them in the first 5-10
years, and spend the rest of their career just phoning it in and not growing.

I sometimes ask candidates what the letters "MVC" stand for. The successful
response rate is around 50/50\. I ask candidates to briefly explain the
_advantages_ of the Model View Controller pattern, and only 10-20% can field
the question. We bring in Java and C# candidates who have been working with
their respective language for 10 or 15 years, and they get COMPLETELY EXPOSED
during the face to face round when asked a series of basic certification exam
style questions. Nothing tricky, just core fundamentals.

People who post here are _not_ the norm. The "norm" is atrocious. So yes,
unfortunately we all must endure technical interviews... to filter out people
who have enough confidence or personality to excel in the other interview
segments, yet are utterly useless.

Let's put it this way: the more experience I've had with interviewing, the
more selective I've been in my own job searching, and the more aggressive I've
been in my salary negotiations. If you are really good, and are located
outside of San Francisco, then you are worth your weight in gold and should
value yourself accordingly. You wield _tremendous_ leverage once you make a
strong showing in a technical interview.

~~~
mtviewdave
> "I see that you've spent X years using Subversion for source control. What
> are your opinions on trunk-first development vs. branch-first development?"

Actually, this demonstrates a problem right here. It's not clear to me what
you mean by these terms. A quick Google search ("trunk-first vs branch-first
svn") suggests that you're not using common terminology.

After thinking about it for a minute or two, what I _think_ you're asking is
"should all development happen on a common branch, or should developers create
separate branches for individual features/fixes, merging back into the
mainline when finished". And, indeed, I'd be happy to have a conversation with
you about this.

But if it takes me a couple of minutes to figure this out while just sitting
here at home, in the pressure of an interview, I'm probably going to fumble,
or say "I don't understand what you mean", which will make me wonder if I've
blown the entire interview. This despite the fact that I've been programming
for 20 years and have used a number of version control systems (CVS,
ClearCase, P4, Subversion, and most recently Git).

~~~
StevePerkins
In an actual interview, I would only pull out that specific example for a
candidate who's worked with CVS and Subversion almost exclusively. The actual
question is also a bit more verbose:

 _Have you worked for a company that strives to do as much development as
possible in 'trunk', creating a release branch near deployment time for
production bugfixes? Have you worked for a company that preferred to branch at
the outset of new development, merging back to 'trunk' periodically? What did
you find to be the strengths and weaknesses of each approach?_

Their answer lets me read between the lines and gleen much more information
about their work history. Have they worked in settings where multiple work
streams were in development simultaneously? Do they have substantial
experience in collaborating without trampling on shared resources? If so, then
they usually mention the different pain points in merging. I don't consider
there to be a "right" or "wrong" answer to this question, and not having a
clever answer certainly doesn't disqualify someone from being a strong
programmer. But it does help to level-set, and identify candidates who think
like team leads or might be well suited for responsibly beyond raw coding.

However, I don't want too much text between parentheses that are injected into
a long sentence. So you get "trunk-first vs. branch-first". :)

~~~
tptacek
Exactly why is it that you would making a screening decision based on version
control practices? Of the many things you'll need to ramp candidates up on,
this seems like one of the very easiest. Not only that, but because most firms
do release management a little differently from each other, there's some VC
ramp-up effort you'll need even with people who have mastered your VC tool.

I feel like dev interviews are full of questions like this, things that
require some expertise and experience to answer, but do virtually nothing to
predict on-the-job performance.

When you ask an interview question, you are _pricing candidates_. That's
obvious when you think about it: you're screening, and so your questions alter
the supply of candidates that will hit the bottom of your screening processes.
Fewer selectees -> poorer employer BATNA -> higher prices.

Do you really want to price candidates based on how they use version control
tools? How much are you willing to pay extra for people who have a lot of
experience with different VC methodologies? Are you sure the weight of your
questions about VC match up with the (hopefully minimal) premium you're hoping
to pay for VC expertise?

~~~
StevePerkins
I'm not sure that you fully read the parent comment. I could care less about
their Subversion expertise. My company doesn't even use Subversion anywhere.

However, if you list " _10 years experience with <Technology X>_" on your
resume, then you should absolutely be prepared to discuss your experience with
<Technology X> at a high level. Not anal minutia or contrived trick questions,
but certainly you should be able to respond to an open-ended question about
the basics with enough context to show that you weren't lying to pad your
resume.

More importantly, as I explained in the parent comment, I am interested to see
if their response reflects experience with multiple teams working on parallel
over overlapping efforts within the same codebase simultaneously. Everyone
_says_ that they have experience like that, but if you poke a bit deeper you
find that half the time it's exaggerated. They may have worked in a context
with multiple teams, but affecting the same area of the application or system.
If you _have_ had legit experience of this kind, or at least show a high level
of insight in talking through the issues that can arise, then you might be
considered for a team lead role sooner than you otherwise might have been. As
I said earlier, it's not a "right or wrong" question that can disqualify you
from being a capable programmer... it's a "level setting" question that helps
gauge which level of responsibility you might start out with.

~~~
tptacek
There is no less productive form of interview than the "resume validation
interview". Resumes are practically useless in the best case. Here, you seem
to propose paying a premium for _candidates who can properly estimate their
facility with VCS systems and then cogently discuss that estimation in an
interview_. That can't _possibly_ be a skill relevant to building software on
your team!

~~~
santoriv
Personally I think it's an excellent question. He's probing into a candidate's
understanding of _why_ you use version control systems and _why_ different
approaches might be used.

If you have a significant amount of experience then hopefully you have seen
enough different situations to be aware of the advantages and disadvantages of
each approach.

It's really a question to figure out whether or not you can reason about high
level concepts.

~~~
tptacek
Stipulate that it is. Why is that a good idea for an interview?

~~~
jes5199
personally, I think the SVN question is a pretty good example of "can you talk
intelligently about _any_ aspect of _any_ complex system you've ever worked
with?". If a candidate can explain _one_ , I'll recommend to hire them. The
resume provides a good list of _possible_ subjects to discuss, otherwise I
just cast about randomly, try to chase threads in the conversation until I
find something with some depth.

~~~
politician
I completely agree with this - asking about VCS is a cross-cutting question
that nearly all candidates have experience with unlike, for example, graphics,
desktop, or cloud experiences. So, requesting that the candidate explain their
VCS workflow is a way to generate a practical and relevant data point to use
when evaluating multiple candidates.

For example, the 1-in-a-million candidate that has used git with a CI/CD
configuration (note: I'm also outside SV) versus the candidate who uses TFS
("git? You mean Github? _Yes_ ).

~~~
tptacek
I've carefully read every comment on this subthread, and here's what I think:

This question is a trifecta of ineffective candidate screening tactics:

(a) It's a technical screening question, one a strong candidate could get
wrong, based on a technical aptitude that is trivial to teach on the job and
thus rarely worth paying a premium for.

(b) It's a _subjective_ technical question, for which reasonable engineers can
have differing opinions, which means it's an outlet for interviewer
subconscious bias. Did the interviewer just eat lunch? Candidates will do
better on this question if they have.

(c) It's a tea-leaf-reading question about engineering/team management: it's
superficially and overtly about technology, but subtextually about a bunch of
other things. Let's hope the candidate realizes that.

A typical interview lasts about 60 minutes. Let's say it takes 10 minutes to
pursue this particular line of inquiry. That's 16% of your interview you're
spending with a question that _greatly_ rewards people who are _good at
talking about technology_. Worse, if you ask that question early to a quiet
but excellent candidate, you can psych them out, which means you pay for that
version control question in every other question you ask.

It is totally reasonable to assess soft skills and team compatibility. But you
have to _design an interview_ that does it. You can't improv it based on
candidate resumes.

Avoid questions like this.

This thread started with someone saying they're pretty good at interviews. It
turns out that they try to assess soft skills with technology questions based
on the luck- of- the- draw of candidate resumes. Candidates with effective
resumes will have an easy time passing these interviews. From the comments on
this thread, that _obviously sounds reasonable_ to some people.

I submit: those people are not competing for talent. They may think they are,
but the real contenders in this market won't be OK with letting good
candidates slip past because their job-hunting skills aren't finely tuned. In
fact: they'll do the opposite: those candidates are _steals_ in this market.

The original commenter acknowledged that when he said that his experience was
that the market was full of poor candidates. If that's the case, you
especially can't afford to filter out effective devs because they fail to
impress you when they explain how they use version control, or when their
resume overstates their facility with version control.

~~~
politician
Thanks for being open about your perspective on the interviewing process, and
for taking the time to follow up on these threads.

Do you believe that technical phone pre-screens are ineffective in general?
From what I've read Matasano doesn't pre-screen candidates, but provides
complementary study materials instead. Is that because the subject matter is
specialized? Would you approach hiring web dev roles differently?

From your remarks, it sounds like you would reject the practice of asking
open-ended interview questions (e.g. describe your workflow, describe a
typical day, describe a recent project) due to interference from the
interviewer's bias. What, if any, value do you place on open-ended questions?

~~~
jerematasno
In general, my feeling is that the Matasano process (which I currently manage)
works outstandingly well where there isn't a flood of qualified candidates. If
you have a glut of folks who are ready to start working, you can get away with
a terrible process.

We do free-form technical interviews, but only to try to detect candidates who
really aren't ready for the work-sample challenges. Our in-person interviews
are standardized and try to evaluate consulting/architecture skills that are
hard (impossible?) to measure without people. These involve open-ended
intermediary questions, but the final answers are structured.

The bottom line is this: you _cannot_ compare candidates using free-form
interviews. You _must_ compare candidates who are going to be doing similar
work. Thus, free-form interviews have no evaluative value.

------
fsk
False negatives are a lot worse than most interviewers think. (False negative
= you pass on a good candidate, false positive = you hire someone who turned
out to be unqualified)

If a "good" candidate is a 1-in-100 find, then each false negative means you
have to look at another 100 candidates.

Also, if you decrease your false negative rate by more than you decrease your
false positive rate, you're actually hiring MORE bad candidates (while
spending more time interviewing). I.e., every time you pass on a good
candidate, that gives you another chance to make a mistake and hire someone
bad.

Your odds of hiring one specific "bad" candidate may be small, but if most
candidates are "bad", that actually makes it more likely to hire someone bad
each time you pass on someone good.

~~~
sytelus
No. Many companies have hiring rate of 10% [1] which means hiring 1 candidate
may cost you 10 person-days assuming full day interviews for each candidate
[2]. You can probably double that to cover the cost of phone screens, so let's
say 20-person day of work per good hire. Assume you had 4 false negatives just
because you are so aggressive and so a good hire ended up costing you 100
person days instead. This cost is still _tiny_ compared to damages that a true
negative would cause otherwise, i.e.,

(1) loss of entire person year or even two because annual reviews need to
accumulate evidence for HR to fire (may be less time if you were startup
without real "HR")

(2) amount of cleanup other people have to do after that new bad hire

(3) loss in moral for good people in your team who now perceives your hiring
process at the company as "broken"

(4) delays + bugs introduced in product because actual work probably didn't
got done or badly done despite of you filling up your headcount

(5) amount of money lost in salaries, signing bonuses, office space and
benefits (typically > $200K)

(6) amount of productivity lost because of wasted time by good people in the
team trying to "ramp up" your true negative

(7) emotional stress you caused to good people wondering them about _their_
job stability and to managers who wasted their time in months of paper work
and lot of explaining

(8) emotional stress you caused to your true negative being fired who had
moved across the country for you, bought a house on mortgage and had 3 school
going children

(9) Most likely, if you are big company, true negative didn't actually got
fired because hiring manager never wanted to admit it. S/he was encouraged to
join another team or role or even learned political tricks to get promoted
contributing to ongoing _bozo explosion_ [3]

(10) I could go on and easily justify probably 3X-10X loss compared the case
if true negative was avoided

Footnotes:

1 - Many companies specify their hiring rate over _total_ resume they received
which is wrong. I'll use 10% as total number of full interviews that needed to
be conducted which is average of 5%-15% at most companies.

2 - This is bad math. Assuming random trials, it would be actually 5 person-
day on average but intuitive approach doesn't produce entirely bad results
here so we will go with that.

3 -
[http://guykawasaki.com/how_to_prevent_/](http://guykawasaki.com/how_to_prevent_/)

~~~
itsdrewmiller
You didn't actually respond to his comment. Assuming, for example, you have:

* 99% of applicants are bad * 50% false negative (You look over about one good developer for every good developer you hire) * 1% false positive (one out of a hundred bad devs can snooker you into hiring)

In that scenario, you're twice as likely to hire a bad dev as a good one. And
if you halve your false positive rate by increasing your false negative rate
by 50%, you're still twice as likely to hire a bad dev, it will just take you
twice as much work.

~~~
fsk
I see what you're saying. I was pointing out the dangers of false negatives,
but he responded that he hires 10% of the people he invites for an onsite
interview.

Even if you hire 10% of the candidates you on-site interview, that says
nothing about your actual false positive or false negative rate. For all I
know, he could be weeding out all the top candidates at the pre-interivew
stage, and then hiring the best of a mediocre group of people.

It's easy to measure your false positive rate, people you are forced to fire
(or wish you could fire if not for corporate bureaucracy).

It's harder to measure your false negative rate. The only way you could
measure your false negative rate is to pick a random sample of people who fail
your interview, AND HIRE THEM ANYWAY. (However, that could be a lawsuit risk.
It would be unfair to the people who hire despite failing the interview. A
small business couldn't afford to do it, only some huge corporation could do
the experiment.)

Also, I doubt the ability of most businesses to identify the best performers
AFTER THEY ARE HIRED and working there for a couple of years.

~~~
sytelus
No, I did not say _I_ hire 10% of candidates I interview :). What I said was
that's a fair estimate in industry.

I feel you are truly confused about FP and FN. Whether there are 99% bad
developers out there or if you hire 10% of candidates you interview - these
both quantities are _independent_ of FN and FP. FN says that you are turning
away X good people and it's again _independent_ of FP which ultimately decides
how many bad developers you would eventually end up hiring regardless of other
3 quantities I mentioned. See here:
[http://en.wikipedia.org/wiki/Confusion_matrix](http://en.wikipedia.org/wiki/Confusion_matrix)

It's not easy to measure FN, FP, TN or TP. Even good people fail due to
different reasons like bad manager and bad people may succeed despite of
mediocre skills. Looking at who you had to fire or who got promoted doesn't
give accurate measurements at all although they may serve as weak proxy. The
scenario I described was hypothetical to point out that cost of FP is far more
higher than additional cost in hiring due to FN.

~~~
fsk
If you increase FN and FP stays the same, you are more likely to make a bad
hire. Do you understand this?

~~~
sytelus
May be I'm completely missing something here but my understanding is this: FP
= (good hires you made) / (all hires you made). Your _likelyhood_ of making
bad hire is 1-FP. If you hired 100 people and your FP was 10% then on average
you would have 10 bad hires on your team. So FP determines the number of bad
hires you would eventually have. FN has nothing to do with it - it only
determines how long before you make a good hire, it doesn't influence actual
number of bad hires you will make.

I'm using standard terminologies here. There are plenty of textbooks and
articles on confusion matrix, precision, recall, RoC etc. Not sure what
definitions you are using to arrive at conclusion that FN increases the number
of good hires (it only increases _effort_ ).

~~~
fsk
You aren't using statisics correctly.

FP = probability, given a bad candidate, you will hire him

FN = probability, given a good candidate, you will pass

Suppose 100 bad candidates, 10 good candidates

FP=10%, FN=10%

You make 10 bad hires and 9 good hires

FP=10%, FN=20%

You make 10 bad hires and 8 good hires.

So increasing FN lowers your yield.

Your statstic (good hires / total hires) tells you nothing about your actual
FP or FN value.

If you don't get it, I'm not wasting time on you anymore. You are very
dangerous. You think you know statistics, but you don't.

~~~
fsk
Your statistic (good hires / total hires), using the jargon from your link, is
precision or positive predictive value (PPV).

Using the math from that link, if you decrease FN, then PPV increases.

~~~
sytelus
Sorry, I did mixed up precision in my reply. I just got time to think about
this whole debate more carefully and I realize you are actually right if we
fix up some of the terminology you have used. The mis-statements and confusion
on my part has occurred due to this terminology differences.

First FP and FN are _not_ probabilities. They are just unbounded numbers. This
may feel pedantic but in a moment I'll show you why this is critical. Let me
draw the confusion matrix first (G = Good candidates, H = Hired candidate
etc):

\ H NH \\--------- G | TP FN B | FP TN

What you are referring to as probabilities is actually _False Positive Rate_
or FPR and TNR respectively which is defined as follows:

    
    
      FPR = FP / (FP + TN) = FP/B
      FNR = FN / (TP + FN) = FN/G
     

Now the quantity you are after is probability that given you did hiring and
ended up with good guy which is, nothing but _precision_ :

precision = P(G|H) = TP/H

So how do we get TP to calculate precision if we only knew FPR, FNR, G and B?
I did little equation gymnastics using above and got below:

TP = G - G _FNR H = TP + FP = TP + FPR_ B

So now you can plug this in to above equation for precision and find that as
you increase FNR, precision goes down while you keep FPR constant. So you are
actually correct. Although it might look like unnecessary exercise vs
following intuition I think above equation can actually help calculate exact
drop in precision and multiply that with cost of FP vs FN to get the operating
sweet spot. On my part I need to do some soul searching to figure out why this
didn't triggered to me before :).

------
leadgenguy
I am a self taught PHP programmer. Been working in tech as a freelancer since
early 2000s. Built real applications.

I thought I was pretty good so a few years ago I moved to Silicon Valley to
get a startup job. Boy was I wrong. Even though I had created real
applications and knew MVC, etc. etc. I was basically told I was worthless
because I didn't know algorithms, unit testing, etc. etc.

So I moved back home (Tel Aviv) and started my own company (lead gen market).
Programmed everything myself. Last year I did over $2 mil (70% margins) and
this year I am on track for $3.5-$4 mil in revs.

The technical interview was the best thing ever for me because if I had passed
I wouldn't be where I am today.

~~~
bhayden
I really like seeing this. I also write PHP for my projects, and I know it's
terrible. I have no delusions about this. I use mysql_query despite knowing
it's deprecated and possibly insecure, and if I'm building something that
doesn't store any sensitive data and the worst case scenario of someone
finding an exploit is that I have to restore from a snapshot, I really just
don't care. I focus on building the product quickly and efficiently. Let the
programmers of the world thumb their noses at me, at least I'm building
something of value for me instead of some company.

~~~
mahyarm
The thing is when the company gets big and its not just you, your going to
wish your original language was something more scalable.

For example, you tube engineers probably curse that the thing was made in
python, because in a 1000 engineer org, making changes can break things you
wouldn't be aware of elsewhere. While something more statically typed like
java or go would break on the compile step instead of the run time step.

Python was fine when the project was 1 - 20 engineers, but it became a
liability later on.

That is why people want you to spend a week or two to learn something better
and more maintainable, so you wont curse your future self later.

But if your making small build utility type things, then it's fine. Contact
websites for small firms. Or if it's for your quick test project, etc.

------
bsder
Look, if you want to talk about code, the easiest and best way to sidestep
this issue is to have people bring code (somewhere between 200-1000 lines) to
the interview. While they may not have a github presence, they've probably
written something on a computer outside of business at some point.

I really don't understand why more people don't do this.

It doesn't even matter if it is their own code. The fact that they will have
to talk intelligently about it means that it will accomplish its task.

I used to get this issue interviewing with VLSI chip companies _all the time_.
"Do you know Perl?"\--"Yes, I do, but I don't know _YOUR_ particular favorite
subset of that write-only language" doesn't go over well in interviews.

After getting tired of getting dinged on questions about a language that I had
used for almost daily for 5 years and that I abandoned _for good reason_ , I
took an old program of mine from Python and ported it to Perl. I bring both.

Now, I have something concrete to talk about, can actually _compare_ the
points of two languages, and most interviewers realize that "Gee, you're
probably better at this than I am."

This shifts the conversation from "Technical Jeopardy!" to "Ah, you know how
to program, let's look.", "I did this. Here's why.", "Oh, that's interesting.
What does that do?", etc.

~~~
Darmani
I tried this with the first few people I interviewed. None of them had any
code from outside work to bring. (This includes a brilliant and hardworking
coworker that I practice-interviewed -- he just hadn't coded outside work in a
very long time.)

We wound up switching to giving code challenges instead.

~~~
scoggs
Why not pull down a decent example of code that fits their resume's skill set
and having them go through it with you line by line to tell you what they
would do differently or why they like it, etc.?

------
mingmecca
I lose about 20 points of IQ in a technical interview. It's nerve racking (I
wear black shirts to hide the sweat) and it cost me a job working for a well
known VR company. I passed the 'practical', which was a 2 hour coding test to
generate a working game application, but the tech interview probably did me
in. I've been programming games for almost 20 years, and as far as I know I'm
one of the few who has done lead/senior level work on games that were both #1
and #2 simultaneous chart-topppers, but I still couldn't get the job. Their
loss I guess, but it still sucks, as I was really enthused about what they
were doing and could contribute significantly. I was about 10 years older than
their typical engineer however, so maybe ageism played a part. I'll never
know. I just wish companies would look at my past body of work, see that I'm
friendly and un-abrasive, and bring me aboard.

~~~
blister
Have you thought about jumping into the world of Serious Games?My company is
usually looking for experienced game developers with industry talent. The
"Serious Games" side of the house is (to me) twice as rewarding as the other
side. We don't live and die on the finicky sales cycle of the public and get
to focus on delivering cool products that are already paid for by one
customer.

If you want to chat more about it, hit me up. me (at) ericharrison.info

~~~
philsnow
> already paid for by one customer

.... tactics sims for militaries ?

------
foz
In my organization, we start by looking at what the candidate has actually
done. We ask for a github profile, and any side projects they might have
worked on. If the CV and body of work look interesting, we go on to a phone
screen - general questions, clarifying points on the CV, explaining the job,
answering questions.

For the on-site interview, we ask candidates to bring their laptop. We advise
them to use a typical, comfortable development environment. We've seen
candidates struggle with the latest Ubuntu, installed that morning to impress
us. So we don't want that.

During the hands-on interviews, we ask to see a side project, or any code they
are mostly responsible for, and familiar with. We ask them to explain code,
maybe change something, refactor a test, etc.

What we don't tell candidates is that this part of the interview is also about
how they use their tools. It's important to see how good they are with their
editor, command-line tools, can they type well, do they get easily distracted,
and so on.

One of the most effective interviews we do is for project planning. A problem
is explained in detail and the candidate is asked to design a solution. Not in
code, but to talk it through in detail, drawing or writing docs/stories if
needed. This phase helps show us how they break down a project, ask questions,
negotiate features, and look for opportunities to reduce complexity. Bonus
points for making a pen-and-paper wireframe or throwaway prototype.

What we refuse to do is the "puzzle" problems, whiteboard code (which makes no
sense), or tricky technical questions. We instead want to find people that use
best practices, don't re-invent the wheel, tackle problems pragmatically, and
are good with their tools.

Over time, this approach seems to work well. However, we also discovered that
we have to re-train and test our own interviewers. Without that step, the
process can change unexpectedly, become inconsistent, or unfair. Don't just
assume your staff is interviewing well - take time to check it out and help
them get better.

~~~
bsder
> In my organization, we start by looking at what the candidate has actually
> done. We ask for a github profile, and any side projects they might have
> worked on.

Fail. You just killed the quality of your pool.

You've just negatively screened against people with a life (experienced, 30+
years old, generally with a family) and screened for people with no life (aka
single, male 20-somethings).

For an example, if John Carmack hadn't been able to release his employers
source code (like many people), your process would screen him out (no github
or side project presence).

~~~
rupert_murdaaa
> You've just negatively screened against people with a life

That kind of works both ways, though, doesn't it? As a 30 year old developer
with a family, I don't want to waste time interviewing somewhere that has the
expectation I'll work outrageous hours. If I get filtered out early in that
process because I'm not involved in multiple OSS projects or whatever, all the
better for me.

That being said, resumes are a really shitty way to get an idea of someone's
experience and talent. If I'm hiring someone and they can show me a github
profile or blog entries or a Stack Exchange profile or slides from a local
user group presentation or _anything_ besides their resume, it really helps me
get to know them better, and -- all else being equal -- will probably set them
apart.

------
jimbobimbo
I've been on both sides of the process and in both cases I prefer the
whiteboard.

As an interviewer, I simply don't have an hour or two to evaluate someone's
project. I want to see a strong resume and ask a few questions to understand
whether the resume is real or not. A simple whiteboard question or two is
great for that. You can always see whether the person is capable of writing
code and it always brings up questions around the choices they made.

As an interviewee, I rather not be engaged in a live coding session: the
whiteboard is a very efficient tool to get an idea across, without sweating on
syntax errors, searching for API documentation and whatnot. Test projects - I
have a job, thank you very much.

~~~
jqm
If you don't have an hour or two as an interviewer, you probably aren't
getting very good results. This is a pretty small investment of time
considering what is at stake.

~~~
jimbobimbo
I actually hired two best performers at my previous company this way. The
amount of people interviewed total is about 30.

------
rsuelzer
I remember with horror one technical interview I had. I was asked to complete
a fairly basic problem, which I had actually just practiced a few hours before
(the fact you can practice for these interviews indicates how questionable
they are). As soon as the question was asked, I proceeded to start typing the
answer and explaining what I was doing when suddenly my mind went blank, I
struggled for nearly ten minutes to regain my confidence but by that point it
was too late. The interviewer quickly ended the interview and told me to apply
again after I had more programming experience, since apparently he was able to
conclude the extent of my knowledge based upon my answer to a single question
made during stressful circumstances.

~~~
army
Um, you can practice for anything that involves communication or demonstrating
your skills. Its just arrogant to think that you can go into an interview cold
and rely on the interviewer to tease out your positive qualities. If they're a
good interviewer they'll do their best to do that but its also on the
interviewee to convince the interviewer that they're a good fit for the job.

I wouldn't present on a technical topic without preparation, because I'm not
going to do a good job of explaining it or communicating the ideas - I'd be
wasting the audiences time. Same with interviews - you need to be prepared to
explain your work and put your experience in as favorable a light as possible.

------
golergka
I'm going to go against the popular opinion here: I don't believe that
technical interviews are broken and need fixing. Of course, we know as
engineers who deployed code filled with hacks and known bugs to production,
everything is broken. The difference is only how broken it is. Let's fix our
approach to quality and security, ability to estimate and other things that
are much more broken first.

~~~
frostmatthew
> I don't believe that technical interviews are broken and need fixing

I think one of the reasons so many developers feel the process is "broken" is
because it's all they know (i.e. most developers didn't have a previous
profession).

For most jobs it's not practical or possible to get _any_ insight into how a
person will do that job prior to hiring them. Whiteboarding isn't intended to
perfectly mirror "real life coding" \- it's intended to give _some_ insight
into one's ability to write software to solve a problem. It's not perfect (and
it can certainly be done extremely poorly!), but it shouldn't be dismissed as
broken or useless any more than one should suggest actors shouldn't have to
audition for roles.

------
tptacek
Requiring a side project from candidates would have cost us most of our best
hires.

~~~
buchweizen2
I was a bit surprised that the article first talks about how bad it is to pass
on good candidates that don't interview well on whiteboards and then suggests
to pass on candidates that don't have side projects. Are all candidates
without side projects not good?

~~~
mahyarm
Maybe we should just offer a choice of what kind of interview candidates want.

------
rifung
I completely get that technical interviews aren't great at accurately
predicting success on the job.

What I don't get is all the whining that seems to happen from interview
candidates who seem to think they are hot stuff and deserve to get hired but
blame it on a bad interview process. It's not like the interviewers aren't
aware of the shortcomings of the technical interview.

I have yet to learn of another industry where people routinely blame
interviewers for their being rejected. I mean it seems like in other
industries people are just hired almost solely based off resumes, and while
everyone also realizes it sucks, they don't seem to think the interviewer is
an idiot for not hiring them.

------
msoad
I live in Silicon Valley so I have tons of Software Engineer friends. I also
have a friend who's a heart surgery researcher at Stanford. When we talk about
our hiring process he just laughs and thinks its impossible to do something
remotely close what we do in Software in their field. It seems Software is the
only field that people have to do some serious work for the interviews.

~~~
swatow
Maybe they should. Studies have found that ratings of a surgeon skill based on
a video recordings of surgery, are predictive of patient outcomes[0]. The
problem with associating prestige and respect with not stooping to being
tested, is that testing can be useful and necessary.

[http://psnet.ahrq.gov/resource.aspx?resourceID=26950](http://psnet.ahrq.gov/resource.aspx?resourceID=26950)

------
realrocker
I bet I can put together dozens of people who think OOP and MVC are bad
programming paradigms, another dozen who can make war on the quantum of
documentation. We already know the solution to this problem: We need to hire
people who have the skills to hire people and willinglines to do it too. Just
being a senior doesn't cut it. Too costly I guess but is it costlier than
making bad hires?

IMO successful startups are the one's where founders had the skill to identify
and hire the right talent. What I don't understand is why big companies are
not creating a separate division of engineers with primary task of hiring? Is
it because that if such a division existed the people there would eventually
lose the skills being cut off from mainstream engineering tasks? I am not
sure. But it seems to me that this can be solved by looking at hiring task as
an engineering task. What's wrong in seeing hiring engineers as another piece
of code which needs to be carefully thought and polished?

------
nperez
If your only goal is to measure the ability to code, then doing a whiteboard
interview isn't the best way to do that IMO.

But what about positions where you will be expected to give presentations, do
pair-programming, or mentor junior developers? I think whiteboard interviews
can be a measure of your ability to take technical concepts and
illustrate/explain them clearly to a team.

I used to really hate whiteboards, but as I grew into positions that required
more leadership, I realized that I personally needed to develop better
presentation skills. If a potential employer were to test me on that, a
whiteboard test wouldn't seem unreasonable to me.

So I have mixed feelings on this. Not every hire needs to be capable of being
a teacher or delivering a solid keynote speech.. some positions would be best
filled by someone with those skills.

~~~
RogerL
If you want to measure giving a public presentation....have them give a public
presentation.

If you want to measure their ability to pair program ... pair program with
them.

If you want to measure their ability to mentor ... have them teach you
something they know.

I want to know how you do X, and measuring X is easy, so please do Y as a
proxy is never a good approach.

I have never, in my 25+ year career, ever had to whiteboard a problem, where
the other person knows the answer (that is laughable - I've given correct
answers and been told they are wrong), I don't, and there are hundreds of
thousands of dollars at stake. It just has nothing to do with the job or on
the job performance.

~~~
lukeschlather
Finding out how someone reacts to being told they are wrong when they know
that they are right is a totally valid interview technique.

Furthermore, I think past basic fizzbuzz questions, complex interview
questions are good, but someone failing to get an objectively correct solution
to an objectively difficult problem is just one signal among many. The goal of
asking the questions should be to gather lots of other signals about how you
approach writing software, not to pronounce you right or wrong.

~~~
fsk
If someone was playing emotional tricks on me during an interview, such as
telling me I'm wrong when I'm right, I'd lose interest in the opportunity.

One interviewer tried yelling at me for no reason, just to see if I would
react or flinch.

~~~
pmiller2
Seriously? People do that? I hope you walked out.

~~~
fsk
I was polite and stayed until the end. It was through a headhunter, so I
decided to be polite. I knew that was a no-go interview after 1 minute, when I
saw that everyone working there was Indian.

------
danans
Articles demonizing coding interviews always attack strawmen that are the
worst kind of interview questions: 1) Trivia questions (i.e. "what is the name
of the method to do X in Java") 2) Questions with precisely 1 solution (you
either get the one solution or you don't).

These are indeed useless technical interview questions.

But that doesn't meant there aren't good technical coding interview questions.
I tend to favor the kind that present a candidate with some example data and a
set of constraints or patterns, and ask them to write code that analyses the
data for those patterns or constraints and reports them. This sort of problem
is ubiquitous in my domain. Importantly, I always choose problems for which
there are multiple possible attack angles, not just one. And I don't give a
hoot about syntax errors, or what language they use.

This sort of question gives you a good sense about the candidate's analytical
capability (breaking down the "word problem"), and their ability to translate
their problem-solving thought process into code. Because there are always
multiple angles of attack, candidates have some leeway to exercise creativity.

In the end, it's not terribly important to me that they get the optimal
solution. I do care whether they demonstrate strong analytical capability in
the literal sense, meaning they can decompose the problem and the associated
programming exercise into their logical parts and implement them. I also look
for good communication skills in the questions they ask when reasoning through
the problem - this is something that only an in-person technical interview can
reveal, AFAICT.

There are probably many smart candidates that don't do well on these questions
because they're just having an off day, or nervous, or don't perform well
under pressure. I sympathize with them, because I've been there and felt all
of the above.

But if the goal of a technical interview is to assess a candidate's analytical
and coding abilities, and their ability to do both simultaneously, there is no
shortcut I know of to just giving them a role-relevant problem to work on.

EDIT:wording.

------
ffran
I usually countered FizzBuzz question with an offer to show of the source code
of a small project I wrote while studying. While some didn't care, most
interviewers were actually happy about it and let me do it. I tell them about
the problems I had and how I approached them, usually leading to a question
from me on how the company had similar problems and how they solve them. In my
experience this "opens up" the interviewer(s) to talk a bit more honestly
about development practices in the company.

On test projects. I generally expect that companies (after a pre screening)
put similar efforts in me as I put in them. If you let me do a two day
project, I'd expect that one person (preferable my future boss or a future
colleague) to show me a "company fitting" solution and take the time to discus
my work and theirs. If you're not willing to spend that time I'll most likely
think you don't respect my time. Which is a factor in choosing a company. I
understand you are busy, I hope you understand that I’m too.

Edit: not a native speaker, sorry for bad english

------
jaredmcdonald
> It is time for engineers–especially excellent engineers for whom demand is
> high–to start to flatly refuse to do whiteboard interviews.

This might be a viable strategy for people who have a well-established
career/credentials/references (etc), but for junior-level candidates still
trying to prove themselves (such as myself) I can't see this working out too
well.

~~~
tonyarkles
That's one of the harder parts of the process, and it's partially because of
poor expectations on the part of those doing hiring.

To me, it's absolutely ridiculous to expect to hire a junior developer and
have them come with a fully-developed set of skills. If they do, that's great.
But if you're hiring someone fresh out of school, you've got to be approaching
it as hiring someone that you're going to train and mentor. For me, the number
one thing I look for in a junior developer is the ability to _learn_.

Here's an example from early in my own career. I was just finishing up my 3rd
year of a combined CS/EE program, and looking for a summer job. I got in touch
with a biology lab that needed a developer for the summer to build some (very
cool) software to support the neurophysiology experiments they were doing. I
looked at the job requirements and thought "well, I don't know most of this
stuff, but I'm sure I could pick it up."

The interview progressed like this:

> Do you know Python?

I'd heard about it, but have no real experience with it. I downloaded it last
week and started playing with it though, and it doesn't seem too different
than other languages I've used.

> How about VisionEgg (neurophysiology module for Python)?

Well, I downloaded it at the same time I downloaded Python. I've managed to
get a window to open up, and I'm displaying a square that's got a cool
animated habituation pattern on it. (Note: 1 week prior, I had no idea what a
habituation pattern was) I do have a bit of OpenGL experience from a class I
took, and that's the underlying library that VisionEgg uses.

> Well, so far, you're the only applicant who has actually made an effort to
> look at the specific tools we're using here. I've got one other applicant
> coming this afternoon, but unless they somehow have more experience with
> these tools than you do, the job is yours.

It turned out to be a great experience, and they hired me back on the
following summer. I went from being a total Python noob to contributing
patches back into VisionEgg. I think most junior positions should probably
progress like that. Give me a keen junior developer, and let me shape and
mould them into a not-junior developer.

The flip side to this: if you're expecting the person to be productive on day
1 or 2, you'd better be hiring someone with experience. Whether or not they
have code they can show you, they should be more than capable of going into
serious detail about past projects they've worked on (within the confines of
NDAs and such, of course).

------
nevaben
I've found this technical interview process to work very well. This is done
after the initial phone screen.

1\. Show them a problem with your product along with the code. For front end
developers it could be how form invalidation errors are being presented to
users. 2\. Ask them to figure out what why the code is doing this and observe
them troubleshoot the code. 3\. Tell them to fix the problem in the code and
observe them apply a fix, test, and debug it. 4\. Ask them to architect a
better solution to the problem and to explain what makes it better. What would
would be drawbacks to their solution.

The benefits of this approach is that you can evaluate directly how someone
solves problems, not how well they communicate, nor how knowledgeable they
are. It also helps me judge how fast they are. Sometimes I ask people to
estimate how long it would take to solve this and time them. The last step
helps establish how they think through their solutions and how well they can
communicate their ideas.

Additionally, I get to see how quickly they might get up to speed with our
codebase and I can hear other solutions to some of the technical problems we
are facing which is incredibly beneficial as a small startup.

------
ditonal
In my experience, a lot of companies are combining _all_ these things. So
you're expected to do a phone interview, a test project/coding test, a
whiteboard test, and the management brown-nosing at the end of it where you
get to pitch that you've studied a hard technical skill your whole career just
because you're so passionate about getting woken up at 3AM on PagerDuty to
build _their_ vision, and definitely not because you expect to be paid for
your time and skill. The fact that you realize money exists and can be used to
pay for things like housing, education, and healthcare means you're not truly
passionate about technology.

The worst is the NYC tech scene where they have the exact same standards
(which they cargo-cult from the west coast), but they decided not to bring
over that aspect where engineers are respected and valued. Instead they
borrowed the west coast interview and combined it with the east-coast finance
style where programmers are considered clerical workers and cost centers.

NYC is actually a fun city to interview in because since it combines so many
different cultures, you can't even study for an interview because 5 different
companies will give you 5 different interviews. Finance companies still love
brainteasers and they _love_ mutexes, seriously, if you're interviewing at a
finance shop just memorize Java Concurrency in Practice and the
producer/consumer wikipedia article, because they are reading questions from
there, even though when you show up on your first day you'll have been better
off having read Spring in Action and Headfirst Enterprisey Design Patterns or
whatever.

In general I love how these companies have elite hiring standards but
incredibly mediocre interviews. You get asked the same questions over and over
again by these companies. Find the biggest sum in a list, find two items in a
list that sum to a given number, sort a list of integers/strings but keep
integers where integers were and strings where strings were, copy files with
ids to all servers with ids, etc. What's the difference between an abstract
base class and an interface (by a Python/Node shop), what is a closure, etc
etc. I've heard so many of the same questions repeated over and over. They
have "high bars" for their candidates but apparently their interviewers can
just use the first link off of Google or re-use whatever Facebook was asking
in 2010 when they got rejected by them.

And then at the end we have a "tech talent shortage". Whatever happened to
that part where we claimed a false negative wasn't a big deal? (glad the
article calls this attitude out).

Tech hiring is totally broken and then they claim there's a shortage. There
definitely is a shortage of qualified interviewers, not so much a shortage of
qualified candidates. I have a friend who I mentored into the industry, she's
very smart but absolutely a junior engineer, and yet she starts a new job and
was doing interviews with _no_ training 3 months later. They just threw her
into the lion's den and expected her to figure it out.

Coding tests are another great one because the exact same people who talk
about the importance and value of data throw all of that out the window when
it comes to evaluating them. There is no calibration or standards, generally.
One person is offended by a hardcoded file path but doesn't care about whether
you have tests, and another person is the direct opposite. Many people expect
you to write extensive optimizations for the 100Kb input file you were given,
another person sees that as absurd premature optimization. Whether you make it
through a code screen is entirely tied to whether your coding style happens to
jibe with whoever is reviewing you.

Again, the West Coast is better because at least compensation packages reflect
the hoops you have to jump through and the monkey dances you have to have
learned on cue. On the East Coast, anybody paying attention is desperately
trying to get into management by the time they're 30 because to do otherwise
is to be humiliated and infantalized during the interview process _and_ during
your tenure working there. My simple solution to all this nonsense is to make
sure your CTO and head tech managers are being made to jump through all the
same hoops with all the same standards. Drop this whole "Oh, the CTO is a
_manager_ role, he shouldn't have to worry about all that." (again, more of an
East Coast attitude).

Another incorrect thing tech companies claim that benefits them at the expense
of labor that engineers brainlessly parrot: "false positive are expensive
because firing is hard". No, it isn't, I've seen many people fired very easily
the second they're not up to standards. At worst they're instantly let go
because it's at-will employment. At best, their given some absurd Performance
Improvement Plan that establishes a paper trail so they can fire them without
severance. And if your employer tries to tell you people have come off of
those things, I have news for you, people lie, and liars are good at reaching
senior management positions.

Something else is how you can put in hours and hours of your life, and get
_no_ feedback, because telling you how you scored on an interview might expose
them to legal liability. Let's be real, the legal liability is when they
reject qualified people for things like "culture fit", and if your interview
process exposes you to legal liability, maybe it's because it's illegal and
unethical.

Another thing tech companies should consider is, instead of paying insane
money to recruiters to be pushy sales people who are trying to dupe engineers
into their low-paying positions, maybe just redirect that money to the
engineer instead, it's 2015, the days that sleazy recruiter types are
necessary to try to fast-talk an engineer into a position that's not good for
him are over because we have the internet and we can read your terrible
Glassdoor reviews.

I'm convinced one huge reason all this happens is to discourage job-hopping,
because in this market, liquidity would probably help salaries move up faster.

I know I sound bitter, but again, these are the exact same companies running
to the taxpayers to spend hundreds of millions of dollars of middle class
Americans money to pay to solve their tech shortage crisis. Yet they
absolutely refuse to evaluate their own hiring processes. And a huge chunk of
engineers just eat up all this dogma about how hiring is hard and these
processes are necessary, and don't think for one second that all these
processes are totally designed to please the employer at the expense of
engineers being treated well. In a few years, there will be another recession
and we're all going to be "rightsized" away, so, demand to get treated well
while you still can.

~~~
napoleond
_> I'm convinced one huge reason all this happens is to discourage job-
hopping, because in this market, liquidity would probably help salaries move
up faster._

Walk yourself through that for a second. You're saying a company that's hiring
would intentionally make the process difficult for applicants in order to
_help their competitors retain their employees_? That doesn't make sense. Even
if they conspired with said competitors for this purpose, they would simply be
ceding the hiring advantage to anyone who wasn't conspiring with them.

Occam's razor suggests a simpler answer: it turns out that identifying good
software workers is a hard problem.

~~~
ditonal
Do you realize that there was a class-action lawsuit where the "competitors"
Google, Apple, Adobe etc were working together to not poach each other's
employees? You're acting like I'm a conspiracy nutjob when there were emails
from Eric Schmidt and Steve Jobs where they admitted they were illegally
colluding.

Do you realize that many "competing" startups have the same VC firms investing
in them who don't want bidding wars between their engineers?

They want new talent but they don't want their current talent bouncing around
for more money.

~~~
napoleond
_> Do you realize[...]_

Yes.

 _> They want new talent but they don't want their current talent bouncing
around for more money._

Of course everyone wants to retain employees, and I'm sure that many will
stoop to less-than-ethical means to do so. I never suggested that the
possibility of a conspiracy between companies was crazy, just pointed out why
it's ineffectual. Google and Apple can maybe make it work, because there's no
substitute for those names on a resume. But once you start going down the
brand name ladder a bit, it is (a) infeasible for the large number of
companies to conspire together effectively and (b) far more likely that some
companies will refuse to conspire, and those companies will get the upper hand
in the hiring market.

------
pandaman
Almost everywhere I went on interviews the questions were about the very basic
things. The things you'd knew if you did what you claimed you did on your
resume. E.g. if you are a 3d graphics programmer with 10 years of experience
you know what cross product is. If you had been shipping multiple 1M+ LOC
projects in C++ you know what the word "virtual" mean and other popular syntax
as well.

The places that asked me things I did not know probably had been looking for
somebody with different experience than I had so it's fine with me too, even
though it would be better if they evaluated my experience from my resume.

So I don't see the author's problem. If the questions on the interview are too
hard - you are probably applying at above your level or in the wrong field. I
will most definitely not spend my time working on some programming test or do
some side project for "show-and-tell" to please you.

Heck, he is appealing to other professions allegedly not doing interviews, yet
it's even harder to imagine them doing things he suggests. What kind of side
project a doctor will have? A civil engineer? A manager? A pilot? A chef? A
lawyer?

------
kelukelugames
Maybe I'd have sideprojects if I didn't use all of my spare times studying for
interviews.

------
TACIXAT
I get to interview candidate team members for the malware research team I'm
on. We write a lot of code, but we're not really software engineers.
Accordingly, I only ask people very basic things in an interview to see if
they can use the tools they list on their resume. Often times it's something
that can be handled with a single list comprehension (though no one every does
that). It is astounding how few people can do what their resumes say.

I'm sure it's stressful if you don't know what you say you do. Then again, if
you list 6 programming languages on your resume and can't xor decode a string
in the one I let you choose from that list, you deserve to sweat a little.

------
suchire
IMO, having interviewed hundreds of candidates, whiteboard coding selects
works very well for hiring a very specific type of fresh-out-of-CS-undergrad
student, but it breaks down in almost every other scenario.

For more experienced candidates, we've swapped out most of the whiteboard
questions for several other types of interviews, including in-depth technical
discussion of one of their past projects, a discussion of how to architect a
non-trivial system, and a more soft-skills interview on how they work with
others, break down projects, and so on. This gives us a much rounder picture
of a candidate, and allows candidates who are very good at something to shine
a bit more.

------
kamaal
The best interview I've had is where I was given a small project but with a
tough deadline and I was asked to come back with a working code/demonstration.
The remaining interview was a code review session, and we talked about design,
choice of tools, libraries and other general stuff.

I was stunned by the end of the interview because they got out everything one
would want to know about how good a person is at their everyday job.

The worst interview I face is when people try to test my algorithm skills,
that generally a test of how much I could memorize from careercup.com

------
yeureka
I conducted quite a few interviews at my previous job and my solution for the
whiteboard anxiety problem is to give a set of coding exercises/questions on a
piece of paper to the candidate and give him/her as much time as necessary to
answer these in private in a separate office.

When the candidate is ready he/she announces it and we review the solutions
together.

It is not perfect, but I find it much less taxing than being asked to solve a
technical problem in a white board in front of an interviewer.

------
fit2rule
I just went through one of these terrible interviews at a local game company,
and I have to say I agree with this article 100%. I can code, have been doing
it for 30 years, and I can handle myself in challenging technical situations -
and I have the resume to prove it.

But put me in front of 3 guys I've never met whose purpose for being in the
meeting is to expose every single one of my weaknesses, push me up in front of
a whiteboard and ask me to explain the best way to solve a maze puzzle, and
I'm going to freeze up and fail the interview. The reason is, this is simply
not how I work. I work by sitting at my desk, thinking about the problem
presented, and writing code to have the computer do all the work. It may very
well be organizationally convenient to have these psychological lynchings
occurring, but I highly doubt it gets the absolutely best candidate in the
door.

I look forward to new solutions to the problems of finding qualified people;
in my case, I feel I unjustly failed an interview at a company I could have
been quite productive. Such is life in the modern software world, alas ..

------
amarjeet
I follow this process for most of tech/product/analyst type roles.
Specifically, in the case of tech, to me, it really depends upon the level of
coding exposure a candidate has. If he/she is a fresher then I would prefer
some real-time exercise (white-board/phone/skype exercises) - it helps me get
the idea of a person's thought process. I don't look for right answers in that
exercise. If the candidate has some code exposure, I prefer to dive deep into
his previous work and figure out how he executed the projects. In both the
cases I am trying to asses a candidate's approach to execute something. I also
try to figure out the candidate's ability to pickup new things and start
executing them quickly. This is measure of a candidate's potential. I assign
similar weightage to 'Potential' as to 'Past Work History'.

------
logn
I think people overstate how much interviews are the problem. Managers/HR are
the problem. In most companies the team manager has little to do with finding
and initially screening candidates and they (or HR) are too afraid of firing
people. I think managers alone should be handling this.

I also don't think hiring by committee decision makes much sense. It reduces
the one-on-one evaluation time by the hiring manager. If you have 6 people
doing 30-minute interviews of a candidate, everyone gets a few good questions
in then the decision comes down to a consensus of gut feelings. Plus the
manager can deflect blame for bad hires.

But changing this require re-organizing people and job duties in a company and
they'd rather just look for the next hot interviewing trend. (Edit) But as far
a trends go, looking at real work is at least better than contrived exercises.

------
tezka
I agree that technical interviews are broken, but not in the way that the
author describes. I think the interviews should retain the algorithmic
questions, but do away with language gotchas and design patterns etc. More
emphasis should be added on the following topics: distributed systems issues,
traits of modern hardware (cache efficiency, concurrency issues). There needs
to be a part about mathematical competency introduced as well. Knowledge of
basic probability, statistics and linear algebra should become part and parcel
of the engineering interview process.

------
monksy
> This is why personal references and recommendations remain everyone’s
> favorite hiring technique… > …which in turn is a major reason why the tech
> industry’s diversity numbers are so disastrous.

I'm not buying that argument. I would however agree that recommendations and
references create a lack of diverse ideas/approaches. Lack of racial/gender
diversity: That would indicate that the people in the minority just aren't
reaching out or participating in the community.

~~~
skybrian
You make it sound so easy. Perhaps they've tried, but it's harder for them to
connect? Without actual data, I think I'll go with that; it fits better with
what people are saying.

~~~
monksy
IMO: Its more of people being shy and/or drop the expectation that they'll
accepted by the community in the first go.

They should come out to the non-discriminatory user groups. (I.e. groups that
don't ask for a particular gender i.e. xNUG/xJUG etc)

I've found, in others and myself before, is that I've held myself back from
going to meetups because I didn't feel immediately welcome the first go, or
that I didn't think I'd know people. However, you can't expect for everyone to
welcome you with open arms in the first go. It's easier to blame the issue on
racism/sexism, than to admit that you may just be shy. (That being said, I
attempt to introduce new people in my meetups to the group and others)

~~~
skybrian
It could also be distance or lack of time or avoiding creepers or having more
fun things to do or not knowing about the event in the first place or some
hidden factor we didn't think of. The point is that there's a filter and its
effects aren't necessarily known to us, so it's hard to deduce what's wrong
when some people don't show up.

It seems like figuring out problems like this is what marketing is all about.

~~~
monksy
> It could also be distance or lack of time or avoiding creepers or having
> more fun things to do or not knowing about the event in the first place or
> some hidden factor we didn't think of. The point is that there's a filter
> and its effects aren't necessarily known to us, so it's hard to deduce
> what's wrong when some people don't show up.

Except for the creepers issue (which only can be solved by bringing it up to
the organizers), thats on the fault of the attendee. You can't make
connections if you're not there. Don't blame the organizers for that.

------
socrates2016
An issue with technical interviews is the shortness of the measurement. Real-
life programming occurs over days, weeks, and months. You think about a
problem and come up with progressively better solutions while thinking through
and understanding tradeoffs. Its more a marathon than a sprint. Technical
interviews are like sprints. So its as if you hire runners based on how well
they can sprint but then race them in marathons.

------
moey
I think the best approach would be options for different people to choose.

Think you can pass a hardcore whiteboard interview? Go for it.

Would you rather spend a day and solve a real problem we have? Lets see what
you got.

Think you have the attitude, just not the knowledge to match our exceptions
yet? Do you have time for an internship? Ok cool. Lets play.

One size fits all is a bad idea for interviews. Just as how teachers test
their students.

------
dba7dba
Why not require programmers to submit samples of work (most likely personal,
side projects due to legal issues) done over a period of time, like graphic
artists do when they apply for positions? I would think that would be a much
more correct assessment of the programmer.

Interviews should really be limited to check personality, and not much more
imo.

~~~
rifung
This seems like what the article suggested. The issue then just becomes that
not everyone has the time to do side projects and some of us are lucky enough
that we really enjoy what we do for our job, so even when we have spare time
we still work on the company project.

~~~
dba7dba
While I agree with you, I've learned no company will stay as loyal to you as
you are to them. You always need to plan ahead to fend for yourself. And
having SOME kind of side project (no matter how small) is always advised,
imho, for future employment.

------
apinstein
I have found the most reliable measure for a candidate is looking at their
public interactions/contributions. Seeing what projects they choose to work on
in their spare time, how they speak at user groups, how they interact on an
"issue" online. Blog posts are also great. For me, ability to communicate is
such a huge part of being a successful developer, that being able to see how
they communicate is always a very strong signal of intelligence, mastery, and
interpersonal style.

So this works great for people that partake in such things, but there are
clear still many great developers (perhaps the vast majority) that don't have
this type of extensive public profile. These are the applicants that I fear
false-negatives for the most, and the candidates for which I am not confident
of my interview techniques.

Could it be that the proper way to technically interview such a candidate is
to offer them a "fellowship" to work on an open-source project? Point them at
any project (used by the company or a favorite project of their own) that
they'd like to contribute to, and offer them a small stipend and enough time
to make a legitimate contribution to open source? And use that as the
technical portion of the interview?

~~~
dgemm
I think the challenge here will be in the overhead of finding and evaluating
suitable projects, given that you can use each one only once.

------
jdmichal
Is it maybe time to start allowing software professionals to have a portfolio?
I would guess that companies are the main driving force behind not having
portfolios that can be shown, but at the same time they are thereby hurting
themselves because they can't properly evaluate employees.

------
BurningFrog
This article seems to be written inside a different bubble than mine. Then
author lives where all technical interviews are conducted in a similar way,
and thinks that is the universal truth everywhere.

In my bubble things aren't perfect, but they're quite different.

------
cpprototypes
I think this would be a fair and effective interview system:

1) College graduate: Keep the traditional algorithm programming tech
interview. But allow the use of a laptop (no whiteboard)

2) 2 - 7 years engineer: Require a github side project or give a home coding
test. Interview will focus on discussing the project implementation. Also ask
them to add a simple feature.

3) 7 - 15 years engineer: Ask them to come to the office and show them some
bad code from the company source control. Ask them to explain why it's bad and
to refactor it to something better.

4) Express route: Instant offer, no interview. Only "interview" is by a VP and
it's more about trying to convince the candidate to join the company instead
of the other way around.

The first 3 scale according to typical life situation and industry experience.
The amount of free tend tends to decrease from college graduation to 30's and
beyond. For example, a college graduate has lots of free time. Someone in
their 20's has less, but still a significant amount. 30's and beyond tend to
have little free time due to family responsibilities.

But the interview styles also match what they should know by that point. A
college graduate doesn't know anything about real industry coding so the
typical algorithm coding interview is ok. Someone with 2 - 7 years experience
should be good at writing lots of code. But they don't yet have the experience
to know that sometimes deleting code is better than adding more. They also
don't have enough experience yet to read code well and refactor, their "code
smell" sense is not yet developed and they think adding more code is the
solution to everything. An industry veteran of 7 - 15 years should be able to
read code well, spot all the issues, and be able to refactor into something
better. These skills can only be gained after years of experience.

So the 3 interview styles scale well according to the free time they are
likely to have and testing whether they've really grown as an engineer. The
last one, the "express route" is reserved only by referral from the company's
best engineers. For example, if the company has this tech ladder:

junior engineer, engineer I, engineer II, senior engineer, senior engineer II,
principal engineer:

Then only senior engineer II or above AND being at the company for at least 3
years would be allowed to make ONE express route referral per year. The reason
is that an engineer has to be in the industry for a while to meet other good
engineers. And 3 years is enough to learn the company culture. This express
route system effectively would give a competitive advantage over other
companies. For example, imagine a very good engineer who is already employed.
They are busy and don't want to go through any kind of interview process. But
if they are given an instant offer, that greatly reduces the barriers for them
and they are much more likely to seriously consider switching to the company.
The main idea here is that good engineers have a lot of options but not much
free time to interview, your good engineers should know a few in other
companies, so the express route is a way to get more good engineers who would
usually not consider moving because they don't want to spend time
interviewing.

~~~
ams6110
Are you actually involved in screening candidates and making hiring decisions?

------
crdb
I've seen two kinds of motivations: that for career climbing, and that of an
engineer who enjoys making things. I personally prefer my engineers to have
the second.

I tried hiring the normal way (CV, white board) and had one candidate worth
speaking to in over 200 applicants.

We decided to try something else. We stopped reading any cover letter or CV
and wrote that in the job ad. We asked candidates to build a simplified
version of what the job was about, taking about 4 hours, then to send us the
code which we'd discuss with them over the phone at their convenience. Anybody
whose code passed would be hired, and if people couldn't be bothered because
they were already famous they could send us their libraries. Nobody took the
Starcraft best of 7 option :(

Why we did this:

\- having to write code eliminates those who can't, or who can't be bothered;
not having complex unrelated hoops to jump through eliminated those who are
good at career climbing;

\- we replaced the hours candidates would normally spend dealing with HR or
travelling to a site for something ("cultural fit") that overselects the
polished anyway, with something people we want would enjoy doing (solving a
problem, writing code);

\- no pressure, access to your "external brain" (Google, your own libraries)
unlike with a white board; allows for the anxious to pass, and those who
outsource a lot of their knowledge to the machine (like me);

\- you can tell a lot about how people will approach their job from the way
they approach the small version of it; documentation, structure, abstraction
levels, choice of libraries, and if you're wrong about the candidate's reasons
you can always clear it up in the chat;

\- it's fair because everybody has to tackle the same problem;

\- no discrimination since it's all ability based: I took to Skype-text-
chatting with candidates instead of voice calls mostly because I don't like
talking on the phone. It was only when she sent us her passport scan for the
contract that we realised one of our candidates was female. Another team
member didn't have a CV; it's because he was just finishing high school, as we
discovered when we talked to him, but his code was good enough, so he got the
offer and skipped university.

I've never had to put out another job ad (not for myself anyway) because I
have so many qualified candidates left over from that round, and from the team
network. Surprisingly, we didn't get spammed; the requirement to post code
seems to have been mostly understood and we only had a half dozen "dear
Sir/Madam"s.

So I disagree with you, but YMMV. We were hiring for a small, distributed team
solving relatively well known problems. Requirements are probably different at
Google or Facebook.

~~~
teen
I would never do a 4 hour project for an interview...

~~~
crdb
150 people did... and most had pretty good jobs already. 40 on the first day!

Think about it another way. How much time do you spend/waste when applying to
a standard corporate job?

I graduated at the height of the Lehman fallout. I really wanted to work in
finance. I applied to over 200 companies and did a seemingly endless string of
interviews. In one case, a fund interviewed me an incredible 17 times for an
analyst position (about half of which were "technical" i.e. "here's a bunch of
accounts, tell us what you think of the company"). In every occasion that led
to an offer, I spent way more than 4 hours travelling, talking with HR, doing
various rounds of interviewing with various team members, having "social"
coffees with other same-level folks... hell just the automated screening
tests, essays, forms for somewhere like Goldman Sachs would take a good couple
of hours if you did them properly. I still know many experienced people for
whom a job search during an economic downturn means sitting for months at home
doing these stupid screening tests and perfecting their cover letter (after
all, putting the wrong company name in your letter is enough to get you
rejected, because of course you really, really want to work at this particular
bank and none of the competitors even though the reputation, work and
compensation are identical).

I think a lot of more experienced people realised the trade-off and thought
this was quite a positive signal on our part - at least based on the number of
people who were willing to quit incredibly well paid jobs for our pretty
average Asian retailer with no equity and a third of the pay. I had one
prominent member of the open source community slam us publicly for "wasting
people's time with free work" whilst one of his work colleagues was gleefully
submitting his solution to the task...

------
sytelus
The guy who wrote this is not even a real professional software developer[1]
but it seems that he has strong opinions on technical interviews anyway. In
fact I looked up few people he mentioned in article and none of them seem to
be real full time software _developers_ either. Nevertheless whoever has
microphone gets to get heard, I guess.

I have seen a lot of PMish/weak programmers make cries about technical
interviews. Yes, whiteboarding interviews are not perfect and false negatives
rate is typically high but the companies who want nothing but the best would
care less about false negatives and rather more about false positives. Anyone
who has worked long enough in software engineering _professionally_ knows the
enormous cost of false positives [2]. These sort of companies also typically
care less about which language and frameworks candidate knows at the moment.
They rather put huge weight on evidence whether a candidate can analyze a
challenging problem, apply computer science and obtain a solution. Why?
Because these companies are actually working on problems that _does_ requires
deep hard computer science.

Most - perhaps 80% - of software development houses are not like that.
Programmers in those companies have never bothered to think about collisions
in hash tables and they often wonder why anyone would care when libraries take
care of everything. They never need to create data structure for trees or
graph or even linked lists because simple things have always got job done.
They wonder why they are being asked questions on those "arcane" things when
nobody really uses computer science stuff that they were taught at school.

That's the culture problem. Those 80% of companies tend to copy interview
process at other 20% even though it is meaningless for what they do.
Interviewers typically chose their questions from algorithms text books or
even internet searches. Then they get stuck on it for a decade and proudly
mention it as their "favorite question". That's completely wrong. I almost
always tend to ask CS questions that I myself needed to solve to get my job
done. I typically retire my questions in month because I probably would have
found another problem where applying right algorithms and data structures was
the key. When I ask these questions to candidates, (1) it's rare chance that
it would have been covered in books like Cracking the Coding Interview (2) no
risk because candidate decides to do interview brain dump (3) I know candidate
is smarter than me if s/he can solve within hour under pressure (4) If they
fail miserably I know this guy could have screwed up our project just last
month had he been hired.

Bottom line: Ask questions from your actual day job. It takes a lot of skill
and effort to abstract away other complexity and form a good interview
questions and that's why being an interviewer is hard. Whiteboarding _works_
if you do that. It doesn't work if you are trying to copy companies which are
doing different kind of work than you do. It most certainly doesn't work if
you never actually needed to solve the problem you are asking to get your
actual job done.

1\. [http://rezendi.com/aboutauthor.htm](http://rezendi.com/aboutauthor.htm)

2\.
[http://www.joelonsoftware.com/items/2007/06/05.html](http://www.joelonsoftware.com/items/2007/06/05.html)

------
yason
The more I interview the less I tend to give weight to the technical part and
the more I focus on evaluating how well I get along with the person, how
easily and naturally we can _discuss_ programming topics, and do we happen to
converge on building a rapport.

People can learn technical stuff and people can grow professionally but the
chemistry is harder to fix. And, unlike some programmers might see it, even
code is communication and, a bit surprisingly, is subject to chemistry. You
can talk to some people via code. Not all people.

I consider the technical part merely as a way to filter out people who seem to
be missing something blatantly obvious. I'm fully aware that my system will
sometimes generate false negatives but there's no such thing as a fully
objective interview.

I have a few technical things or topics that I usually ask about and I expect
every candidate to know at least a few of them but certainly not all. I
generally go forward topic by topic until the candidate has at least N things
s/he knows about. Of those that the candidate does know about, I ask more and
more details until I run out of questions or the candidate runs out of
answers. If I'm getting strong signals early that the candidate is likely to
be ok, I'll just try to finish up the technical part quickly.

Then I proceed to the most important and revealing part which seems to be
asking about a project that the candidate is really proud of, work or hobby.
In the best case I get a lecture down to details on something the candidate
built that s/he's still excited to explain to someone, on the coolest things
s/he could build into that project. A good programmer pours so much passion
into some, possibly minuscule, part of what s/he's building that you surely
should be able to ooze some of that out back.

Another important part is asking about hobby projects: if one particular
candidate doesn't program on his/her spare time, I'll probably go with any
other candidate who does unless s/he's exceptionally strong otherwise.

I try to remove pressure from the interview by acknowledging that given some
rough preliminary filtering, I could go wrong either way. Maybe I reject a
good candidate because I don't know as much myself. Maybe I accept someone
who's really nice and who seems to know about things, passing my filters, but
turns out s/he just can't produce much code hands down. All that will happen
some day.

The trial period is there so that both parties can revert an obviously wrong
decision. We haven't needed that yet but I greatly prefer to postpone some
responsibility for later and relaxing the actual interviews as much as
possible. People don't like to be grilled and I don't like to grill people,
not only because it's very consuming but because it doesn't seem to be a
particularly effective indicator.

There will never be a silver bullet to interviewing because there will never
be a silver bullet to meeting people the right way, but if I were to suggest
one thing it might be to listen more and ask fewer questions. By listening, I
don't mean letting a babbling candidate take over the interview. By listening,
I mean to figure out who is this person, where s/he's coming from and where
s/he seems to be going.

Being the fastest and most accurate shooter isn't much of a value unless you
know what you want to shoot, what needs to be shot, and why.

~~~
jqm
Well... that's certainly one approach. Hire people because you like them.
Though it probably isn't the best approach for getting results.

