
Interviews are Broken - emiller829
http://erniemiller.org/2013/09/19/interviews-are-broken/
======
Peroni
I've been in this game for a while now and the breakdown is pretty simple. If
you are an employer, I implore you to trust me on these two simple points:

1\. Finding decent people that are both technically competent and a good
culture fit is bloody tough and can take a lot longer than you'd ever expect.

2\. Interviewing/screening these people is simple. Don't make it more
difficult than it needs to be.

You've done the hard part and you're confident the person/people you've found
are worth at least a couple of hours of your time for an interview so why
waste that time by applying a boilerplate interview process that every other
company uses?

Give the candidate a realistic technical challenge to complete in a realistic
timeframe, in an environment that will be indicative of the environment they
would expect to work in should they succeed in getting the job.

If you're happy with the technical results, spend at least an hour with them
having an actual conversation. Don't sit there throwing stock interview
questions at them, don't try and nitpick on their CV. Just talk to them. Get a
feel for their personality, the things that motivate them, the things that
annoy them, the things that inspire them, etc.

~~~
Lost_BiomedE
I long for an interview like that. I first left the tech industry due to
having to deal with HR and the hiring process.

I became a chef, where the interview could be moved to having them watch me
make a dish or show knife skills. Then out of interest, I went through a
biomedical engineering degree and am back into the tech field.

Still, due to my negative experiences with the filtering process, I have
stayed in the realm of freelance and contracting. I would jump back into a
tech job if interviews, like the above, were more common.

Thanks.

~~~
mmcnickle
I've always thought the simple coding interview questions was like asking a
chef to prepare a very simple meal; they should be able to do it without any
problem. Though it's apparent that coders, like chefs, can choke on even these
simple tasks. You only need to watch the "skills tests" on "Masterchef: The
Professionals" to see a dozen professional cooks fail to fry schnitzel or
fillet a fish. The same chefs go on the create some fantastic dishes in less
stressful challenges.

~~~
nawitus
One additional difference is that interview coding questions don't have a
realistic working environment. Programmers utilize tools like search engines,
IDEs and documentation, but these may be not available in a coding interview
question, which might mandate a programmer to work on a whiteboard or using
unfamiliar software.

------
tokenadult
Peroni writes in his top-level comment, "Give the candidate a realistic
technical challenge to complete in a realistic timeframe, in an environment
that will be indicative of the environment they would expect to work in should
they succeed in getting the job." And of course that is saying "Give the
candidate a work-sample test." That's a very well validated hiring
procedure,[1] one that every company ought to use for essentially every job.
In most parts of the world, you can add incremental validity to the hiring
process by also testing the job candidate's general cognitive ability (a.k.a.
IQ). In the United States, you have to take careful legal steps to be able to
add a general cognitive ability test to your hiring process, but you would
have to take the SAME legal steps to make a diploma or degree a requirement
for hiring (a little known fact about the key Supreme Court case on the
issue). Anything else you do in hiring has less impact on gaining successful
workers than work-sample tests and general cognitive ability tests.

[1]
[https://news.ycombinator.com/item?id=5227923](https://news.ycombinator.com/item?id=5227923)
(this earlier Hacker News comment gives full references for the statements in
this comment)

~~~
davesims
The challenge here, in my experience on both sides of this, is that a
developer's deep domain knowledge of a large or large-ish app is essential to
a 'work-sample' environment. And that's virtually impossible to duplicate even
in the longest interview time frame.

Making decisions about managing technical debt, adding architecturally-
significant changes, balancing good OOP with responsiveness, knowing the
difference between future-proofing and conscientious coding -- all of those
are both crucial (in many cases the most crucial) to day-to-day work, and also
so highly context-specific that those decision-making traits are nearly
impossible to identify during a technical exercise.

So for coding challenges, that leaves short-term tactical/analytic/algorithmic
exercises, which in (anecdotally) 95% percent of cases cannot begin to
approach a 'work-sample' environment. I've yet to encounter a technical
challenge that would tell me much more about a candidate than basically how
fluent they are with their tools, how well they know syntax and some general
design principles, and what, for instance, their TDD (or lack of) workflow is
like. Probably some insight into line-level analytic and algorithmic ability.

All of that is helpful, but -- Trust Me Here!! -- can also be very deceiving.
The same coders that can knock those challenges out of the park can also be
highly-proficient _Debt Machines_ , all the more destructive because of their
special genius for cranking out architecturally suspect code at a breathtaking
rate.

To get into a real 'work-context' flow of a large app requires weeks,
sometimes months, and only _then_ can you get full perspective on how a given
coder is going to contribute to your team on an ongoing basis. To get a feel
for what that will look like in an interview, I've found I have to pretty much
rely on the candidate's past projects, and informal conversations around
larger architectural and OO principles.

~~~
tokenadult
I think your point is well made that what you can sample in a work-sample test
is not the full set of long-term skills that benefit a for-profit company.
That's why the predictive validity of work-sample tests is only about .50
across a wide range of industries. But the key point is that EVERY other
hiring procedure, except for general cognitive ability tests, has lower
validity, so a company is throwing away a lot of opportunity to hire good
workers if it doesn't use a combination of work-sample testing and cognitive
ability testing for all of its hiring. Your sound analysis can be turned
around to using interviews as a hiring procedure--which is much more
commonplace than using work-sample testing as a hiring procedure--to make the
correct point that an applicant who looks good in an interview may not be a
"team player" once hired. Any hiring procedure is a sample of applicant
behavior, not fully representative of how the worker will behave on the job
after being hired. But work-sample tests get much closer to what the worker
will do on the job long-term than any other procedure besides general
cognitive ability tests. Because work-sample tests and cognitive ability tests
each have incremental validity when added to the other, it's best to use both
in combination to get a hiring procedure with somewhat more than .50 validity
in finding good workers.

------
Aqueous
Yes. I have a new rule with recruiters. If you are trying to recruit me, make
me an offer. Otherwise, go away. You called me, not the other way around. I'm
not going to jump through a bunch of your hoops to prove to you for free that
I can do what you already know that I can do just by looking at my resume,
looking at some of my public projects, and asking me questions in an informal,
short interview. And I'm certainly not going to do that knowing that, even
after the end of an extremely long and taxing process, you can _still_ refuse
to offer me a job.

What I've done should be proof enough. I don't want to jump through a bunch of
arbitrary hoops so that I can get an incrementally higher paying job (maybe)
and devote my life to working on someone else's vision. Gee, can I?

I already have a job, and it might be different if I didn't. But maybe not.

~~~
Peroni
_What I 've done should be proof enough._

It really isn't. It may be indicative of your technical ability however your
past experience doesn't give me any indication if you'd be a good culture fit.

~~~
BigChiefSmokem
Isn't "cultural fit" another way to hide sexism, racism, and prejudice in the
hiring process? I've seen it happen and it's a very good excuse to give to HR.

Just like in social situations you can judge anyone pretty well in the first 5
minutes of a phone conversation, see if they will "click" with your team -
hell you can almost do that just by looking at their open source (how they
think/design). I don't need a high school coding exam and a 2 hour panel
interview to see if someone is a "cultural fit".

~~~
jasonlotito
> Isn't "cultural fit" another way to hide sexism, racism, and prejudice in
> the hiring process?

While that may be, it doesn't mean cultural fit is only used in those cases.

There is a vast difference between someone who cares about doing their job
well and someone who does their job. And 'cultural fit' will be used to
describe that. Are you the type of person to put in your own time to advance
your knowledge, or do you require the company to pay to keep you up to speed
on the latest advances. I've seen both types of people.

And the former is more valuable than the latter. And the former generally
won't want to work with the latter, either. Granted, the former will cost more
than the latter, as he brings more value to the table.

You can argue the merits of 'cultural fit', but it's not just a word used to
hide sexism, racism, and prejudice in the hiring process. And, personally, I
think it's important because I want to enjoy the people I work with.

~~~
JackMorgan
> There is a vast difference between someone who cares about doing their job
> well and someone who does their job.

I like this as a definition of cultural fit. I have worked a place where I
felt like the only guy who cared about his job, and it was suffocating,
similarly, I have seen the one guy who is just doing his job in a team of
those to love to do their job well, and he was like a ball and chain.

Perhaps we need to stop using "cultural fit", as a replacement for
"professionalism". When I think about "cultural fit", I think, "what do I
like, and does this person like it too?" When I ask myself and others, "is
this person a professional", I think, "do they exhibit: passion, discipline,
dedication, drive, and care in their work, skills, and interactions with
everyone?" I would rather tell HR, "not as professional about his craft as we
like to see", than a wishy-washy, "not a good cultural fit". The first sounds
like we are professionals who treat ourselves, our craft, and others with
respect, the second like a frat house blackballing a pledge because he doesn't
like the same beer we all do.

~~~
derefr
This definition (caring about doing your job) is a bit annoying, because it
means it's sort of impossible to work at any company that looks for this,
while also trying to start your _own_ company. All my _passion_ is reserved
for my start-up; you get my _professionalism_ , but nothing more. Why is that
not enough? (This is assuming salary and no equity.)

~~~
JackMorgan
In my opinion, someone who does their job well and professionally is just fine
to work on side-projects. Ideally, your productivity and focus stays the same
as they did before your side-project.

I truly do not think it makes sense to demand every person on a job is
passionate about the job they are doing. There have been many times I have had
to (even at a job I normally loved) do long, boring, and fruitless work. I did
not do them out of my passion for the "Mission to Save the World of Enterprise
Banking", but because I am a professional who does what needs to be done and
who does it to the best of my abilities.

When I said passion before, I was trying to express two things.

First, I was intending to express the attitudes that cause someone to stay
abreast of their field, keeps their own saws sharp, (and ideally helps
teach/inspire others to do the same), comes in and works hard, and pushes
themselves and those around them to be better. Perhaps the best word for that
is just professionalism.

The second concept is that I would rather be around someone who, while acting
like a consummate professional, also enjoys at least some part of it. I love
the fact that software development has more than I could probably ever learn,
and it is a worthy challenge to seek to master its different forms. I have
worked with several professionals who really do enjoy parts of the craft that
I only barely tolerate, and I enjoy that about them immensely. They open to me
a new worlds of appreciation for the craft. Just recently, I had to work on a
few hundred legacy bugs for a few weeks in a row. They were unexpected, not
caused by me, and deeply tedious. Some took hours to locate the cause. One of
my coworkers really came alive doing that. He just loved the hunt and chase,
like a detective working out a mystery. While I was mentally moaning and
groaning after week two, he was just getting started. We talked about it for a
few minutes one day, and I decided to try and see things his way. While it
didn't cause me to love bugs like a fine wine, I was able to bring my attitude
around from a silent scream up to mildly interested. Maybe the best word for
that is having a good attitude? Enjoys some parts more than others, and keeps
quiet when they don't enjoy it? I am not sure, but that whole concept I find
to be important. The person who reads books about development and enjoys
reading them is going to bring a wonderful attitude to the team, and can
motivate others to pursue their own interests.

------
msluyter
I think there's a lot of general recognition that the interview process is
suboptimal, but it'll probably be slow to change at larger companies mostly
out of inertia. Part of the problem is that developers tend to do most of the
interviewing/phone-screening, and a lot of us don't like it and/or aren't very
good at it. Thus, it's unlikely that we'll be willing to put much thought into
improving the process (note, I have _never_ had a performance goal related to
interviewing). Usually, we just go with what we know, which is the standard
white-boarding style interview.

That, and I wonder if there's a certain fraternity-like hazing element. One of
our HR people recently brought up the possibility of removing one of our
hiring steps (a take home programming assignment), and I found myself
irrationally against the idea. I even joked that I wanted new candidates to
suffer just like I did. On reflection, that might not have been a joke... Not
particularly proud of it, but part of me resists changing the process because
_I_ did really well at it.

------
balloot
This person sounds completely insufferable. He basically is declaring himself
too good for interviews (except for C-level execs, whom he apparently
tolerates). Heaven forbid you don't just assume Ernie Miller is god's gift to
your job position, because he is Ernie Miller and he is a special unique
snowflake who is not to be insulted with petty questions pertaining to his
ability to do the job.

Anyway, good luck with that.

~~~
emiller829
Did you actually read the article, or just the big, bold letters? The point
wasn't that I am special. It's that each person is an individual, and a
company that doesn't recognize that situations differ and make reasonable
accommodations is unlikely to be a company at which I (or many like me, if the
comments are any indicator) would enjoy working.

~~~
philwelch
Because nothing makes a hiring process more fair and accurate than changing
the rules for each candidate.

~~~
nopal
I'm not sure that same == fair.

If a worker is going to work remotely, is it a better evaluation of her skill
to have her do her coding test remotely or have her do it in the office?
Likewise, is it fair to ask a potential remote worker to take on an endeavor
that's much more cumbersome than it would be for an in-office worker.

The other part of the argument is that the hiring company may be affecting its
ability to evaluate a candidate by subjecting him to a situation that's pretty
different from the job's reality. Ostensibly, the point of having someone work
out of the office is to see how well he'll do if he gets the job, but in the
case of a remote worker, this may not be a good indicator.

I think all of this is highly debatable, and there's probably no right answer,
but there is lots of food for thought!

------
nissimk
It's interesting to note how many people that actually work at a place had to
jump through all of the pre job offer hoops. I think you'll usually find a lot
of people who were able to avoid much of that stuff because:

1) they were hired before the company started doing it

2) they new someone going in so they got hired without as much rigmarole.

3) they started as a contractor/freelancer and were converted to employee
without interviewing

4) they came to the company through an acquisition

It seems like if you're not good at these annoying pre-offer activities that
you're better off taking one of the above routes into a new job rather than
trying to play these games.

[edit formatting]

~~~
001sky
This is a good point, and sort of supports the Author's thesis. Perhaps better
than the submitted Article. The way to test if interviews are 'fit for
purpose' is to see how effective they are in real use. If everybody is going
around the formal process, by hiring friends/previous project-teamates/warm-
referrals only...then that is pretty important observational data.

------
gedrap
I think that we can simply say "no one has been fired for traditional
interview". Yes, it is broken, ineffective, interviewees mostly hate it, some
egos get hurt while solving fizzbuzz on the whiteboard, etc. But for an
interviewer, it's a safe choice. It's something he has done hundreds of times,
something the interviewee expects. He might miss some really good potential
employees but at the end of the day, everyone will be ok. Trying something
totally new requires some courage and stepping out of comfort zone, probably a
lot, and that explains it.

And if we are talking about major corps, it would require lots of investment
in terms of internal training, would take a while, in other words - it's
expensive. And they still get a lot of top quality candidates, so why bother?

~~~
JackMorgan
>Trying something totally new requires some courage and stepping out of
comfort zone, probably a lot, and that explains it

I think you are on to something here. People want to do the safe, easy, and
known thing. I cannot imagine most people, when finding out a poor employee
was hired would blame the interviewer or the interview system. They assume
that those are fine, because "Joel, Yegge, Google... everybody does them" and
so they become a mental blind spot. The focus falls on the bad employee, "wow,
he seemed so good, he must be a con artist or something".

>And they still get a lot of top quality candidates, so why bother?

This, I do not believe. By definition, most developers are not "top quality",
rather just fit in just well enough to stay out of the "fire zone". From what
I have seen, unless by "top quality" you mean "best development value for your
buck", most developers at most shops are far below "top quality". Most are
either overly conservative or overly aggressive, and have skills just barely
good enough to do their job, which is good, because if they were truly the
"top quality" they would tire quickly of your work and leave. The only way
someone gets to be even close to "top quality" is by constantly doing new
things, constantly pushing themselves, constantly getting better. Those traits
_sound_ good till you consider what then will happen when they master the job
you want them to do and get bored. That is what I mean by "best development
value for your buck". Chances are, most development shops want most developers
to be just barely good enough to get the work done in an arbitrary time with
an arbitrary level of quality. Like the old joke goes, anyone coding
slower/poorer/less than me is "a lazy idiot!", and anyone coding faster/better
than me is "an architecture astronaut stuck up jerk!". Hiring becomes, "are
you within an acceptably narrow band of skill that will all share?"

So, companies get what they hire for. Chances are, they get mediocre
programmers, because that is what they actually _need_. They don't need
language-writing, development world colossi striding about their office,
designing new paradigms for their slightly altered CRUD app. Most companies
wouldn't know what to do with such developers if they had them. Heck, even
Google apparently didn't know what to do with Guido. Instead most companies
need someone happy sitting down to add a new field to this page, or make this
checkbox work on this popup, or fix this bug that only shows up in IE7.
Someone for whom that is just challenging enough to be interesting, but not
impossible. Apparently, programmer quizzo is the cheapest/safest way to
ascertain that.

------
pjmorris
"If a company isn't making efforts to recognize you as an individual during
the interview process, an accepted offer letter isn't going to promote you to
personhood."

This is a great point. I've noticed that culture tends to be pervasive. The
good experiences I've had with companies started from first contact. So have
the bad experiences.

~~~
bliti
Also applies as a consultant. Bad clients have noticeable red flags right from
the start.

------
leothekim
I'd say asking a candidate to implement levenshtein distance in an interview
(probably not the best question to ask I'd say) is more about creating a level
playing field for candidates who might not have "attended and spoken at
relevant conferences" or have "a bunch of apps doing mission-critical work in
production", but might be thoroughly productive and successful at XCorp
nonetheless.

In fact, a candidate might have experience in something completely different
but maybe just as impressive - publishing a paper on an optimization for an
algorithm, creating a programming language, submitting Linux kernel patches,
etc. How would one differentiate a candidate like that with someone with "a
list of public repositories on GitHub as long as your arm"?

~~~
emiller829
I wouldn't differentiate them.

In fact, that's kinda the point. Each candidate should be evaluated on the
unique experience they bring to the table. Trying to create the illusion of a
level playing field by standardizing on generic code challenges in interview
exercises is just that, an illusion.

------
beat
I have a simple rule of thumb when interviewing other people... I don't care
so much what you know how to do, as much as I care how you handle what you
_don 't_ know how to do. Technical skill can be discovered quickly enough in
the interview process. Ways of thinking, that takes work.

As for the original article... if a company puts you through a long interview
process, you should take advantage of it to learn about them as well, and see
if it's where you want to work. Don't just be a replaceable part.

------
codegeek
My personal opinion. If you are hiring for a fulltime role where you hope to
have that person on the team for a long time , then it is a heck of a lot more
than just technical ability that matters. I learned this from an ex-boss who
taught me the value of skills that matter in the long run. Hint: technical
skill is just one aspect and not enough. It matters what the attitude is, what
encourages/discourages them etc. Why do these matter ? Because we are humans
and we use our experiences/emotions/values in practice a lot more than just
technical skills.

The game changes however if you need an expert
consultant/contractor/freelancer who can solve a _very specific_ problem with
a _very short_ period of time. For those, go hard on the _specific_ tech.
questions.

------
Apocryphon
You know, there's a lot of these interview articles. Where whiteboarding
doesn't work. Or whiteboarding works but filters out a vast majority of
applicants. Give homework problems instead. No, applicants can easily cheat or
resort to copy pasting code when given time away from the interviewer. The
future is GitHub repos to see candidates' prior work. No, open-source code is
not verifiable and easily falsifiable. What about determining if an applicant
has solid fundamentals, not overthinking it, and having him or her learn on
the job? Whatever happened to providing training? Is there really no room for
apprentices or novices in modern tech companies?

~~~
JackMorgan
Plenty of places do this, I have run training programs and helped with co-op
training for new developers now in three different companies, and I can say,
yes, there is room.

And, yet, our demand for developers with experience surpassing the average co-
op or college grad doesn't change. It can be hard to make much progress with a
team of mostly younger developers when every technical discussion someone is
asking, "what is FTP", or "I read that GET is the only safe way to pass
values", etc. Sometimes, it _really_ pays off to just hire someone who has
already figured out a certain level of the fundamentals and can hit the ground
running into more advanced walls like lack of business domain knowledge.

------
vytasgd
I've done very few interviews (on either side of the table) but I wish I would
come across one that had two main questions in the technical interview:

1\. Write some piece of code that you think is cool. It doesn't have to
originally be yours, and if it would take too long to recreate exactly, write
it out in pseudo code and explain how it works and why you chose it.

2\. (Slightly more common, and I've asked this before) We are having Problem
A, and it would be within the scope of your responsibilities. How would you
approach it?

the interviewer asks follow up questions to both and hopefully learns
something in the process.

~~~
grammaton
Ugh, coming up with something "cool" under time constraints, while someone is
sitting there and watching my every move, is not exactly a realistic gauge of
anything. One of the reasons the hiring process is so broken for so many is
that they fail to take in to account the nervousness and pressure the
candidate will invariably be feeling. Speaking as someone who has done _many_
interviews, questions like this always make me cringe. Sitting there in a
suit, sweating and angling for a job, isn't really the best time for me to
come up with something "cool."

~~~
vytasgd
i meant this more in the sense of "what's a piece of code you've seen(or
written) that made you excited and made you think 'wow that's really clever!'"
It shouldn't necessarily be something earth shattering, could be very simple,
just something you genuinely liked. I personally believe enthusiasm (including
intellectual curiosity) is the most important quality somebody can have in the
workplace.

If your problem with the question is trying to think of a response that the
interviewer would want to hear, then you aren't approaching the question
correctly and/or neither is the interviewer.

------
anmalhot
I am attending interviews these days and can echo a lot of what I read.
Sometimes the interviewers just keep forcing you to the answers they have or
they ask the same question to everyone - it's new to the first one but anyone
coming in later already has the perfect answer. Your past doesn't matter -
only the 30 min demo of you will sell. Only if in real world products got sold
by their demo and not final functionality.

------
rlu
I never got to interview for a startup. Is it common to have a 1:n interview?
At every single big company I've interviewed with, interviews were always 1:1.

He threw out 1:n interviews being bad as if it were the norm.

I also have no idea what he meant by this:

"All of this for a position that will involve working remotely (you are
working remotely, aren't you?)."

Stopped reading after that because it didn't sound like he and I were in the
same industry at all.

------
rrouse
I recently failed a tech interview. I was asked to pair program on a
calculator.

Something so simple became more difficult under the pressure of someone
watching you the entire time.

It's also nothing I've ever even thought of building nor something I would
ever be asked to build in a real work environment.

I have stellar reviews from my former employers. I know I don't suck, but hell
if that didn't shake me to the core.

Maybe I'm just venting.

~~~
gaius
Similar experience recently, writing some trivial awk while being watched, on
a keyboard with an unfamiliar mapping too. Probably took me 5x as long as
normal, which was unfortunate as this was a timed test!

------
tieTYT
> Once I started to apply this simple filter to the incoming opportunities,
> things got much easier

What filter? I didn't see him mention the filter he's using. Maybe he means he
rejects a company unless it has an adaptive interview. But, how can you do
that unless you take the interview?

------
richardlblair
Ive been doing interviews for a few months now, and I try to make it as casual
as possible. Why? Because I don't want to ask the candidate about who they
think they are, I want to know who they _actually_ are.

------
antidaily
__________ is broken.

------
ada1981
Hired.com can help this guy.

