

Ask HN: coding challenges as part of interviews: yes or no? - rtc

just curious what the group thinks of coding quizzes / challenges that are often part of job interviews for programmers.  how predictive are they of a coder's future performance?  are they appropriate only for entry level positions (when someone has less industry experience), or are they suitable for any level?  are they more likely to pique your interest or to come across as cliche?
======
raganwald
My experience:

Successfully pasing a code challenge is not a good predictor of future
success. However, failing to pass a code challenge is a very good predictor of
future failure. Furthermore, making the test harder does not make it predict
future success well.

Therefore, I belong to the "FizzBuzz" school: Make the test ridiculously
simple. It should take no more than fifteen minutes to complete. If the person
gets it obviously right, throw the result away and move on to more important
quetsions. Do not try to deduce a lot of malarky about their programming style
or ability from an obviously contrived problem.

For example, writing:

a = 5; b = 10

Does not mean they don't know Ruby well, and writing:

a, b = 5, 10

Does not mean they are a Ruby expert. And writing:

fsquare = lambda { |x| x * x }

Does not mean they are a closet Bipolar Lisp Programmer. If the program works,
it's a pass, move on. If there is a syntax error or some such, who cares, move
on.

But if they struggle... You need to investigate the reason for their
difficulty with a simple problem.

------
jamess
Trick questions, or questions with some kind of gottcha have no place in
interviews, However, simple, straightforward programming exercises are
essential. I've seen people who claim to have X years of industry experience
who can talk a good game, but literally can't write a switch statement without
reference to the internet.

I also like to ask people to peer review some code with both obvious and more
subtle errors in it. That's a pretty important skill to me too. Anyway, ask
them to code some fairly simple algorithm and then critique their own work. If
there are several ways to solve the same problem, some more efficient than
others, all the better.

~~~
ericb
Hmm... I know > 10 languages. I'm not sure I _could_ remember the exact syntax
in several of those languages on the spot. Is it case, is it switch, etc. If
you do this, your test is on rote memorization.

Give them a novel problem and have them solve it with a real computer and in
tact syntax lookup. Then see what and how they did it and go over their
reasoning. I'll hire the guy with the wisdom over the guy who can spit out
syntax trivia flawlessly. Those are different skill-sets that do not predict
each other.

~~~
astrec
Agreed. My first job they gave me a copy of Deitel & Deitel's "C How to
program" along with the test. You actually lost points if you didn't open it!

~~~
ntoshev
You know the syntax of switch in the language you have been using in the last
three months. If you don't, something's wrong!

~~~
raganwald
I never use Ruby's case or switch or whatever-it's-called.

I could make up some mumbo-jumbo about case being a code smell and that it
should actually be refactored to a method dispatch over polymorphic objects,
but the truth is that I have no idea why I don't write case statements, I just
don't.

~~~
hugh
Add me to the "never uses switch statements" list. I've never seen the point
in learning the syntax in any of the languages I know, since a bunch of else
ifs (or elseifs, or elsifs, or elifs, depending on the language) does the same
thing. If I waste a few keystrokes, but save a few brain cells, I think it's a
worthwhile bargain.

Of course if I had to write that kind of construction often I'd probably learn
the syntax of the case statement, but in the kind of work I do it hardly ever
actually comes up.

~~~
etal
It's really just a performance optimization for C, in situations where the
switching value can be hashed. In languages that make method dispatch easy,
and the overhead of a function call is not something you're terribly concerned
about, it makes sense to just roll your own as needed -- that's why Python
never bothered with a switch statement.

------
gcv
A trivial coding problem is essential. I've seen people with 10 years of
experience in XML#Java.NETEnterprise2012 who don't understand conditionals,
loops, or boolean expressions.

For phone screens, I find Steve Yegge's guide a useful starting point:
[http://steve.yegge.googlepages.com/five-essential-phone-
scre...](http://steve.yegge.googlepages.com/five-essential-phone-screen-
questions)

I'm not a fan of asking candidates to reverse a string or to implement atoi or
a binary search, because those have corner cases which, under interview
pressure, might be too much to ask for. I just ask for an implementation of a
gcd (greatest common divisor) function, in any language. I always explain the
problem in detail. Only one guy (a PhD in physics) ever gave me Euclid's
algorithm (section 1.1 in Knuth's AOCP Volume 1), so I don't expect it. I do
expect a trivial loop, and most candidates take 15-20 minutes to come up with
it, and need lots of hints.

As far as I'm concerned, anyone who can't come up with a trivial loop (or
recursive or tail recursive function) for something like this shouldn't be
programming for a living. Maybe "developing enterprise solutions," but never
programming.

~~~
raganwald
"As far as I'm concerned, anyone who can't come up with a trivial loop (or
recursive or tail recursive function) for something like this shouldn't be
programming for a living. Maybe "developing enterprise solutions," but never
programming."

Priceless, If you ever put this on a cafepress mug or tee shirt, you can count
on selling me a dozen or so as office gifts. Thanks for the laugh.

------
tonystubblebine
This reminds me of a group of main frame programmers I knew who had been
forced to become Perl web developers. About a month later they were asked to
hire another Perl programmer and one of their interview questions was what's
the difference between 'chomp' and 'chop' (chomp removes the final character
of a string if it's a newline, chop does it regardless).

They were shocked that the interviewee didn't know the answer because they
considered it basic knowledge, "it's in the first chapter for gosh sake." The
first chapter in this case covered differences between Perl 4 and Perl 5.

In Perl 5, nobody uses 'chop', they all use 'chomp.' Many experienced Perl
programmers might even have forgotten that 'chop' existed.

And that's my major problem with gotcha questions, they tend to be pretty far
from measuring if a person can actually write productive code.

------
astrec
Programming challenges sort the wheat from the chaff - in particular those who
cannot code at all, but beyond that I've rarely found them predictive of
future performance. Much more important that you hire a good fit for your
company. An unhappy or toxic coder can bring a startup to its knees - hire
slow, fire fast and all that.

~~~
goofygrin
This is true. I aced the coding portion, but after 6 months, I was definitely
not a good fit in the culture of the last company I worked for.

------
icey
I like them as long as pseudo-code is acceptable. I don't really care how good
someone's memory is with regards to language syntax; I care about whether or
not they can solve problems and think them through.

------
cconstantine
/agree with most of the people here. Coding questions have a purpose.

You need to consider what you are trying to determine by asking any question.
Really, what is it about the candidate you want to know? After pinning down
precisely what you want to know then tailor a question to figure this out with
as little peripheral stuff as possible.

If you want to know if someone can generate some code in their preferred
language on the spot, fine. Pick a very simple problem, like 'Write a program
to display numbers in the fibinocci sequence', and explain the fib seq to them
if they don't know it. I wouldn't even count points off if they make any kind
of mistake that a compiler would catch as long as they know how to fix it.
Have them do it on the white board.

This kind of Q isn't designed to figure out how much they know of 10 different
languages. It's designed to see if they can put some code down in their
preferred language and have it do something useful.

The biggest constraint in an interview isn't the honesty of the candidate, or
even their knowledge; it's the N hour time limit. Your goal as the interviewer
is to squeeze as much information about who the candidate really is in the
time allowed.

------
sfpeter
I like to believe that I can get a pretty good understanding of someone's
technical abilities by asking thorough questions about his previous
experience, and maybe ask them to explain a couple of basic patterns or
programming concepts to me. I remember interviewing a senior java developer
who claimed 5+ years of experience and had no idea what a Singleton was. What
made it worse was that even after explaining it to her, she was unable to
figure out how to implement one. Developers asked the strangest questions when
interviewing other developers. I worked with a guy once who liked to ask his
interviewees what the decimal value of 0xFE is. One candidate didn't know the
answer so my coworker didn't think he could possibly be a good software
engineer. I admit that (having done assembly coding in school) it seems odd
that any software engineer wouldn't be able to figure out what FE is. But I
still thought it was a bad way of testing technical skills. Would you rule out
a candidate who doesn't know 0xFE ?

~~~
icky
> Would you rule out a candidate who doesn't know 0xFE ?

If they didn't know it after a moment's thought, I'd cut them some slack; If
they couldn't calculate it, with pencil and paper, in 10 minutes (and that's
giving them plenty of extra time for interview-nervousness), then I'd be
seriously concerned.

~~~
andrewcooke
i read through some answers here and i nodded in agreement with raganwald
saying that this only worked as a filter on failure.

but then i read this and i thought "would i really want to give a job to
someone who needs a pen and paper to subtract 1 from 255?"

i'm not sure what the lesson is here - perhaps that programming tests are
dangerous because it is very easy to over-interpret the results, even when you
start with the best of intentions?

[edited for grammar]

~~~
icky
No, I would be very liberal in accounting for interview-nervousness.

Now, if it appeared that the candidate had no idea _how_ to solve the problem,
that would be a different story.

------
jarnold
I think taking some time to sit down and _read_ some code is one of the best
and least used techniques during an interview. I think it's more of an
indicator for job performance than writing some green-field code. After all,
on the job, more time is spent reading and understanding code than writing
code with no preexisting context.

~~~
eru
You may use the solution your last candidate came up with as an example.

------
neovive
I think it depends on the position and responsibilities the person will have
once hired. If the programmer will be coding directly from specs without any
room for "interpretation" then a coding challenge may be very applicable. If
you are looking for a programmer that also understands your business and will
be required to make decisions and think on their feet, the value of a coding
challenge diminishes.

It also depends on the time frame of the position. If you are looking to hire
a programmer on a per project basis, then a solid understanding of the
development language is critical. However, if you are hiring an developer for
the long term, it's better to hire a smart and agile person that can
contribute to the overall business as well -- which is hard to measure from a
coding challenge.

------
sfpeter
I also worked for a company once who had several enthousiastic , seemingly
bright people with somewhat limited experience come in for interviews, but
they didn't make the cut because they didn't pass some coding challenges. The
guy who did get hired went to a good school and was a good coder, but was way
too arrogant, difficult to work with and repeatedly refused to do what his
manager asked him to work on. And how did that guy interview other candidates?
He screened them by their education, and when they would come in he would give
them one programming challenge without asking any other questions. Nobody
seemed to realize that there were big problems with the company's hiring
strategies and that developers themselves were largely to blame for that.

------
arupchak
I've always found that asking a reasonable numerical solving question where
you can easily figure out the answer in your head is a good starting point to
gauge programming/algorithm ability. If someone knows the solution they are
trying to get at, they can focus on the program itself, as opposed to worrying
about solving the problem and coming up with the algorithm. Ideally there are
a few ways to solve the problem, brute-force methods or more efficient ways.
It can be very difficult to test a programmer's ability in an interview, you
need to focus more on seeing how the candidate thinks, how they approach
problems, and whether they can admit when they do not know something.

------
jkkramer
I use a simple coding challenge as a weeder question. It requires basic use of
recursion (without explicitly saying so) -- no math, no big-O, no gotchas,
just writing a simple recursive function to process a small tree. It takes
decent coders a few minutes to complete. Less-than-decent coders really
struggle with it.

This is only a wheat/chaff separator, though. If possible, I like to see code
from actual projects they've worked on to get a feel for the bigger picture
(tip: open source your personal projects and obsess a bit over the code, so
you have something to show off). Plus there's all the other non-code aspects
to the interview.

------
jzell
If you have the sense from whiteboard discussions that the candidate
understands computer science (like gets recursion) but you want to make sure
they can code, why not choose a subset of a problem that you (your company)
needs to solve?

~~~
gaius
Good way to get some free consulting ;-)

~~~
scott_s
I don't think he means _open_ questions, but things related to the problem.

I interviewed once with a company that OCRed medical documents. One guy held
up a sheet of paper and asked "How much memory does this take up?" which was a
deliberately open-ended question. I was initially stumped, but then he started
feeding me domain knowledge and assumptions, such as common DPI values and
expected fidelity. Then I started coming up with estimates.

I don't think he expected me to have answer right off. I think he wanted to
see how much I could figure out on my own if given hints.

------
ig1
They're a necessity. For the simple reason that too many people lie on their
CV, no matter how good their CV looks you have no-idea how much of it is
accurate.

Obviously there are cases where you wouldn't use it, but generally they only
apply when you know someone's capable of coding anyway (i.e. due to an
indisputable reputation, experience working with them, etc.).

However as a predictive measure it's fairly limited in that it's a binary
indicator. If someone fails then there's a fair chance they're a terrible
developer, if someone passes there's a fair chance they're not a terrible
developer.

------
edw519
Necessary but not sufficient.

I always give a fairly simple problem, pencil and paper, a soda, and leave
them alone for 15 to 30 minutes. Before I leave the room, I make sure they
clearly understand what's expected.

What they do isn't important. _Our discussion_ about what they did is. I don't
care what syntax or mannerisms they use; I _do_ care if they understand what
was needed and what they did well enough to explain what they did and why they
did it.

Almost every time some good follow-up questions and segues ensue. What about
this? What about that? I've learned more about people in the discussion after
the problem than from anything else.

------
jaydub
If I were interviewing someone, I would be more likely to test someone's
general problem solving skills, questions that illustrate how a candidate
thinks and approaches a challenge. This of course becomes more subjective, but
can really help identify creative talent. Who knows, you might even learn
something too. This is probably a big part of the puzzle type questions that
companies like Microsoft and Google seem to favor.

------
aggieben
Agree with jamess, but additionally, I think interviewers that _don't_ ask
coding questions might be an indication of something negative at the company.
Basic coding ability has to be one of those checkbox questions that determines
your eligibility for the position, and if the interviewer doesn't ask anything
technocal, it makes me wonder what kind of dumbasses I might end up working
with.

~~~
dsplaisted
Indeed, whether candidates write code during their interview is one of the
items on the Joel Test
(<http://www.joelonsoftware.com/articles/fog0000000043.html>).

------
mlazur
Writing code is artwork. You can ask the artist about the specifics of his
tools, but you will never grasp his artistic ability until you see the
completed artwork. I just ask some general questions about the tools. The
tools for code writing are: starting, constants, variables, addressing,
comparing, branching and ending.

------
eru
From the employees point of view: I would have no problem to do a coding test
as part of an interview.

