
Ask HN: Am I being too hard on the developers I interview? - jlangenauer
I'm the CTO at a startup in London, so I'm spending a lot of my time interviewing developers as we try to grow our team. As part of that, I ask each of them to do what I consider to be a very simple programming exercise (See below.)<p>Yet, the results of this are disappointing. About 10% of candidates will be able to produce a solution in 10 minutes or less, another 30% of candidates will eventually produce something close to a solution in 30 minutes (though often with errors), and 60% of candidates will simply be unable to produce anything close to a solution. These are not junior or graduate developers, but senior, experienced programmers asking for £50k or £60k.<p>I try to make the candidates as comfortable as possible while doing this, but even so, the inability of 90% of the people I interview to develop a solution to a simple question is astonishing.<p>Am I being too hard on the candidates in asking them to do this?<p>-------------------------------------------------------------------<p>This is, verbatim, the problem sheet I show them:<p><i>Pascal’s Triangle</i><p>Each element in Pascal’s Triangle is found by adding the two numbers to the right and left of it on the row above it; where these don’t exist, they are assumed to be zero.<p>1<p>1 1<p>1 2 1<p>1 3 3 1<p>1 4 6 4 1<p>1 5 10 10 5 1<p>(This is correctly centered in the actual document.)<p>Write a function that takes a single integer n as a parameter and returns the nth row of Pascal’s Triangle.<p>For example:<p>pascal(1) should return [1]<p>pascal(4) should return [1, 3, 3, 1]
======
UnoriginalGuy
> Each element in Pascal’s Triangle is found by adding the two numbers to the
> right and left of it on the row above it; where these don’t exist, they are
> assumed to be zero.

That gibberish. Unless you know what "Pascal’s Triangle" is prior to seeing
this you have no shot of even understanding the exercise.

You haven't even said how you're meant to iterate through the numbers. If this
is meant to be an explanation of a "Pascal’s Triangle" it fails at that.

You aren't testing programming, you're testing obscure maths knowledge and the
ability to decrypt your question/explanation.

I bet a lot of the other posters had to google this just to answer it in their
impressively quick times.

~~~
logn
I agree. "adding the two numbers to the right and left of it on the row above
it" makes no sense to me whatsoever. I could interpret that in about many
different ways, and during an interview I'd probably crack. To put this in
perspective, I got a 1450 on the SATs.

My thought process: I take a number, add the right and left on the same row,
then put the result on the upper row. Or, add the left and right and current
and put it on the row above. After a while I guess you mean any position is
the sum of the numbers to the 1 position upper left and upper right.

~~~
DanBC
> My thought process:

Would you still think all of those with the diagram?

    
    
              1
    
             1 1
    
            1 2 1
    
           1 3 3 1
    
          1 4 6 4 1
    
        1 5 10 10 5 1

~~~
malyk
This phrase:

"...adding the two numbers to the right and left of it on the row above it..."

Doesn't make a whole lot of sense. What is the "it"?

row 1: is obviously easy. row 2: I think you are adding 0 to 1 and 1 to 0 to
get 1 and 1 row 3: You are adding 0 + 1 for the first item, 1 + 1 for the
second, and 1 + 0 for the third. row 4: You are adding 0 + 1 for the first
number, 1 + 2 for the second...but the directions say "to the left AND right
of it" which makes me think that the answer should be 4, not 3. 1 + 2 + 1.
Really, you are only adding the number to the left of "it" (which still isn't
clear). And actually, for the first half of the line you are adding the number
to the left and for the second half of the line you are adding the number to
the right.

What you want:

row 1: 0 + 1 row 2: 0 + 1, 1 + 0 row 3: 0 + 1, 1 + 1, 1 + 0 row 4: 0 + 1, 1 +
2, 2 + 1, 1 + 0 row 5: 0 + 1, 1 + 3, 3 + 3, 3 + 1, 1 + 0 row 6: 0 + 1, 1 + 4,
4 + 6, 6 + 4, 1 + 0

So I think your directions are probably confusing people.

Of course, you still probably aren't going to increase your percentages much,
but you could be clearer in what the problem actually is.

~~~
DanBC
Smart people suffer from analysis paralysis - that's why they suck at simple
multiple choice questions. "But it _could_ mean [...]"

No, just take the obvious.

> _Each element in Pascal’s Triangle is found by adding the two numbers to the
> right and left of it on the row above it; where these don’t exist, they are
> assumed to be zero._

Subject of the sentence: 'Each element'

So, "adding two numbers to the right and left of it on the row above it" means
"adding the two numbers to the above right and above left of an element."

> So I think your directions are probably confusing people

Not my directions!

Still, this thread should be useful to OP. There are people who get the
problem, and there are people who don't get the problem, and smartness isn't
the deciding factor.

(I'm stupid, and not a programmer, but I understood what the problem was and
what the desired result would be. Other people are smarter than I am, and are
programmers, but didn't seem to grok the question.)

~~~
malyk
Oops, didn't mean your directions but the OP's, of course. Just thought this
thread was a good place for the answer and you had a nicely formatted pyramid.

And, for the record, I'm a programmer, think I'm reasonably intelligent, and
still not entirely sure I get the question based on the description. Language
is tough sometimes.

------
hkarthik
Many developers who don't come from a CS or Math background will run from such
questions for the following reasons:

1) They've never solved problems like this with code before.

2) They fail to grasp how such problem solving skills can judge their "real
world" experience.

3) Someone across the street/city/village will pay them more than you are
paying and have more softball interview questions because they are more
desperate to bring in any programming talent they can find.

While reasons 1 and 2 are good reasons to pass on candidates, reason 3 is
unfortunately a bigger challenge to solve.

It may be unpopular to say this, but I would take a hard look at the problems
you need your developers to solve and come up with a more real world scenario
to use for interviews and see if you get better results.

------
tocomment
That would probably make a better homework problem. People get nervous in
interviews and can't even do fizzbuzz.

In general though it's better to ask people problems related to things they'll
actually be working on. E.g., have them analyze a bug you recently found in
your own code.

~~~
kls
I never make a candidate take tests. After 20 years in this industry I have
come to the conclusion that they are useless. It's applied Curse of knowledge,
one assumes because they can do it, all others should be able to. I have seen
some of the best developers in the world that cannot do fizzbuzz in an
interview situation no matter how comfortable you make them.

Personally, I have a really good track record of finding and hiring talent. So
much so that, friends in other companies will ask me to find them someone.
Anyways, one of the most powerful things I do to find talent is ask them
either send me some code ahead of time or better yet bring a laptop with a
working project they have done, on it. I then ask one simple question, show me
the piece of code in this project that you wrote and are most proud of.
Someone that is BS'ing can't BS through that. It shows real world application,
and people are generally most proud of things that they had to overcome
adversity on, they are happy to talk about how they solved the problem. People
try to make interviewing hard, but with that single question you will know
everything you need to know about their technical skills. Personality skills
is another matter, but I just take them to lunch and hang out with them to
figure out if they are a fit personality wise.

~~~
klaut
That is an excellent way of conducting an interview. Are you hiring? ;)

~~~
kls
At the moment I have no plans to expand. My current company is just a one man
consultancy (service based). At some point though I will start or join another
product based start-up, at which point I will be. If you would like, contact
me via my info in my profile and when that time comes I will let you know.

------
pbreit
The question completely confused me, probably because I've never seen it
before and the triangle is apparently not printing correctly here.

I don't really care for interview questions that are unlikely to show up in
the real world. This feels like sort of a "gotcha" question. Perhaps you will
get the best of the best in your 10%. But I suspect you are missing a lot of
candidates who are very good at real-world programming.

------
EliRivers
Took a minute to code a solution using the formula for line n of Pascal's
triangle, and ten minutes (my usual mix of 30% manic typing, 70% bug-fixin'
and finding the typo where I wrote "naemspace" :p ) to get an iterative line-
by-line working solution in C++. You say you're hiring? I'm in the UK and I'm
more than a little bored where I am.

~~~
jlangenauer
We do mostly Ruby on Rails as well as C#/Unity 3D. Send me an email at
jason@beyondthestory.com .

~~~
EliRivers
I tick precisely none of those boxes. Ah well, back to debugging today's C++
infinite template error :p

~~~
Peroni
Don't stick with a job that bores you. It's soul destroying. My site might
have a few bits and pieces that may interest you:
<http://hackerjobs.co.uk/jobs>

------
richeyrw
I don't think you're being too hard, but your instrument may be too blunt. If
you basing a large part of your hiring decision off this one test then you may
be eliminating people unnecessarily and occasionally including someone who
doesn't belong. For example I'm pretty rusty as a programmer, having graduated
to management many years ago (and a linux admin before that) and I could pass
this test, but you wouldn't want me as a senior programmer.

The point I'm getting at, is that like any science experiment you need to
control for the variables. Just like an economics experiment might control for
wealth, or education, you have to control for people who get nervous during
interviews, people who have a blind spot for this particular problem, etc.

An easy addition to your methodology would be to have three different problems
and let them choose. I would see how that affects their performance.

In my company I give potential hires a test for them to complete at home,
before even interviewing them. The average time to complete the test is 6
hours, but we've never had anyone outright refuse to attempt it, and it gives
us a giant steaming pile of data. So if my experience is any guide (and it
might not be) I would think you could make your test more intensive, and
potentially end up with more and better candidates.

------
steferson
Do you actually ask them to do this on a piece of paper?

because I went through an interview process not too long ago, and they asked
me to take a test that was actually very simple stuff, fibonacci sequence,
dealing with strings, etc.

And for the life of me I just couldn't do it on paper, if I was still in
college I might have done it, but now that I'm a professional developer, I
DON'T code on paper anymore, it feels weird and awkward. Maybe do a test, and
let a few take the test on the computer, doesn't have to be in an IDE, just
let them hack away at notepad, or something.

I know most people believe developers should be those guys that can code their
way out of any problem with just matchsticks or something, but really, would
you ever encounter a situation where you need a developer to actually write
code on paper?

And before anyone says I'm just a bad programmer, I am usually the go to guy
at my workplace, the guy that usually solves hard issues, and the more
competent I become at actually useful stuff, and actual day to day programming
challenges, the more I feel far from things like your test.

Alternatively, you could also use a real world problem, I think there is
nothing that can get a guy to prove they are a good developer than making
subtle bugs in some code and asking the guy to fix them.

~~~
jlangenauer
No, it's definitely done at a computer, I know I'd find it more difficult to
write code on paper too.

------
mtrimpe
Also consider that you might be below market rate and all the good guys are
passing on you.

The story reminds me of a blog post from a while ago [1] which concluded with
an update that after raising his rates by $5 the first candidate to apply
passed with flying colors.

[1] [http://yakovfain.com/2012/10/11/the-degradation-of-java-
deve...](http://yakovfain.com/2012/10/11/the-degradation-of-java-deve..).

------
Ologn
I have interviewed people at several companies, but at one company I went
through hundreds of resumes, probably over 100 phone screens, and dozens of
interviews. I noticed some patterns.

One was that every technical question after the third one was usually a waste.
Anyone who hit it out of the park on the first three, hit almost all of the
subsequent ones out of the park. No one who struggled through the first three
would suddenly start doing well on the subsequent questions.

Another thing I noticed is people fell into a Gaussian distribution. Some were
on the left end of it, but most of those were weeded out by the recruiters, or
by resumes or finally by phone screens. Just like a bell curve, most of the
people were in the middle. Of the dozens I interviewed in person, only three
were on the right end of the bell curve. Which is before you even get to soft
skills.

Your results speak for themselves. You have found an exercise that helps mark
the top 10% from the 11-40% from the 41-100%.

You might want to read some blog posts like Jeff Atwood's "Why can't
programmers program?"

------
samwillis
Because I wanted distracting for 5 min, in python:

    
    
      def pascal(row, prev_row=None):
          if row == 1:
              return [1]
          else:
              if not prev_row:
                  prev_row = pascal(row-1)
              new_row = []
              for i in xrange(row):
                  if (i == 0) or (i == row-1):
                      new_row.append(1)
                  else:
                      new_row.append(prev_row[i-1]+prev_row[i])
              return new_row
    
      def none_recursive_pascal(row):
          current_row = None
          for i in xrange(1,row+1):
              current_row = pascal(i, current_row)
          return current_row
    
    
      for i in xrange(1,10):
          print pascal(i)
    
      for i in xrange(1,10):
          print none_recursive_pascal(i)
    
    

EDIT: Added a none recursive version.

------
dgunn
I'm not sure what your startup does, but I would bet there are better things
to ask a candidate than this. You get these problems in school and become
quite good at them in the moment but after doing real world work, this mindset
will fade. I consider myself a pretty good coder. I've made full production
apps that work just fine. But who knows how I would handle this problem if
handed to me at an interview I'm already nervous about.

Give this problem in the same setting to some of your currently employed
programmers. If they have similar results as your candidate then your
candidate filter sucks and you should re-evaluate it.

------
Throwadev
I wouldn't expect them to write the code for it, maybe some pseudo code, but a
candidate should be able to talk through the problem and come to a solution
for this in 10-15 minutes. It also depends on what kind of work you expect
them to do. If you are building iPhone apps or doing plain old web dev, you
probably don't need candidates who can solve this in <10 minutes.

I'm not surprised by your 90% no-solve stat. Candidates I interview who put
Java/C# on their resumes can barely "write a function that returns true if the
first letter of a string parameter is capitalized". At least half flat out
can't do it.

------
adcoelho
It took me a couple of minutes to remember the pascal triangle due to the way
you are presenting the triangle. I'd rather have it structured as such:

    
    
       1
      1 1
     1 2 1
      ...
    

It makes it easier to understand when you say 'the two numbers to the right
and left on the row above'. However, the way you present it may make it easier
to find a solution since it hints to storing the numbers in an array that can
be used in each iteration and highlights the relations between the positions
of the numbers. But i guess much of this shouldn't be a problem to experienced
programmers.

~~~
jlangenauer
That's an artifact of the HN display system. When it's presented to the
candidates, it's definitely correctly centered.

------
bdfh42
I would say that for an average group of candidates the results were about
right. About 10% of "programmers" can write a simple program that solves a
simple problem when asked - most can't.

------
bdevine
Hi, new guy here. My solution took me about 15 minutes to code in Python, but
I looked up my algorithm (I am not nearly as math-oriented as I should be;
programming came later in life to me). I'm genuinely curious about anyone's
opinion of where I would be ranked among candidates.

def gen_pascal_row(n): row = [1] for i in range(1,n+1): row.append(row[-1] *
n/i) n -= 1 return row

ETA: This might be better. <https://gist.github.com/4612320>

------
stevenameyer
I'm a firm believer in talent dilution being a huge problem, so if this is the
standard that you set for full time hires then keep it. Although it may be a
good idea to think about wether or not this problem is representative of
someones abilities to fill the position. You don't want to fall into the trap
of asking an interview question (albeit a fairly easy on in my opinion) that
may not tell you how good someone would be at filling the postion your hiring
for.

------
vinnybhaskar
Unless you are hiring a developer for a critical banking software or something
similar, you should never base your interviews on Math. Rather, interviews
should be based on the problem solving attitude of the candidate. I'd prefer a
candidate who can come up with an innovative and cheaper solution to the
problem at hand than a person who is better at Math. Though that person would
also need to be good at the programming skill that he/she is being hired for.

------
hcayless
Are you asking them to do this in a particular language, or just to come up
with an algorithm? This might work better as the sort of problem you ask an
interviewee to talk you through, rather than an "I'll check on you in 10
minutes" kind of exercise. But in general, no, this kind of thing isn't out of
bounds, and I'd say if 10% of your candidates are passing it, you're in the
right sort of neighborhood.

~~~
jlangenauer
Also, they don't need to come up with an algorithm: to solve the problem, you
simply need to implement exactly what's written in the problem statement -
calculating each element based on the previous row, and with a couple of
conditional statements (or other ways) to handle the edge cases at the, er,
edge of each row. There is a bit of a conceptual leap in working out a
recursive solution is possible (as is an iterative one), so I'll sometimes
gently drop hints if a candidate seems to be struggling on this point.

------
loungin
It took me about 20 minutes to solve using a recursive solution error-free,
and I have not been doing any math intense work lately and am well out of
practice.

Anyone with decent exposure to math has at least SEEN pascals triangle, so I
think it is a fair question. Try as you might to make them comfortable, they
are probably still nervous regardless.

------
darkxanthos
This doesn't seem too hard to me but maybe that's because I'm mathy. At one of
previous places my boss had found that he was able to boil down the interview
to a take home problem that the candidate should just be prepared to discuss,
not even solve. Literally 1 out of 10 did this. The rest gave up before hand.

It's hard to find good people.

------
jbrooksuk
Something like this should do the trick, good old coffee.

pascal_r = (n) -> lines = [[1]] for i in [1...n] last = lines[i-1] lines[i] =
[] for j in [0..i] lines[i][j] = (last[j-1] or 0) + (last[j] or 0) lines[n-1]

console.log pascal_r(1) console.log pascal_r(4)

Edit: Bloody HN formatting got the better of me again.

------
iamjochem
given that "Ruby Programmer" >= "PHP Programmer" (not necessarily true, but be
honest PHP is considered pretty low on the food chain around here :)), and
given that I have no CS or Math background, I whipped up an answer to this in
about 5 mins:

    
    
        https://gist.github.com/4611869
    

Even taking into account the effect of 'nerves' in an interview setting, on
the face of things the 10% mentioned seems a bit depressing.

I do wonder how relevant or helpful this particular exercise/test is with
regard to accurately gauging a candidate's potential efficacy within [a|your]
development team. My instinct suggests to me that being able to write an
algorithm under scrutiny and/or pressure is actually a fairly low priority
skill (in most development shops) when compared to the other many other skills
that are required to be a successful senior developer.

------
FilterJoe
data point: I started learning Python/Django 4 months ago (after a 20 year
break from programmiung). Took me 40 minutes to solve. I'm not yet fluent with
syntax so I had to look up (and still managed to mangle list processing syntax
and was slow to interpret error messages related to accidentally swapping my
variables for size of triangle and the list. My logic was fine once I got got
the syntax down and swaps corrected. So your test would have caused me to be
weeded out, I guess.

Had you asked me to pseudo code it I could have done it in less than 10
minutes.

Note, however, that learning web development requires learning many different
pieces in addition to Python, so I simply haven't had the time to thoroughly
learn all the syntax of any given piece.

~~~
FilterJoe
Worried that I'm too slow, I then tracked down the reference to the fizzbuzz
test: [http://www.codinghorror.com/blog/2007/02/why-cant-
programmer...](http://www.codinghorror.com/blog/2007/02/why-cant-programmers-
program.html) That one took me 11 minutes. There was less syntax and more
logic tested with fizzbuzz. The only thing that tripped me up for a few
minutes was I found out (through debugging) that integers in Python stay
integers when divided. So I had to float(i) before dividing by 3 or 5 to get
it to work.

I don't know if you're trying to weed out people weak on syntax but your test
may very well do that. Syntax fluency generally gets better over time, I would
think. Not sure basic logic and debugging skill gets better over time.

------
thisiswei

         def cal(n):
             if n == 1:
                return [1]
             else:
                x = cal(n-1)
                return [1] + [x[i]+x[i-1] for i in range(1,len(x))] + [1]
    

who wanna hire a junior developer? I will work for free in SF,CA

------
pandaexpress
I actually had a bit of trouble with this problem, it took a while to iron out
the bugs. I copied down the triangle for reference but accidentally had 1 2 2
1 as the third row.

Took me about 15 minutes to do. Doesn't seem like an unreasonable question to
ask.

~~~
pandaexpress
The solution (in Objective-C)

\- (NSArray _)pascal:(int)n { if (n == 1) return @[@(1)];

NSArray_ array = [self pascal:n-1]; NSMutableArray* mArray = [@[]
mutableCopy]; for(int i = 0; i < array.count+1; i++) { int r = i - 1; if (r >=
0 && i < array.count) [mArray addObject:@([array[r] integerValue] + [array[i]
integerValue])]; else if (i < array.count) [mArray addObject:@(0 + [array[i]
integerValue])]; else [mArray addObject:@(0 + [array[i-1] integerValue])]; }

return [mArray copy]; }

------
Peroni
Have you set the same challenge with the same time constraints for your
current team? If you have and they all pass with flying colours then don't
change a thing. If you haven't, then it might be worth testing the challenge
with them.

------
dear
10% is disappointing but it also means it's a good filter, particularly when
you think recursive algorithms are critical in your projects. A question with
90% passing rate is useless as a filter.

------
squidsoup
What is your business domain? You might find it more beneficial to ask your
candidates to solve something relevant to the work they would be doing for
your company.

------
jyu
Are other senior developer positions in your area offering £50k to £60k? The
equivalent $80k to $95k seems to be the going rate for junior/mid developers
in the US.

~~~
whichdan
Outside of SF and NYC, where are all of these $80k junior positions?

~~~
aoprisan
Boston, DC area, Atlanta, to name a few

~~~
whichdan
$80k in Boston is definitely closer to mid/senior starting salaries.

~~~
dragons
Depends on how you define mid/senior. I started at a Boston company with about
10 years of experience in '08. My "mid-level" salary was ~ $90K. I ended up
around $100K after about 3 years working at the same company (my title got
prepended with "Senior" at some point, but it didn't start that way). These
numbers do not include bonuses, which varied.

------
simonster

      pascal(n) = [binomial(n, i) for i=0:n]

~~~
deservingend
lol, now define your binomial function.

~~~
simonster

      binomial(n, k) = factorial(n)/(factorial(k)*factorial(n-k))
      factorial(n) = n <= 1 ? 1 : n*factorial(n-1)

------
px1999
I'm working from random statistics for London that I found on the internet, so
please forgive me if this doesn't apply.

You seem to be paying average - average+10k (for all, not just senior)
developer salaries, yet you specify senior programmer. At this level, you're
only going to be attracting mid-level talent. There could a problem with your
recruitment targeting, which could mean you're not getting ideal people in the
door to begin with.

At the mid-level, you should expect a high failure rate on simple tasks.
You're dealing with people who wound up on the job market after getting a
bunch of experience. This means that a lot of them were fired, came from
untenable/poorly managed companies which went under/downsized, or were working
under such unbearable conditions that they quit. Employed devs trade up -
unless your pay packet, benefits or feel-good factor (eg if you're an NGO,
charity, working on 'real' problems, etc) give your company an edge, you're
not dragging people away from jobs where they're encouraged to develop their
skills, work under good conditions and understand software engineering
processes. Job postings regularly ask for ridiculous and irrelevent things, so
a lot of the employed devs who read your job ad and apply will not meet the
criteria you specify. I know it's a huge generalisation, but the best devs are
usually only on the market a couple of times in their lives, and they're
quickly snapped up by companies at a premium and kept happy with various
incentives. Tl;dr: you should be prepared for most candidates that you
interview to not be appropriate for the job you've posted.

Thirdly, there's a problem with your, ahem, problem. It's not a great problem,
imo because there's a fair amount of nuance to it. I'd want to shine by giving
a good solution - vaguely remembering that a value in Pascal's triangle can be
derived from combinations or permutations or something similar I'd spend some
time trying to figure out if there's some algorithm "n!/(n+k)! - no, that's
not it...", then I'd try to do it in-place calculating only half of the
triangle, and then probably wind up using two rows of half length to calculate
the numbers there. I'd be trying to impress you by coming up with a solution
with more useful properties (memory boundedness, complexity, speed) than
"returns the row of pascal's triangle". You're probably not even hiring people
for their comp sci experience.

Maybe you just wanted to vent, or to advertise here - that's ok, but keep in
mind that you're going to have to quickly reject unsuitable candidates and
quickly pick up suitable ones, that it's just a numbers game, and that this is
why most companies that offer mid-range compensation are "always hiring". The
problem isn't _too hard_ as such, it's perhaps a little undirected. Also, some
people just interview terribly.

Dunno, just some food for thought.

------
michaelochurch
No, you're not being too hard on your interview candidates. That's not a hard
problem, at least under basic assumptions (not concerned with performance,
integer overflow, etc.)

I would say you should pass on the 60% who can't solve it, unless they take
the initiative and send you a solution after the interview (I've done this,
when I could solve the problem but not in time). That might prevent you from
passing on an otherwise good candidate who happened to choke once. The 30% in
the middle you can attribute to interview stress, if other indications are
that they know how to write code.

The truth is, though, that a large percentage of "professional programmers"
have never built anything from scratch and have been wearing training wheels
the whole time. 90 percent of what professional programmers do is maintenance
at ~10 LoC per day, never leaving the comfort of the IDE or building a system
from the ground, and so there are a lot who've never been in a green field or
had to solve a hard programming problem. You don't want to hire such people to
build new stuff. You're not being "too hard" on them by filtering them out.

In fact, I'm surprised that your fail rate is only 60%.

