
Why 37signals Doesn't Hire Programmers Based on Brainteasers - teaspoon
http://37signals.com/svn/posts/3071-why-we-dont-hire-programmers-based-on-puzzles-api-quizzes-math-riddles-or-other-parlor-tricks
======
dmbaggett
I helped start the practice at ITA Software (acquired by Google last year) of
using puzzles as part of the hiring process. We ultimately even put our
puzzles in the Boston subway as recruiting ads. My assessment, after more than
a decade of doing it:

\- Good puzzles are actually a talent attractor; many smart people found out
about our company via our puzzles.

\- Good puzzles are ones that scale well; i.e., where the basic problem is
pretty easy and can be solved by a decent programmer in a few hours, but with
harder variants that can take much longer.

\- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people
out who don't really like to program and/or can't finish things; this is a
useful filter. If someone complains that "they shouldn't have to spend three
hours writing code to get an interview," that itself is a pretty good counter-
indicator. (Yes, we are all busy. But you're talking about starting a
relationship with the company that may last 10+ years. You can do three hours
of prep work for your interview. And you have to code a fun puzzle in the
language of your choice, not some subroutine in a 30-year-old COBOL banking
system.)

\- It seems that in-person whiteboard or locked-in-a-room tests are pretty
poor indicators of success. Many good programmers put in this situation
significantly underperform their true abilities.

\- It's not a great idea to evaluate someone _purely_ on the basis of puzzle-
solving ability.

\- Many one-liner puzzles are bad indicators, because you either need to "know
the trick" or have memorized the answer. Many, but not all, Microsoft and
Google interview questions I hear about -- I've never interviewed at either
place myself -- sound like they fall into this category. (E.g., from a recent
article I read about Google: "You're reduced to the size of a nickel and
thrown into a blender. How do you get out?" I don't care if you're clever
enough to answer that; I just want you to able to write enormous amounts of
high-quality code for me.)

Im my experience, the best way to find out if someone is a good programmer is
to talk to him/her at length about something he/she has built. Can he/she talk
in detail about how it worked? What challenges were involved? What tradeoffs
were made? And, above all, was there passion behind the work and these
choices?

A good interview for me was one where the candidate came in having written or
contributed to a large system -- for fun or work -- and was excited to tell me
about it. It didn't really matter whether it was relevant to the work we did
(airfare search). Anecdotally, it seemed like people who really loved to code
generally worked out pretty well. People with great-looking resumes that
didn't love to code -- but maybe loved other things, like arguing over which
language, operating system, architecture, or business strategy was better --
usually didn't produce much.

If the first thing you want to do when you wake up in the morning is code
something, you're probably going to do well as a coder on a hard-core software
team. Otherwise, not. Everything I personally did on programmer-hiring
strategy was a proxy for figuring out whether someone was like that or not.

~~~
ekidd
Maybe I'm weird, but I find puzzles to be a turnoff. I love writing code, and
I enjoy working through a math book.

But compared to coding and math, puzzles feel sterile. I don't get to build
anything, and I rarely get the kind of insights I would get from figuring out
a proof. (Raymond Smullyan's puzzles are a marvelous exception to this.) Even
cryptanalysis is more fun, because there's a real opponent.

I know lots of amazing programmers who love puzzles, and that's cool. But if
you only hired puzzle lovers, you'd miss out on a lot of good people.

~~~
skrebbel
Same here. My impression is that the puzzle lovers turn out to be the
algorithm geeks. You need a bunch of those, but I can't see how a room for of
complexity analysis experts necessarily creates well-structured software.

I'm an API design geek. I've _never_ seen a cool puzzle about API design, or
something remotely related.

~~~
andrewcooke
i'm pretty sure one of quora's questions is/was designing an api for something
in python (probably time, given how bad the current api is).

also, replying more to the other post above - you can get insights from
puzzles. a lot depends on the puzzle (which is why ita is getting praise
here). for example, the ita puzzle i remember taught me a lot about handling
data in large numbers of dimensions (in particular, it made me think about the
"curse of dimensionality" which was something i was aware of before, but
hadn't considered in detail; it also led me to understanding some tools for
dealing with it - locality sensitive hashing being the most interesting at the
time).

------
marknutter
So people who handle these stressful situations well and solve these problems
under pressure only represent one type of developer - which Google has in
spades - the kind who does very well in a structured environment, got great
grades in their undergrad program, loves to worry over interesting and
difficult problems, etc. The kind of people who don't do well in these
situations (like myself) are often valuable in other ways. Some people cave
under pressure, and even more people have very low self-esteems when it comes
to programming and intelligence in general. Yet these same people can be
highly valuable and productive given enough time to make mistakes and solve
problems in their own way. These types of people may be better at seeing the
bigger picture and are less likely to over-engineer a solution or fall into
the not-invented-here trap. I think it's interesting that Google finds a lot
of bright engineers using these high-pressure hiring tactics, but that they
often turn to wholesale talent acquisitions when it comes to building products
like Google Plus. My guess is the vast majority of the talent Google has
gotten through acquisitions would have failed miserably going through Google's
traditional hiring process.

~~~
gruseom
_My guess is the vast majority of the talent Google has gotten through
acquisitions would have failed miserably going through Google's traditional
hiring process._

That's a very good point that I've not seen made so well before.

~~~
kyt
I've heard that Larry Page could not pass the Google interview today.

~~~
jrockway
This is probably a lie. The Google interview is not particularly hard. It
might be hard if you absolutely refuse to ever let your eyes look at anything
that could possibly be considered an "algorithms book", but otherwise... it's
stuff that you will know if you care about computer programming at all. The
questions are about applying big concepts, not implementation details of
particular algorithms. (I couldn't write A* from memory, but I do know what an
"admissible heuristic" is. And that's good enough to be hired by Google.)

------
kstenerud
Hiring developers based on brain teasers is like hiring journalists based on
how well they do at scrabble.

~~~
Vargas
Couldn't agree more! I will quote you in the future.

------
robterrell
My first interview out of college was for a position as a Mac developer for a
company that made a PC email product (for Novell's LAN message exchanger --
this was in 1990). I'd done several business-focused Mac applications while in
college, including a really complex EDI app, and I thought they'd be
interested in seeing them as part of the interview. I even brought selected
portions of source code.

Nope! Instead they gave me an hourlong verbal interview, which went great,
took me to lunch with the team, which went great, then sat me down with a lead
developer, who had printed out some of the winning entries from the obfuscated
c contest, and asked me to read them and tell him what each program would do
when compiled. That did not go so great. I didn't get the job.

That experience remains burned into my brain, and as a manager I won't give
puzzles to an interviewee. I'll ask to see code and expect them to speak
intelligently about it & the choices they made, but I won't ask them to write
code (or, compile code in their head) on the spot.

(Postscript to my story -- that company never shipped a Mac email client. I
believe they interview-filtered themselves out of the entire Mac market.)

~~~
agentultra
One of my biggest pet-peeves with the current trends in industry hiring
practices are the puzzles.

Mainly because I do bring in lots of source code. I write tonnes of code in my
free time. I contribute to open source projects on occasion and have released
some too. It's all out in the open. All my blog posts, everything. I'm an open
book when it comes to my ability, what I've worked on, etc.

Yet 95% of the interviews I've walked into the people sitting across the table
from me had to look at my resume while they were shaking my hand to find out
my name.

Then the puzzles come in the hopes that I will be filtered out quickly so the
person interviewing me can get on with their day.

I also find that in such an intense situation my brain is hard-wired to find
the "right" answer so that I get what I want without getting burned. It's kind
of hard to have a jovial discussion about code and logic when you've got a lot
riding on getting the job. It's almost Pavlovian.

------
abalashov
After considerable reflection upon this point, I tend to fall on the side of
those that say that a really _simple_ problem, like FizzBuzz, is quite
valuable as a negative filter. It saves one valuable time in dealing with
people that, as it turns out, can't code at all, literally, but are incredibly
good bullshitters. There are those. But that's about as far as I think
whiteboard code quiz shows should go.

It took me a long time to appreciate the utility of something like FizzBuzz in
terms of how Pareto-optimal it is.

I think it finally clicked when I was getting interviewed for my US
citizenship by USCIS in 2008. In the preliminary stages of the application
process, the officials had gotten us all psyched up about how there is going
to be a US civics test and an "English test". We were given a 100-question
booklet of Civics 101 and a lot of anticipatory anxiety.

So, imagine my surprise when the interview came around and with it, the
storied, fabled English proficiency test. You know what it was? Listening to
one simple sentence read aloud and transcribing it accurately. It wasn't even
a hard sentence, it was something on the order of "The man walked his dog
across the yard and over the turnpike". Really? Really?! I was flabbergasted.
Come on, I could pass this in like 4 languages. One sentence? That's all?

After thinking about it, I realised there was more utility to it than I had
appreciated, though. Yeah, I've been speaking English for 18 years, but there
were people in the room who literally did not. They could not pass this test.
And for someone who really cannot speak, write or understand English much,
this test would be genuinely challenging or impossible to pass. At the same
time, 90% of people who can pass it probably are sufficiently proficient in
English to function in this society. I'm not arguing that it's a particularly
insightful, perspicacious or revelatory test, but the point is that it
accomplishes more than you think, from a statistical perspective, even though
anyone who is even marginally literate would dismiss it as ridiculously easy
to the point of absurdity.

So it goes with "a priori" programming tests, I think.

------
edw519
Funny, every time I read "puzzle" or "brainteaser", I think, "I don't know why
manhole covers are round," or "Who gives a shit that it took 7.612 seconds for
my program to identify the 78,498 prime numbers less that 1,000,000." (Yes I
know, get a life.)

But every time I read "FizzBuzz", I drop everything and go back to refactor my
latest version, aspiring to get it down to one conditional.

~~~
dpritchett

        >> (1..100).each { |n| puts `curl -s fizzy.heroku.com/#{n}` }

~~~
skrebbel
win!

------
ekidd
I don't enjoy brainteasers. If an interview question begins, "Four people and
a goat need to cross a river...", I cringe.

But when I'm interviewing, I _do_ ask people for a pointer to their open
source projects. And if they don't have open source projects, I give them a
workstation and ask them to write code. I've seen too many self-proclaimed
programmers who claim, "I spent the last 2 years writing Python," who can't
sum a list of 10 numbers without using Google. (Hiring can break my heart.)

One way or another, you've got to see code. Real code is best, but it's better
to ask people about toy data structures than it is to hire 3 team members who
can't write FizzBuzz.

~~~
vidarh
> who can't sum a list of 10 numbers without using Google.

I did a hiring round once for my old team when I worked at Yahoo years ago,
where out of 10 or so CV's the recruiters passed me, about 5 people who were
given a short list of screening questions _by e-mail for them to answer at
their own leisure_ produced answers that were obviously plagiarized from
various online sources.

I got suspicious when one guy presented me with two pages of nicely formatted
answer complete with section headings that didn't make any sense to a question
that should have required a one line answer. He'd copied and pasted the whole
thing from an on-line copy of an Oracle manual, so I started doing searches of
random sentences from their answers.

Another one managed to cut and paste an answer that was flat out wrong from a
forum post answering pretty much the same question, and hadn't bother to read
further in the comment thread, where several people corrected the answer he'd
used...

It's when they can't even answer simple stuff WITH using Google it gets really
depressing, and it happened regularly enough that using screening questions by
e-mail was a real time saver in weeding out undesirables, despite the ease
with which people could "cheat".

~~~
lutorm
This sounds like what happens when you give students problem sets...

------
jroseattle
To DHH and team: thank you for using reasonableness and sanity in your hiring
practices. Wish this was more the norm, but indeed it seems to be the
exception.

Trying to evaluate programmatic skills is surely a challenge, but I've found a
method that seems to work very well -- simple walkthroughs. I'll do at least
two walkthroughs of code, asking a candidate to explain to me what's happening
using two different artifacts:

* Some code they've written that solves some problem

* Some code we've written that solved some problem

This has proven to be quite effective in rooting out those who look great on
paper and talk a great game, but fall apart when put into action. On the flip
side, I've yet to find someone who could explain their code in sufficient
detail that then proved incapable once in the job.

------
HarrietTubgirl
Here's the main thing that fucks this all up: there are a very limited set of
brainteasers out there.

This means that if someone gets your brainteaser right in an interview,
there's, say, a 90% chance that they've heard a question just like it before
(or one with a similar "trick"). The false positive rate is too high. Many
people know the answers to brainteasers _far_ beyond their ability.

If you were to ask a completely new brainteaser, as difficult as the canonical
ones, but requiring a different trick, you would get a very low success rate.

Side note: there's too much criticism on HN about "what does this question
have to do with my day-to-day job". This is a bullshit point. The point of a
brainteaser is to test problem solving ability, creativity, etc. It may not do
a very good job of that, but just because you won't be "reversing sentences in
place" in your job doesn't mean that the question is invalid. Think about what
the question is trying to test.

------
sunir
The best interview is a pair programming session around some 1-2 hour problem.
It takes longer, and it may come later in the interview process, but it's a
high bandwidth way of assessing someone.

If they don't have a strong open source background (not everyone does),
earlier in the process it helps to ask a couple very simple programming tasks,
such as writing a function to reverse a string in place or the infamous
fizzbuzz. It will weed out people with great resumes but poor skill.

~~~
fogus

        The best interview is a pair programming session
    

Maybe, but I've come to find that pair programming is a certifiable skill that
unless you've been doing it for a while needs honing. Pairing with someone
great might not seem great if they have no experience with pairing.

So once again, we're back to: there is no single best-way to fully assess a
candidate.

~~~
sunir
Well there is the intense XP understanding of pair programming, and then there
is just sitting next to someone while they code version of pair programming. I
can't believe anyone hasn't done something in that spectrum.

Even for juniors and new grads, at school people code next to each other all
the time.

~~~
fogus

        sitting next to someone while they code 
    

Then that's not pair programming. Pairing is an act of give and take, Ying and
Yang, Ping and Pong, and sometimes Tweedledee and Tweedledumb. Sitting next to
someone looking over their shoulder solving a problem that you know the answer
to is not pair programming. There is no act of mutual discovery at all.

~~~
sunir
Why limit interview questions to problems you already know the answer to? You
can have them help you with some problem in your actual codebase you're
already working on.

~~~
kooshball
> You can have them help you with some problem in your actual codebase you're
> already working on.

This is a terrible idea. If they come up with a brilliant solution can you
actually use this in production? Who owns the code they write during an
interview?

------
raghus
Don't know about the rest of HN but I found it strangely reassuring that even
someone as talented as DHH can end up in front of a white board during an
interview and feel stupid. I've been there...

~~~
statictype
There are various types of talents.

He's obviously very talented at design (not necessarily only UI design but
general program structure and framework design). But if you needed someone to
write packet routing algorithms or design a high-availability database engine
from scratch, that may not be his cup of tea.

OTOH, I bet there are developers who love brain teasers and puzzles who may
very well excel at that type of work but couldn't implement a huge software
system with many moving parts without having it collapse under its weight.

------
latch
"The only reliable gauge I’ve found for future programmer success is looking
at real code they’ve written"

This a thousand times over. Go through their github (or whatever) profile,
look at their commits, see how they write tests, etc. It doesn't take a huge
amount of time, if you are any good at what you do, you'll quickly be able to
filter out a failure (like, he doesn't have code that you can browse).

~~~
nik_0_0
Every time I read this it frightens me, I don't have an active github account,
therefore I am a failure? I'm currently working a 16 month internship before
graduating with a Computer Engineering degree. My code from this work will not
be available anywhere of course.

Not through any fault of that though, I realize that having a github account
with projects and such that you work on is a great help, and I accept that
myself not having one yet is a great disadvantage. What do you recommend to
get started out on github? Is helping out open source programs still viable?
Or now it seems either writing self-run projects, or forking others projects
and helping them with those. Is there any clear starting point for githubbing?
(and how do you squeeze it in after a 10 hour work day!)

Just questions from someone worried about my empty github profile :)

~~~
CPlatypus
Don't worry, the "github or git out" attitude is only common among people you
wouldn't want to work with anyway. They're the ones who will change languages
or frameworks according to what's fashionable, rat-hole the team on fun but
irrelevant projects, and not talk to you unless you dress like a hobo and
swear like a sailor. They're over-represented here on HN, but real engineers
know that most good code isn't on github and never will be.

~~~
kd1220
My experience with gung-ho githubbers is the same. It's great to try out new
technologies, but gitguys take it to an extreme. They'll shoehorn any new
technology into their stack without a second thought about product stability.

Gitguy: "I used EC2, Blazboo and Chingbang to create an HA job queue that will
never fail! It uses counting Bloom filters and I wrote it in Brainfuck."

Me: "What do you use it for?"

Gitguy: "Sending password reset emails."

These Rube Goldberg wannabes are generally more trouble than they're worth.
You end up with a system that's a technological pastiche. The drawback is
apparent when you try to hire new teammates. It turns out you can't find
someone who knows the 12 esoteric packages your business is running on.

~~~
latch
The state of programming is so awful, that I'm literally just looking to know:
do you write fat controller actions..or whatever equivalent exists for the
specific technology you used.

That, to me, is 100x more important than "how many zeros are in 100 factorial"

------
pors
> The only reliable gauge I’ve found for future programmer success is looking
> at real code they’ve written, talking through bigger picture issues, and, if
> all that is swell, trying them out for size.

I agree with the latter. This is how I used to hire programmers, one month on
a freelance contract first with a single project. The importance of the code
itself is also overrated IMHO. It's just one tiny part of someone you add to
your team, what about creativity, taking initiative, never giving up, being a
team player, etc. Each of them are equally important to plain coding skills.

~~~
ori_b
It's reliable. However, it's not competitive. You're assuming that I have
nothing better to do with my time than to work for you on a freelance contract
for a month, with no guarantee of a full time position, probably no benefits,
etc.

If I'm job hunting, I've got other offers on the table. Ones that do guarantee
a salary for more than a month. Ones that include benefits. Ones that expire
long before your trial hire would be over.

If you try using this method, I don't know of a top notch programmer that you
would be able to attract, simply because the hiring process is too painful.

~~~
dpritchett
I imagine David is savvy enough to present it as a mutually beneficial
arrangement.

As a prospective employee wouldn't you _like_ to kick the tires before making
the leap into a new job that you might find yourself wanting out of six weeks
in? There's so many things you just can't know about a company until you're on
the inside.

Edit: Remember that 37s has a lot to offer: Remote work with great people,
top-tier name recognition in the tech world, and a long history of building
and releasing open source tools. That's a really compelling package compared
to some of the bigger names in tech.

~~~
ori_b
If I'm uncertain enough to want a trial period, I'd also want to keep my
options open. If the position wasn't a good fit, then by the time I find out,
I've been forced to turn down my alternatives.

No matter how you look at it, you're asking me to take on significantly more
risk with this process.

For developers that are in demand, it's a tough sell.

------
ericmoritz
I think the best interview I had was at Mochi Media. What each interviewer did
was pick some programming headache that gave them trouble in the past couple
weeks and present the problem to me to see how I would solve it. It was
probably the hardest interview I ever had but I think it's great way to assess
wether a candidate is able to solve problems they see.

------
msg
I would hope "you just broke the user experience for 50 million people, there
are 30 internal teams screaming your name" is higher pressure and stakes than
an onsite interview loop. You can make or break your career in that situation
too.

I had to deal with a crisis this week that boiled down to a poorly optimized
SQL query or two. Of course, the manner in which they failed meant our DB
started disk swapping and hurting our availability.

Contract to hire is something we do but it is rare. An initial try out period
doesn't do us much good since the learning curve is long.

I don't trust take home assignments unless their solutions can't be googled.
And it's a lot of work to create and evaluate problems of that nature. You
want them just long enough, not a huge time sink for you or the candidate. You
want them to demand a range of solutions so there are many ways to fail and
succeed.

I agree we should not be doing riddles/puzzles. For me the most
straightforward way to gauge a candidate's ability in an hour is to ask them
to program on the spot and see how they act. Yes, it's a game and the
candidate can learn to beat the system and oversell themselves. But the vast
majority of people I don't want to hire do not go to these lengths.

------
jakubw
I don't get the puzzle aversion. Sure, you can stare at someone's code as much
as you want to but I don't think you can infer a lot about how their thought
processes worked when they were writing it (a proper use of version control
helps here but not much). You only see the outcome. You may say - I don't care
how they think, I want the job done. Well, that would make sense if they were
contracting on an self-contained project. But usually you hire someone to join
a team, where how the process works is more important than the actual outcome.
Especially whether or not the process is flexible and fits your workflow. So
puzzles are good for that, especially that it's not about getting the right
answer but approaching the problem. The reason they're abstract, which I agree
sometimes crosses a line, is that leaving out the technical details of a
problem makes sure that both candidates familliar with the technical details
and those that are not have equal chances of doing well.

The problem with this post is that it puts "puzzles", "API quizzes" and "math
riddles" into one category. I can see how one can be offended by an API quiz
such as "what does X do in Y?", especially if they have a track record in Y or
could just look it up on the Web if they didn't know. But math riddles?
Puzzles? I'd actually be grateful that I don't have to deal with technical
catches and can purely focus on a solution.

One thing to consider is that recruiting, especially at a larger scale, is
about minimizing false positives rather than maximizing the collective value.
Sure, someone good at puzzles might turn out to be an average engineer but a
bad one? I don't think so. Whereas, I can see how someone with a huge pile of
code on GitHub can get stuck in a critical situation where if they write unit
tests or how they design their APIs is meaningless.

------
jiggy2011
I suppose the type of development 37 signals will do is more concerned with
product design and less to do with fast algorithms etc.

I haven't really used their products but in terms of technical sophistication
they probably aren't much advanced beyond CRUD type web apps (I may be wrong
here).

If you wanted to hire a games engine developer or someone to design algorithms
for google then brainteaser problems might be more relevant.

~~~
jiggy2011
Not sure why I'm downvoted? Am I wrong?

------
cletus
Interview and hiring techniques are a perennial favourite on HN. I just have a
couple of comments (disclaimer: I work for Google).

I personally detest abstract puzzles like "why are manhole covers round?" or
(worse) "How do you move Mount Fuji?" AFAIK these types of questions are not
and have never been used in engineering hiring at Google (sales, etc I know
nothing about). WSJ and other sources have made this (AFAIK inaccurate) claim.
It's possible individual interviewers do this.

My own philosophy is that a _simple_ whiteboard coding problem is an
incredibly useful litmus test. I don't mean anything incredibly complicated
either. If you can't write a function that gives me a row of Pascall's
triangle, that's a problem. Note: I don't expect you to know what Pascall's
Triangle is, merely implement it when explained.

Some of you may think such a test is a waste of time but I assure you it is
not. It's astonishing how many people can't do this (and even more astonishing
how many that do make it through resume and even phone screens).

The key here is "simple". One problem is that interviewers fall into the trap
of thinking these problems are too simple and make the problems increasingly
harder. Worse, they may get in a situation of asking someone a problem that is
either something they know (because they've covered it before) or they don't
(where it's not something you can simply nut out). This is a useless indicator
in my experience.

Just because you can create a function to return a row of Pascal's triangle
doesn't mean you're a great programmer but if you can't, it almost certainly
means you aren't. It's a negative signal filter, nothing more, but an
incredibly quick and useful one.

API pop quizzes I'm not a big fan of either, generally speaking, but on the
other hand if you've used a language sufficiently long, basic questions about
commonly used libraries aren't unreasonable either.

Interviewing is a numbers game. Your goal is to come up with a process that
balances time spent by the interviewer with finding a suitable candidate. Note
that I didn't say accurately assessing each and every candidate. It's
sufficient for the employer to simply find one qualified candidate even if you
falsely determine someone who is qualified isn't.

A certain error rate is to be expected. It reaches a point where the time
investment to reduce it outweighs the benefit of higher accuracy.

One final note: I detest (both as an interviewer and an interviewee) any kind
of "take home" test (other comments have mentioned this).

As an interviewer, it's extra work to assess, people cheat, good candidates
won't bother and so on.

As an interviewee, I'm not going to spend hours on you without speaking to
someone first to find out how likely it is that you're a good match for me and
the likelihood that you believe I'm a good match for you.

I would argue that a take home test is significantly more effort than a simple
coding problem that doesn't have a significant increase in accuracy.

The one exception to this is automated testing where the problems themselves
are relatively interesting. Some of the Facebook Puzzles fell into this
category (and some don't). People may do those anyway for kicks. But again,
there's nothing stopping someone going to Github and copying and pasting
someone else's solution so what value is the filter, really?

EDIT: clarified my Pascall's Triangle comment. This isn't a trivia problem. I
don't expect people to know what it is, merely implement it when explained.

~~~
fragsworth
You know what? I just tried implementing the Pascal's Triangle function. I
implemented the general algorithm, recursively, but I had a bunch of logic
errors that took me about 25 minutes to fix (one was obvious and embarrassing)
- in the privacy of my own office and with a debugger.

I have over 10 years of programming experience and I probably would have
miserably failed your "litmus test", especially if it were in an interview
situation in front of a whiteboard. Maybe this just means I'm a bad
programmer, but I tend to think that problems like "Pascal's Triangle" don't
represent what we work on day-to-day.

~~~
jemfinch
Basic programming like Pascal's triangle represents the easiest stuff we do on
a day-to-day basis. We write code. Pascal's triangle is code. Would you rather
be tested on your ability to comprehend a multi-kloc codebase and make
correctness-preserving modifications to it? Would you rather be tested on your
ability to integrate or upgrade third-party libraries in a massive existing
codebase? Coding is the _easiest_ thing we do on a day-to-day basis; if it's
not representative of what we work on day-to-day, it's because it's _easier_
than what we do, not because it's harder.

I was astonished at the 25 minutes you claimed this took you, so I did it
myself. It took me about two minutes to write this, which worked the first
time I tried it:

    
    
        #!/usr/bin/env python
    
        def pascal(i):
            if i == 0:
                return [1]
            else:
                L = pascal(i-1)
                ret = [1]
                for i in xrange(len(L)-1):
                    ret.append(L[i] + L[i+1])
                ret.append(1)
                return ret
        
        if __name__ == '__main__':
            import sys
            print pascal(int(sys.argv[1]))
    

What problems did you encounter that required you to use a debugger? I'm
genuinely curious, because it seems like such a simple problem.

~~~
rdtsc
After 15 years of programming one doesn't really remember what the heck
Pascal's triangle looks like. You know why? Because knowing about Pascal's
triangle doesn't bring in a paycheck.

If you program for a living, ask yourself in the last 5 years how many times
did you have to write Pascal's triangle? How many times did you have to solve
a brain teaser at work? I'll answer for me -- 0 times.

So I am advocate of posing a relevant problem and taking the candidate on the
whiteboard through that problem. Not "let's imagine you have 100 horses, now
one dies, blah blah..." but "Here is what we are working on. We have 1000
concurrent users, there is an open TCP connection to each, help us solve this
particular latency problem". Yeah they might not know about Pascal's triangle
but they if they can understand and show an ability to solve your particular
problem then they are a candidate for you.

~~~
ori_b
After 15 years of programming, one would hope that seeing the definition of it
would be enough to let you implement it in short order. Expecting to have it
memorized? Of course not. Expecting you to be able to code it up after a good
explanation of what it is? I wouldn't expect you to have any serious issues.

I honestly had forgotten the definition of it (the last time I saw it was in
high school algebra), but I was able to get a working solution in about 10
minutes, including looking at the definition on Wikipedia. I'd say I'm a
rather average coder (I'm certainly surrounded by people far better than I am,
in any case).

If you had never seen or heard of Pascal's triangle before, I'd say maybe 20
minutes to code it would be reasonable.

~~~
bad_user
Well, if the Pascal triangle question is not a question of what is a Pascal
triangle and the problem is properly explained, then it is easy to come up
with a solution.

However, this doesn't highlight good engineers. You know why? Because in the
real world finding out the problem is many times a lot harder than coming up
with a solution.

Here's a sample:

If I say to you that " _I want these and these people classified / sorted,
such that I want to find the people that have the best probability of being a
good hire_ " ... I haven't actually told you the problem. I don't really know
the problem myself. All I know is maybe that the hiring process is broken for
some reason and I'd like to fix it, but I don't know the reasons (I can't put
my finger on it).

And that's not a properly defined problem. We don't even know how to measure
properly if a hire was good or not. And in the real world, that's how most
problems are described to engineers - i.e. I have this pain, solve it somehow.

~~~
jemfinch
> However, this doesn't highlight good engineers.

It's not intended to. It's highlighting bad engineers, not good ones. It's a
negative filter, not a positive one.

> We don't even know how to measure properly if a hire was good or not.

Your company doesn't do performance reviews for employees?

~~~
bad_user
> Your company doesn't do performance reviews for employees?

Every performance review I ever attended was a bad joke, for the same reasons
companies have problems filtering between candidates.

Seriously, if there's a problem with the hiring process (it was just an
example btw, I'm not saying there is one at your company), then why do you
think performance reviews work out well?

------
chrisaycock
Whenever I see a firm use brainteasers during the interview process, I just
wonder why they don't simply recruit the best Sudoku players. That seems about
as valid a way to judge software development skills.

~~~
mirkules
Sudoku measures how creative you are at finding the best logical solution, so
that may actually be a good idea :)

I suggest this with tongue-in-cheek, of course, but it's about as effective as
any other method.

------
znt
I work for a web development agency, which develops projects for some of the
largest companies. We generally work on App Engine (python).

At the day of my interview, they didn't really ask me any brain teasers.

They asked if I knew App Engine, I showed them my open source project for App
Engine.

They asked if I knew django-nonrel, again I showed them another open source
project of mine, that used django-nonrel.

That was it.

Since the day I was hired, I did plenty of good work. Now I'm entrusted with a
global CMS project which is being used by hundreds people and viewed by
millions.

If they had asked me brainteasers, I would probably have failed most of them,
as I don't really practise with those kind of problems. But I like building
web apps. And that's what I am doing right now.

If your company earns money by solving brainteasers, then you should ask them
during interview.

------
danieldk
Companies like Google hire hundreds (thousands?) of developers per year,
hiring policies at 37Signals are bound to be different from those at Google.

If you have to weed out thousands of candidates, trying brainteasers to
estimate their problem-solving capabilities doesn't seem to be a bad strategy.
If a candidate is not familiar with a particular problem, his/her solution may
be suboptimal, but the process of getting to that (suboptimal) solution may be
a good indicator of the candidate's problem solving skills.

Obviously, syntax-checking a candidate's on-paper Javascript code isn't going
to help anyone.

~~~
spot
google does not use "brainteasers" and in fact has a specific policy against
that kind of question. they ask substantive technical questions that often
involving coding on the whiteboard.

~~~
dangrossman
I applied for a developer job at Google once and eventually had two phone
interviews.

I was asked to come up with a sorting algorithm that would run on 4 computers
with more data than could fit on any one computer. I was then asked to do a
complexity analysis on my algorithm.

The only thing I think that really tested was how long it had been since I
took an algorithms course in college.

~~~
andrewem
I used to work at Google. When I interviewed I was rusty on algorithms, so I
studied up on them, even delaying my on-site interview to have more time to
get ready. Sure enough they asked algorithms questions and I did better on
them thanks to my studying. So I think it's also testing whether, given strong
evidence on what would be helpful preparation, you choose to do that
preparation. (Obligatory Paul Graham essay reference - "If college applicants
realized how quick and impersonal most selection processes are, they'd make
more effort to sell themselves" [0])

[0] Two Kinds of Judgment <http://www.paulgraham.com/judgement.html>

~~~
vidarh
A problem with that is that they assume they're the most attractive option.
They might be to some, but I think that pool of people is dwindling. It took
five attempts from different Google recruiters before one managed to "sell"
Google sufficiently for me to be willing to do a phone interview.

Then I was given interview questions of a type that nobody else would give for
a position at the level I interviewed for, and I quickly lost interest again
as to me the interview questions advertised a lack of understanding of the
skills required for the position, or a fundamental disagreement about how
technical management should be carried out.

And pleading from the recruiter was not sufficient to get me to do another
interview (after she got the result of the initial interview thrown out
because she agreed with my assessment). The last recruiter I dealt with was
good, but the impression I got of the interview process put me off for a long
time.

It took several months, and in that time they'd still not been able to find
anyone else. I've never, ever spent that long on a recruitment process before,
and the only reason I bothered was out of curiosity and because I wasn't
really looking. If I had been looking, I'd have found and taken another job
long before Google would've managed to get their act together.

I make an effort to sell myself for positions I badly want, but Google doesn't
seem to be interested in selling itself in the recruitment process whereas
most other companies I've interviewed for are, and they certainly isn't even
top 10 of list of places I want to work.

But every time I've dealt with Google recruiters, I've had to waste time
dragging information out of them that I'd normally expect the recruiter to be
eager to present to me.

I'm not going to make an effort if they've not gotten me to really want the
job first.

Recruiters I've spoken to all tells me the people they take out of Google are
very often disillusioned and thought it would be very different to their
actual experience, and that dislodging Google employees have gotten much
easier as it's no longer so much the pinnacle of cool places to work in the
eyes of a growing number of engineers, so I think that's going to become a
bigger and bigger issue for them when hiring.

~~~
andrewem
All fair points. I had one friend who interviewed with about 13 different
people at Google before getting an offer, which was totally absurd, and a lot
of their friends got a negative impression based on that (soon after they set
a cap of about 8 total interviews per candidate). So I can imagine having
their interview process do other bad things, like not being forthcoming with
information or trying to sell the job.

------
ewanmcteagle
I would submit that the biggest reason for this view and the reason many
people hold it is that 37signals as well as most of the programming jobs are
CRUD jobs where non-algorithmic skills are probably more important than
algorithmic skills and IQ is also less important. Brainteasers are an IQ test
where you are not concerned about false negatives.

~~~
wenbert
I agree with CRUD jobs. And I think in reality most programmer jobs are like
that. Unless you get to the point where you design something (ex: a system)
and then implement it.

------
nickolai
I'd feel uneasy if asked to show some of my code.

Obviously i cannot show something i'm currently working on, and everytime I
look back at some code I wrote six moths ago I find myself wondering "did I
_really_ write this piece of crap?!". And it's always been this way - at least
since i've been six months into programming. I hope it is a good sign.

~~~
ssharp
It's a great sign as long as code you wrote 6 months ago is better than code
you wrote 12 months ago--it means you're getting better and learning.

I think as long as you're applying for jobs you feel you're appropriately
skilled for, you shouldn't get too self-conscience about your code. Like most
personal things, it probably looks worse to you than it does to someone else.
The fact that you are self-conscience about it, likely means you have enough
awareness to prevent you from becoming overconfident in your skills.

------
krschultz
I knew an engineer who used the 'why are manhole covers round' question in
every interview. He was always shocked at how 'quickly' the interviewee
figured out the 'answer' of "so it doesn't fall down the hole". These new kids
- they're smart. As far as I know, he never figured out that 90% of people
have already heard that one.

~~~
jacques_chester
My answer to the question was that a covers follow the shape of the hole
underneath; the hole underneath is round because they are generally cut with
rotating tools that create a circular shape. Cutting square holes is more
expensive.

------
john_flintstone
Read the comments, and I've got to say: I'm not one of you guys. I don't write
code, I build products (otherwise known as software), that human beings use.

I looked up Pascal's triangle in Wikipedia, so now I know what it is, and I
could write the code tonight to handle the shit out of it, but I'd probably
mess it up on a white board when commanded to perform like a monkey in front
of guys in T-shirts.

I own and run a software company that I started myself. I built the first 5
successful products (not code, products - there's a difference [kind of a big
one too]) myself, and I don't give a crap about Pascal's triangle. I'd
probably fail every coding test on the planet, but when it comes to delivering
finished, polished, working products, that 1000s of human beings will actually
use without picking up a phone and calling customer support, I've got that
covered.

Stay classy with your 'code' guys.

------
bdg
I was rejected in a recent interview (2 months later I'm waiting to 'hear
back' means rejection) where I wasn't competent at writing algorithms to do
various tasks such as implement an insertion sort based on this image:
[http://en.wikipedia.org/wiki/File:Insertion_sort_animation.g...](http://en.wikipedia.org/wiki/File:Insertion_sort_animation.gif)

My confusion was the job was looking for a generalist who had experience with
lots of different technologies. Specifically, working with node.js and
socket.io polling methods, API design, working with some python framework and
backbone.js.

I'm honestly not sure what the lesson there was. Am I a bad developer? Do I
navigate interviews poorly? Does my breath stink?

~~~
brown9-2
Was the image the only description of the insertion sort algorithm you were
given? If so, sounds like a pretty flawed interview process.

~~~
jerfelix
Not sure I agree.

I wouldn't have been able to tell you what an insertion sort is, by name. But
by looking at that image, I can describe it pretty well:

Have an outer loop that counts from the first position in the list to the last
position (call it n...). At each step in the loop, make sure that the first n
items are sorted. This probably implies an inner loop that finds where, within
the first n-1 items to insert the nth item. And given that with only one item,
it's already sorted, we can probably start with n on the second item.

Yeah, I think I could code that, given the image.

------
johngalt
I'm convinced that interviews are a complete crapshoot for both sides. My
first job interview went like this.

1\. 45mins late for the interview.

2\. Quizzed a chain of brilliant engineers. Couldn't answer a singe technical
question they asked. Lots of "Sorry my experience was more with X, not Y".

3\. Flatly said "You guys are looking for someone else, I don't have the
skills you're asking for".

->Hired because I was honest. Learned quick. Promoted every year.

~~~
dspeyer
Awesome story.

Who was the employer?

------
absconditus
Unfortunately a large number of us work for employers with strict IP
agreements and we cannot just show a potential employer some code or work on a
small project for them so that they may "try [us] out". Even working on open
source projects can be tricky.

------
motoford
I haven't had to really be interviewed in 15yrs, but the last one I had was
probably the best. I walked into the office of the guy who would be my
manager, he introduced himself then immediately turned to the whiteboard and
started writing and drawing while saying "This is what I'm about to start
working on, we need to figure out how to do it.". We spent the next 3 hours
working out a solution together.

I ended up not accepting their offer, mainly because of the things I learned
in the interview. But I still feel like that was a good way for both sides to
evaluate each other.

------
j_col
I thought they hire them based on their choice of personal computers (sorry,
couldn't resist)?

<http://david.heinemeierhansson.com/arc/000433.html>

~~~
Drbble
I think that article was pre-Win7, so it isn't really relevant anymore, except
maybe in a specifically Ruby niche (which is fine for 37signals).

I can't say for sure because the post doesn't have date (day of year but no
year). I wouldn't argue smart-aleck technical shibboleths with someone who
can't display a date unambiguously without overflow rollover bugs.

~~~
j_col
Here is Jeff Atwood's response to that post dated Feb 2008, which gives you
some idea about the date of the DHH post:

<http://www.codinghorror.com/blog/2008/02/douchebaggery.html>

~~~
robertp
Turns out David was right, go to any open source focused conference and you
will see everyone on an Apple computer.

------
overgryphon
Puzzles and brainteasers seem rather off-putting to me in an interview
environment. It's like the interviewer is saying, "I can solve this, are you
as smart as I am?", which isn't the type of relationship I want with
coworkers. I prefer to steer interviews into a more collaborative
conversation, to see if I would like working with the person.

Anyone who expects me to spend 3 or more hours or so on some take home test
just to interview with them, or thinks that the only people worth hiring are
those that "the first thing you want to do when you wake up in the morning is
code something" aren't people I want to work for. I love programming, but at
the end of the work day I'm going to go home and engage in other aspects of
life. I wouldn't want a boss that had different expectations.

------
rays
I saw outright that I'm not an algo guy, you already have those types of
people -- namely the guy interviewing me. But they dont know how to scale
systems or know overall product. If they want to look at code even I call bad,
I can point them at my github account :)

~~~
jebblue
I love the honesty and refreshing look your post delivers. It hadn't occurred
to me until I read it that maybe there is a programmer personality type that
actually likes being on display, coding on a whiteboard capable of doing
recursive algorithms in 2 minutes flat. Thanks.

------
peacemaker
I made a post on HN a couple of weeks ago basically saying the same thing. It
can be rather frustrating to be asked questions with seemingly no relevance to
the actual position.

I understand what they're going for though - the companies that do it
correctly are looking at the candidates thought process through a challenging
situation.

While I think it's important to figure this out, I believe it's more important
to know how well the candidate can do the actual job, that of sitting down and
writing quality software. Brainteasers and writing modified sorting algorithms
won't get this (unless the company is in the sort algorithm business I
suppose) but having the candidate write code which could actually be used in
the job would.

------
neebz
I remember reading Heroku does similar. Using applicants in a project works
well.

I have always tried to push my potential employers to use me on a small real
life project. I know it's time consuming (and you end up working for em for
free) but it works both ways. I get to know what kind of work environment they
have and they get to know me in a real deal. I find it much more comfortable
because the puzzles always seem to be a hit and go. The solution may click or
it may not.

I know it's a start but if this catches in the industry I expect employers to
even pay for your 30-40 hours interview.

~~~
mechanical_fish
_I expect employers to even pay for your 30-40 hours interview._

It's great if the new employer pays you. But what about your current employer,
the one at which you've just had to spend half your annual allotment of
vacation days in order to audition for a new job that you might not get?

Auditioning is definitely the best way to, well, audition people. But it puts
a constraint on your candidate pool, and depending on your company and your
industry that constraint can be a problem. In 37signals' situation this
doesn't matter - they're built entirely on open-source tools, they don't
require relocation, they can hire freelancers from a pool of freelancers that
_they themselves created and fostered and routinely market to_ , they are so
famous and impressive that even in a tight labor market people will line up to
try and impress them, et cetera. But there are many companies that aren't
37signals, though the 37signals evangelism team is certainly trying to change
that. ;)

------
kaitnieks
I don't understand why people are forced to write on whiteboard. Some get very
stressed and lose their ability to think, some can't multitask (paint on
whiteboard, think and talk to the interviewer at the same time). Just give
them the task, give them a computer, give them 30 minutes of privacy and then
let them explain the complete, working and tested solution - you get better
results as this situation is more like regular coder's daily routine.

------
jhack
There's nothing more infuriating and insulting to me than being asked to write
some lame string sorting function on a pad of paper during an interview. I've
been burned more than once because of pop quizzes like these before... lots of
missed opportunities because some HR person thinks that writing dinky code no
one actually uses in a real project is more important than my portfolio of
projects that I've designed, created, and maintained.

------
lox
Been considering the following hiring technique, interested in
thoughts/opinions:

We accept applicants of any experience level. To apply, find an open source
project on github, get a patch accepted to it. It has to be of useful scope
with a unit test, object-oriented code only. Send us the diff, a brief work
history, a reference and why you want to work for us.

If we like it, come work with us for a day, paid at market rates, and we can
both assess whether it's a good fit.

------
laconian
Google doesn't ask these questions! It's a fiction that is probably
perpetuated because the topic is SEO linkbait.

Again: Google doesn't ask these questions!

------
keithnoizu
I'm always amused by interview questions.

    
    
       Primarily because I do terribly at white board problems thanks to disgraphia(terrible handwritting), and secondily because while the questions might give some indicator of whether or not you can code hello world, or a binary tree depth first search algorithem they do a terrible job of testing your actual domain knowledge. 
    
      For instance, when interviewing for a SDET position i've never been asked a single question on implementing a DI framework, designing custom asserts, designing a test object/mock object, etc. 
    
      When interviewing for a web development position for high volume sites i've never been asked about cache invalidation, reduccing http requests, designing an ajax event system to reduce unnecsesary calls. 
    
      Etc. 
    
      I have been asked a bunch of IQ tests masquerading as mid-low level programming questions, and random trivia questions on language features.

~~~
motoford
I'm going to have to remember that "disgraphia" word. It's a lot easier to be
excused for something that sounds like a medical condition, rather than just
saying my handwriting sucks.

~~~
keithnoizu
Also helps if you were diagnosed with Dyslexia & Disgraphia in middle school
and have a paper trial. ;)

------
viveksec
Puzzle solving ability is a reliable indicator only if the candidate hasnt
specifically prepared for them. Many Indian IT companies in the mid-late 90's
conducted exams with difficult programming puzzles in them. This was great for
a while, but soon those who wrote these exams told everyone else and the
puzzle pool dried up. Future groups scored really well due to being better
prepared against a known pool of puzzles. These days most IT companies have
moved on to SAT/GRE style analytical problems.

Imagine your luck if you are at a Google interview and already know the Pascal
triangle. You can just put up an act pretending to analyze various aspects
before unveiling your grand solution.

If companies are merely using analytical ability as a filter, a SAT/GRE style
exam will do better because the problem pool is much larger making it less
vulnerable to preparation.

------
dutchrapley
We can all agree that there are different kinds of programming. There are some
really hard problems out there that need to be solved. There's also the kind
of programming that's a matter of getting users to interact with data. When it
comes to web application development (my background), I think puzzles aren't
the right way to approach the interview. To be honest, it shouldn't be the job
of the company to find out what the interviewee does and doesn't know. It's
the duty of the applicant to control the interview, come in with their laptop
blazing, and start walking through building a web application from scratch or
to show off something they are building, talk about the decisions that were
made through the process and why.

------
trustfundbaby
I've been in a couple of these interviews before and I've also had to try to
interview programmers I wanted to work on projects for me ... an interesting
approach that I haven't had the opportunity to test exhaustively is to start
out like the computer SAT exams.

Let me explain.

On the SATs, they ask you a simple question, and if you get that right, they
step up the difficulty and on an on until they discover you're a genius. If
you get the answer wrong, they ask you another question of similar difficulty
to the last question you got right and on and on.

Now with regards to interviewing, I like to start from the basics ... I'll
start out with simple questions about languages they've used, so that they're
comfortable. Syntax isn't _majorly_ important to me, you can google it ...
however I take note of how comfortable the interviewee is with the syntax of
the languages they're using (especially if its one we use). So while it won't
count against them, it can count in their favor.

From there I'll ask them more difficult questions. How would you implement
this one thing? We have this one problem we're dealing with, how would you
solve it? If they can't do it, I'll try another and another, giving them
multiple chances to show that they can hang. All the while I'm poking and
prodding to find out how they think about stuff ... and what makes them tick.

And then I'll go to the really hard computer sciencey stuff (which bores me to
tears to be totally honest. sorry.) ... just to see how proficient they are
with it. Obviously a person who eats algorithms and datastructures for dinner
in addition to acing all the other stuff is someone I'd love to have if they
work well with us on a small project we'll have them do.

I do the same thing with programs they've written, side projects they've done
etc. To summarize, I'm just trying to find out where they are on a continuous
scale from "total newb" to "could write os x from scratch if they needed to"

Basically, I think brain teasers CAN be useful, but I think we should use them
carefully. Value the way someone thinks over just them just getting answers
correct.

my 2c

------
xianshou
I personally adore riddles and algorithmic puzzles, but I too have had to
confront the reality that they simply don't provide much information about the
candidate's actual coding talent. Instead, where I'm working, we combine a few
different strategies to gauge an applicant's overall potential:

1\. Nobody gets an onsite interview without solving a coding problem first.
These are designed to take a couple of hours for a competent programmer. For
each applicant, we send the challenge most functionally relevant to the
position they're interviewing for. Despite the negative comments elsewhere
about "take-home" problems, I think they expose a lot of information about a
person's coding style and thought process that whiteboard code does not. For
instance, we recently received a solution to a simple graph-related question
that was technically correct and usually fast, but contained at least five
nested layers of loops and took forever on a certain class of edge cases. The
candidate obviously understood the problem, but seemed oblivious to the
fragility of his solution. You can't grade for style or really pound the edge
cases in an onsite interview, because candidates usually have their hands full
enough finding any solution.

2\. At least two of the interviewers spend the majority of the time on big-
picture, conceptual, and software design questions.

3\. I split my interviews between puzzles (the 17-minute bridge, the 200m
building with 150m of rope, etc) and algorithmic questions. However, I don't
care about whether the candidate answers my riddles correctly or not; I'm only
observing behavior and process: in other words, whether they remain calm
despite uncertainty, and whether they ask relevant questions that reveal more
information about the problem and make observations that narrow the possible
space of solutions. (For example, in the bridge question, realizing that the
slowest people should cross together is as good as a solution to me.) I do
care about their responses to the algorithm questions (e.g., find a subrange
of an array that sums to 0), but mainly to determine whether they have a
reasonable idea of which data structures to use, and how effectively they
detect problems with the rough solutions that go up on the board at first.

------
golgo13
Every Time i see these posts on here, I get a sad face. I am a SQL Server DBA
and most of these puzzles don't translate well to SQL. That doesn't stop me
from trying!

~~~
absconditus
Take a look at Joe Celko's SQL Puzzles and Answers.

------
sdizdar
I really don't know what s right way to interview, but I do know that you are
actually hiring a team not set of individuals. You need to have balanced team
(means to be smarter than you on each of the important fronts) to deliver a
successful product.

So I guess it might be ok to hire somebody based on brainteaser if you need a
sharp, quick thinking person which will be a great asset during brainstorming
sessions. I don't know.

------
ig1
It's worth noting that 37signals believes in hiring by requiring candidates
work on a 20-40 hour project before making the hiring decision. There approach
would be completely unacceptable to a large number of developers, it's only
because of their strong reputation and remote hiring that people are even
willing to consider it.

------
vinodkd
I once interviewed for an architect position for one of the "web scale"
companies and it was a algo fest all through. All the while I remembered the
dev manager (whom I'd have reported to) saying that they were looking for an
architect for a new project and the key challenge was learning all the
existing components they have, picking the right ones for the job (or
proposing to build new ones as required) and then - most importantly - getting
it sold to a bevy of other architects and stakeholders.

This was right up my alley as I currently lead 3 teams and we regularly have
to deal with architecture and design issues on a large application with tons
of other teams involved; and I was therefore pretty stoked about the job
-except that the algorithm wall stood between me and it. Not having ever done
an algorithms course formally(although I read the books out of interest) I
went down in flames; at least I think I did - because I'm yet to hear from
them. Wanting closure I even contacted them and asked point blank if I was
rejected, but I was assured that I WASNT rejected and that they'd "get back to
me".

The process itself was largely the standard "how would you write <algo x> in
java", which I did not do very well. One interviewer asked me to implement a
Queue using two stacks and maintained that "there's a better way" despite my
multiple attempts. I finally asked him if this was an actual real world
problem that they faced in their company to which his answer was a glib "No, I
was just testing your problem solving ability"

To be fair, however, some of the other interviewers were better.

One actually glanced through my resume, picked on one of my open source
projects and grilled me on what I'd do given a hypothetical implementation
constraint on a particular piece of the architecture. In hind sight it looked
like he was trying to get at my algorithmic problem solving ability while
connecting to something I'd built; but at that moment I was so hung up on the
_problem_ that that my software solved than how the software itself was built
that I ended up frustrating him with my pre-thought solutions. He kept asking
for "from scratch" solutions to which most of my responses were "but it
already exists in X, so I'll just use it".

The best interviewer was a senior guy who was totally OK with my lack of algo
knowledge; and very quickly switched to designing a solution to a web scale
problem. He asked me for one solution, picked out the flaws in the solution
using Big-O notation and asked me to refine it. This went on until we reached
the asymptote that they had actually implemented in production. Along the way,
he gave me more of the solution than I did to him, but I ended up with a
better appreciation of the whys and wherefores of the problem; and more
importantly, problems of the kind.

My general take away was that companies that use algorithm tests make the
following assumption:

    
    
            algorithm solving ability => general problem solving ability
    

...and this is obviously true for companies where scale is a fundamental
issue, ie , where the scale of the problem has not been addressed by _any
existing solution_.

However, the problem with using this definition to measure candidates are:

1\. Your company problems may be solvable using existing solutions. Then the
real litmus test for your candidates is how well he can assess and assemble
existing components/frameworks/libraries to realize the solution. This is
where attitude matters more than anything else; as does prior work and
interest in such work. "What was the best project in your career and why"
might be the best question to ask such a candidate.

2\. Your interviewers may never connected the LHS of this equation to the RHS.
This is where, IMO, questions like "queue using 2 stacks" or "manhole covers
being round" come. The interview becomes cargo-cult worship at the
Algorithm/Puzzle altar.

So my suggestion to the companies that still have algorithm-based interview
processes is: when presenting such a problem in an interview at least tie it
to a real-world scenario that you had to deal with in your work place so its
not a test of how much practice I've had with Cormen and Skiena (yup read the
yegge post).

Speaking of which: Am I the only one who finds all algorithm books as being
too prescriptive? They all seem to dwell on the laundry list of data
structures followed by the laundry list of algorithms; leaving the reader to
figure out the "what to use for what" and the "why" all by himself? I'd pay
for an algorithms book in the style of Polya's "How to solve it". Skiena's war
stories are an attempt in that direction, but a full frontal attack on A&DS
would be great.

Aftermath: I left that interview with two takeaways:

1) I had to shore up my knowledge of algorithms and data structures simply
because it was the "basic tools of the trade" and there _ARE_ still problems
around that need you to dig that deep in the industry (thanks to the final
interviewer)

2) Large organizations have "how to assemble software using existing pieces"
issues in their software development, not "build from scratch" issues; but
developer(turned interviewer) hubris will always skew towards testing for the
ability to build (proxied by algorithm skills).

Either way - 37 signals notwithstanding - it makes sense to catch up on your
algo skills :)

Oh, and please, please, please lets do away with the Hollywood principle when
it comes to developer job rejections. If I'm rejected, tell me. I can take it.

------
zyzzy
There are so many different views here about interviewing that it can get
quite dizzying.

1.) 37 signals view is that there is no point in asking programming puzzles or
brain teasers; this goes for white board questions as well. Their view is
that, it's just better to go over real code the person has written, and if
they seem like a good fit, then why not try them out.

2.) Cletus's view is that you should ask a "simple white-boarding problem",
that would weed out in theory weed out the "bad programmers", with a certain
error margin.

3.) Dmbaggett's views is that solid programming puzzles (one's that are
usually are take home, or one's that you do at home) are good for attracting
programmers that will fit (the role of a computer scientist at ITA). However
cheap whiteboard problem's aren't necessarily the best for finding great
programmers.

4.) edw519 commented a while back, that written code sample done at the time
of interview is best, as it provides rooms for some discussion material that
will gauge if the interviewee is a great fit for the role.

[1] <http://news.ycombinator.com/item?id=834513>

4.)Another article on hacker news, "I Won't Hire You", got pretty much down
voted for it's view, that you have to be greater than great to work at Golem
Technologies.

[2] <https://news.ycombinator.com/item?id=3428567>

5.)There was one other article on hacker news; the article mentioned how in
the end programming interviews puzzle, didn't matter and the programmers that
they hired and did the best, ended up not being the most productive
programmers.

6.)An article about hacking the current interview system. Apparently the top
commentator timr, really thought that you should grill the interviewee to find
out if they are hacking the system.

<http://news.ycombinator.com/item?id=3370341>

The list can go on and on.

So in the end what is the best algorithm for interviewing a programmer? Are
you hiring out of fear, that you need X to do a Y job, but you don't want X to
destroy your company? Are you hiring to find a rock star programmer, who can
take your company to the next level? Are you hiring a programmer to do a
specific well defined job? Are you hiring to fill your ego, that you have the
power over someone?

It seems very hostile to get a programming job. It seems that some
interviewers want you to come in and contribute from day one, which I have
never seen quite happen. Why are some interviewers so harsh and dicks with the
interviewees?

If a programmer fails your interview, does it mean they are truly stupid?

Is your interview really a true gauge of IQ? Does your ego play into the
interviewing process as one HN commenter wrote

"The biggest problem with anything like this is the idea that 'here is some
test of inherent intelligence - I am far better than you so you are inherently
unable to do this thing' which is just the biggest barrier to actually trying
to do something - if you think you inherently suck or at least are simply
mediocre, your motivation to do that thing is severely reduced."

The general feeling I get is that as interviewers, we are trying to put down
and ridicule, interviewees for whatever reason did not pass our test. Maybe in
accordance with our company's guidelines or our own ego's.

Instead of seeing the glass half full with interviewee's potential we are
seeing it half empty, as their ability will never be good enough.

I think with determination and practice anyone can be "great" programmer. It's
sad to see many interviewer's don't realize this, cause "they don't have
time."

------
startupcto
Why isn't anyone testing candidates' ability on reading and working with
existing code? I do that all the time with interviewees.

Why I do that? The obvious being the potential hire is required to work within
a large code base (often code not written by yourself) and having the ability
to read and understand quickly what a piece of code does is a great skill that
not many great coders have.

------
dsolomon
I thought it was because 37signals didn't want their brainteasers splattered
across reddit & ycombinator.

Today I Learned ....

