
One hiring filter that works - yummyfajitas
http://www.chrisstucchio.com/blog/2012/one_hiring_filter_that_works.html
======
wheels
I call BS on this one. I'd even say that this will filter out the top level of
folks (†).

If you give me a synthetic puzzle to solve I wouldn't bother applying to your
organization. It's much better to craft a problem to the specific domain that
your company is dealing with. Then you can test for both domain expertise and
problem solving ability in one fell swoop with the extra bonus of it not
seeming pointless.

† The important addendum on this is that you're not going to have the top
level of folks sending out blind applications unless that person is in the
process of making a lateral jump into another sub-field. And even then
probably not. But if you give a top applicant a synthetic puzzle after your
recruiting contacts them, expect a large portion of them to thumb their nose
at you. Similar note on top folks, though this is mostly pedantry -- the top
OSS projects aren't on GitHub and similar. The largest and most prestigious of
the OSS projects host their repos themselves.

~~~
yummyfajitas
To clarify, I wasn't discussing recruiting Linus or Guido.

I'm actually more likely to apply if there is a puzzle. I've sent a few
resumes out blindly, it usually goes into the black hole. If there is a
puzzle, I figure my odds of getting a response (assuming I get it right) go
way up.

Of course, if a correct solution results in a recruiter replying and asking me
to submit to resumator, I'm probably ignoring your company in the future.
(Yes, this happened to me once.)

~~~
wheels
"Not trying to hire Linus" is a bit of a straw-man. There are a lot of folks
that fall between "Linus" and "have to actively look for a job". Probably tens
of thousands of people in the programming field.

The important point that I'm getting at, however, is that the dynamics of
hiring shift dramatically as you move up the skill / connectedness scale. As
you move towards the higher percentages potential employees are not competing
for employers, but employers are competing for hires. And the dynamics of
hiring are dramatically different at that point. If you're designing
heuristics for hiring, you don't want to create a low-pass filter, and I
believe that synthetic puzzles are exactly that.

The point of raganwald's post was on the surprisingly high cost of false-
negatives. What I'm postulating is that your suggestion is just yet another
way to reduce the false-positives rather than a way to reduce the false-
negatives.

~~~
tesseractive
If it increased the number of overall respondents as well as the number of
respondents who were above a certain bar (passed a phone screen), then even if
it created other false negatives, it still seems to have helped.

Good developers don't just go around submitting their resumes to every job in
existence, and not filtering out false negatives was insufficient in and of
itself to produce a good applicant pool. If doing something that produces
false negatives (which, yes, are undesirable) nevertheless convinces enough
developers to consider your company who never would have considered it
otherwise, then it seems like the advantages have outweighed the
disadvantages.

Obviously, an even better scenario would be to produce the same level of
developer interest without also producing any false negatives. Perhaps by
investing sufficient resources in networking or in making his company famous
to other developers, he could accomplish this. This would still be a
possibility to explore in the future.

But just because his tactic has not produced the best possible outcome does
not mean it has not produced a positive one.

------
edw519
How to hire C players: Do what everyone else, including recruiters and HR
departments, are doing.

How to hire B players: Implement serious recruiting processes and plan to
spend a lot of time on it. Get creative with hacks like OP's.

How to hire A players: Identify them through never ending networking,
research, and keeping your finger on the pulse of the industry. You'll never
find them the same way you find B and C players.

~~~
bbwharris
How to hire A players: Offer way above current market value.

How to hire B players: Offer market value.

How to hire C players: Offer below market value.

You do two things when you offer more money.

1) You are more careful about who you hire. 2) You actually attract good
people who would make a move for that kind of money.

I would love to hear an inverse argument against this.

~~~
eric-hu
The finance industry tends to offer way more for people with a strong
technical background to do quantitative analysis work.

To use real world examples, no amount of money you offer Salman Khan now would
pull him away from his non-profit Khan academy to go back to the hedge fund
industry.

If you're going to point to your step 2, then that sounds like a tautology.
"Attract good people who would make a move for more money. Offer them more
money"

~~~
bbwharris
That's actually an excellent point. Uber personal success would completely
remove the desire to make a move for money.

Isn't that what acquihiring is about though? Purchase the talent that wont
make the move for money.

------
tokenadult
The HUGE peer-reviewed professional literature on industrial and
organizational psychology has many articles devoted to business hiring
practices. There are many kinds of hiring screens, such as resume reviews for
job experience, telephone interviews, in-person interviews, checks for
academic credentials, and so on. There is much published study research on how
job applicants perform after they are hired in a wide variety of occupations.

The overall summary of the industrial psychology research in reliable
secondary sources is that two kinds of job screening procedures work
reasonably well (but still below the .50 level, standing alone). One is a
work-sample test, where the applicant does an actual task or group of tasks
like what the applicant will do on the job if hired. Another is a general
cognitive ability test (an IQ-like test, such as the Wonderlic personnel
screening test). Each of these kinds of tests has about the same validity in
screening applicants for jobs. Neither is perfect (both operate at about .4x
level in validation studies), but both are better than anything else that has
been tested in rigorous research, across a wide variety of occupations. So if
you are hiring for your company, it's a good idea to think about how to build
a work-sample test into all of your hiring processes.

For legal reasons in the United States (the same consideration does not apply
in other countries), it is difficult to give job applicants a straight-up IQ
test (as was commonplace in my parents' generation) as a routine part of a
hiring process. The Griggs v. Duke Power, 401 U.S. 424 (1971) case in the
United States Supreme Court

[http://scholar.google.com/scholar_case?case=8655598674229196...](http://scholar.google.com/scholar_case?case=8655598674229196978&q=Griggs+Duke+Power&hl=en&as_sdt=2,24)

held that cognitive ability tests used in hiring that could have a "disparate
impact" on applicants of some protected classes must "bear a demonstrable
relationship to successful performance of the jobs for which it was used." In
other words, a company that wants to use a test like the Wonderlic, or like
the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring
process had best conduct a specific validation study of the test related to
performance on the job in question. Some companies do the validation study,
and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring
and hope that no one sues (which is not what I would advise any company).
Companies outside the United States are regulated by different laws.

~~~
john_horton
This is interesting - do you have a good academic reference for that
summarizes this literature? Maybe like a handbook chapter?

~~~
keenans
I think the the study he is referencing is the 1998 paper by Schmidt & Hunter.
We studied it in business school. I couldn't find an actual copy of the paper
but these two articles have abstracts of the findings:
[http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=1...](http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=1998-10661-006sample)
[http://www.onetest.com.au/awms/Upload/documents/whitepapers/...](http://www.onetest.com.au/awms/Upload/documents/whitepapers/Schmidt%20and%20Hunter%20Summary%20-%20Onetest.pdf)

Schmidt, F.L. & Hunter, J.E. (1998) The validity and utility of selection
methods in personnel psychology: Practical and theoretical implications of 85
years of research findings,” Psychological Bulletin,

~~~
tokenadult
Thanks. I had read the conclusion I mentioned in monographs about psychology I
was reading for other research interests, and I'm glad you cited a key review
article. I found the article's full text

[http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...](http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%20Validity%20and%20Utility%20Psychological%20Bulletin.pdf)

via Google Scholar.

------
michael_nielsen
I've never been involved in hiring at a startup, but for 8 years I was
involved in hiring as a prof in academia.

In that context, there is a peculiar and consistent gap between the best
hiring strategy and the strategies institutions actually use.

For hiring faculty whose main focus is research, the best strategy is targeted
attempts to hire superstars -- the research equivalent of a Guido van Rossum,
say, or a Linus Torvalds, or someone near that level -- or early-career people
where you have very good reason to be confident they will be future
superstars.

It sounds obvious to say "go after superstars". But what's strange is that in
any given field only a very short list of institutions actually do it. At most
academic institutions it's surprisingly difficult to convince the institution
to go after superstars.

I think the reason for this aversion is instinctive. If you're hiring it's
much more comfortable to go the conventional route, advertising a position,
interviewing a short-list and so on. This produces a situation where most of
the interviewees want the job pretty badly. This gives the hiring committee a
lot of power, which is a comfortable position to be in.

Unfortunately, it also produces lower-quality candidates than identifying the
best people and going after them directly. A superstar will be interviewing
you more than you are interviewing them. You'll need to get creative in
convincing them that you have something to offer, something they can't find
anywhere else. This is much less comfortable.

A useful heuristic: if most of your job offers are being accepted, your
standards are way too low.

------
pgroves
So with all the how to hire threads lately, may I make a suggestion:

Focus on how good your company is on running an engineering department.
Especially if you're a startup.

Do you have a VP of engineering who knows what the hell they're doing? Do you
insulate the devs from sales people making ridiculous promises? Is your code
base in good shape so someone can come in and make a difference without having
to learn the complete spaghetti stack? Do you have a well designed deployment
procedure that has dev, test, and production environments?

How do you approach scheduling? I'd say this is the number one issue that
leads to bad experiences for developers, so tell us up front that you have
this under control. Do I ever have to hear a business guy say "I don't see why
that would take more than a few minutes..."

In other words, prove that every problem isn't solved with "developers work
more hours."

Say something that at least implies you know that these issues _exist_. They
are much more important than ping pong tables and calling each other ninjas. I
bet you'll find plenty of pro developers who are sick of their current
employer if you do.

Imagine if developers actually learned something about running a software
company just from your job ad itself. That would be amazing. I would point to
JoelOnSoftware as proof it would work.

------
AndrewDucker
Providing you're testing the actual skills that you want, this works well.

But if you're throwing out maths puzzles when you actually want people to
write Java business logic then I can see you ending up with the wrong people.

~~~
yummyfajitas
The math puzzle we used required only prime numbers and modular arithmetic.
Any competent programmer could wikipedia it and solve in 20-30 minutes (and
many did).

~~~
bconway
So it was like FizzBuzz, then?

------
guard-of-terra
This reminds me of a stupid NH job posting where they said they want, and then
there were two hashes: Googling for those hashes revealed in the first snippet
first standing for "good", second for "developers".

The message was that anybody who has the common sense to google an unknown
string is qualified duh?

The first puzzle job ad is nice, but it quickly wears off, and would start to
bring bored "oh, not again" feelings. And many people just aren't good at
making puzzles.

It's the same as with paint-drawn NFS ad.

------
shanemhansen
I used to think I knew something about hiring. I worked in a fairly large
engineering organization and was involved in all our hiring. I did college
recruiting, intern programs, phone screens and interviews. I mention that for
background when I say: you can't reliably make good hiring decisions on the
basis of a couple hours of your/the interviewees time. It just doesn't work.
I'm personally a fan of hiring within your network (for engineering
positions). You can't beat a genuine recommendation from someone who's spent
months working shoulder-to-shoulder with the candidate.

So please, when you hire people, have some humility. Know that your decisions
are affecting someone's life in a major way, and you're probably making the
wrong choice anyways, just do the best you can.

------
LaaT
I wonder how much of this is due to novelty factor. I doubt that one could
reach such good numbers, if asking for solving puzzles becomes a common
practice.

~~~
K2h
There is something to be said for the novelty probably getting more interest,
but I think that giving puzzles around the real work the hire will perform
tells them as much about your company and the problems they can help you
solve, as it tells you about the candidates ability to do so. A win/win as
they say.

I already ask about 10 technical questions in each of the interviews I
conduct, and having a problem online for each candidate to hack on would be
really useful.

Now that I think about it, I may even consider using something like this to
drive internal improvement projects. Some people just love the challenge, and
the 'a' people always step up, you just have to ask.

------
mikeryan
I think this post misses one of the key takeaways from "unlucky". That post
isn't only finding good candidates. Its about casting a wide net and doing the
laborious part of filtering out good candidates solely on the basis of their
qualifications.

This seems like a good way to only get good candidates but you may also have
some very qualified people who don't want to play this game and find it
unnecessary. In some ways its the equivalent of throwing out resumes with
misspellings.

------
mattbee
Yup, this worked really well for us too - we asked candidates to submit a
20-line "binary chop" search over a given data file (well, that was the
solution, we described the problem and restrictions such that that was the
only right answer). All the interviewees were good when we did that, and it
also seems like a good test for recruitment agencies (i.e. which ones
understand what we're trying to do).

We did get one answer that was both incorrect, too slow, and massively over-
engineered. I got emailed a multi-megabyte Rails project, boilerplate, tests
for correctness and all. Bit It Didn't Solve The 20-line Problem. He was the
guy we wrote the test to avoid!

------
gruseom
What if companies just open-sourced their code? Then a programmer could look
at the very system they would be paid to work on and a company could look at
examples of real work. No more puzzles and arbitrary tasks. And if the system
is of any interest and a community develops, that's a source of new
programmers. The company gets access to a talent pool and hackers get to know
if it's work they want to do.

The obvious downside is that competitors get your code too. But is there any
credible analysis or evidence of how bad a risk this is? Because the risk of
not finding good programmers is pretty bad.

I suspect that people are still way overestimating the value of source code in
and of itself, relative to the value of the team that produces it. If so,
that's a market inefficiency and open-source might provide an edge. It would
also be more rational and satisfying from the point of view of the actual
work. Imagine where mathematics would be if mathematicians still kept their
proofs secret.

Can anyone point to examples that shed light on the effect of open-source on
hiring?

------
Sottilde
I worked for a startup that used to advertise its open positions as "Silver
Java Guru" or "Java Geek".

Needless to say, we got one awful applicant after the other. Most could
Fizzbuzz. Some could not. And others must have cheated their way through the
phone interview - when you got them in person, it all fell apart.

One thing I noticed from the process is that little things like spelling
mistakes, incorrect formatting, etc., in a resume was very telling. Without
fail, the kind of person who had typos in his resume was the kind that failed
our programming tests, miserably.

------
csomar
If you are looking for a good candidate, and ready to spend the time doing it,
why not look around the developers who have done something in the field you
are active on.

For example, if you are looking for a Front-End guy, why not look around web
magazines, Web 2.0 products and services, Github, Dribbble and pick the guys
you like and contact them.

------
doki_pen
Part of the trouble is that the best people are well employed and busy
working. They are in high demand. They aren't going to do puzzles or beg you
for a job. You have to find some way to meet them and convince them to work
with you.

------
brandall10
This reminds of the Google billboard test from years back:

[http://haacked.com/archive/2004/07/12/google-mysterious-
bill...](http://haacked.com/archive/2004/07/12/google-mysterious-
billboard.aspx)

------
lutorm
_No discussion of rockstar ninjas, just saying exactly what we are looking
for._

Like I posted in the "who's hiring", that doesn't work for me because it
doesn't tell me whether _you are what I'm looking for_. What do you do? Why
should I care about you?

------
dxbydt
Like it or not, this particular hiring filter seems to have caught on bigtime.
The only downside to it is that there are people like myself who love to take
a stab at solving puzzles regardless of whether they actually want the job or
not. In the past 2 years, I solved online puzzles & scored interviews with the
founders of 3 yc companies ( greplin, poundpay & freshplum) & 4 giant
financial shops ( BlackRock & 2 other hedge funds I can't name/nda). In some
cases like BlackRock, the puzzles are quite involved & can take a week to
complete. Puzzles are a lot of fun....I just wish the associated job was
equally interesting.

~~~
K2h
Do you have a source for where you find these interview puzzles? I would like
to take a few and see how they are set up.

~~~
morenoh149
interviewstreet.com

------
jackfoxy
Before I look at a resume I send every applicant a coding problem with simple
instructions. It looks very simple at first, but actually requires some
thought. It is fairly real-world, but unusual. I even provide a sample test
case, and the instruction that the solution must pass all test cases. It would
take the ideal candidate less than half an hour to code and test. Acceptable
candidates less than an hour. Despite having all the time and resources they
need to solve it, 85% to 93% of the candidates do not get it. I don't even
glance at these resumes.

~~~
gergles
Have you considered that 85-93% of the candidates don't want to do an hour's
worth of unpaid work for you before you've even given them an interview?

~~~
jackfoxy
It's not 85-93% not responding, but 85-93% responding with crap that took some
effort to produce. Also if you're actually interested in the position, you're
going to be doing more than an hours _work_ (if that's what you want to call
it) in the interview process. Lastly, it would be _work_ if I were tricking
people into doing something I used as R&D or actual production end-product.
This exercise simply establishes qualification.

~~~
jarek
What is the response rate? How much contact do you have with the applicants
before sending them the problem?

~~~
jackfoxy
Lately these are contract positions, and being in a corporate environment here
I'm dealing with an agency that deals with other agencies. I still do some
direct (usually by Craigslist ad) recruiting. I only submit the screening task
to people who have already responded to the position posting. Very few people,
maybe 5-7% do not complete the task, and I assume they found another position
in the interim.

------
somewhere1
It is an interesting idea, but this may be limited in its use for small
startups that hire only 1 or 2 programmers or for someone willing to wait. My
reasons being: Only a small sliver of developers visit sites like HN. They are
mostly busy working, going home to kids and such. Getting the news out that
you are hiring, to this larger set of candidates is a real problem. Which is
where the traditional methods (including job web sites) become crucial. If you
use any job borads (like Monster), or a headhunter, it is hard to pipe a
problem through them. They are not setup for this. If you had asked me to
solve a problem just to find the email to apply to a job, I would have not
done so, just because I do not have time to do it. If I were a kid out of
college, or did not have time constraints, I would have loved to. But where I
am today, I would rather send the resume in and talk to the hiring manager.

------
SagelyGuru
Good puzzle, as opposed to a bad one, is an excellent attention getter for
good programmers. They can judge you and your intellect by your puzzle and be
attracted to work with you. Ie. your input makes the recruitment a two-way
street.

------
rheide
"we hadn’t really planned for so many good applicants so we were very slow to
respond to everyone"

So they weeded out the really bad applicants and as a result were left with so
many good applicants that they're now in the same situation as the original
article.

The key takeaway for me is not how people filter their applicants on real
criteria, it's what they do when they have a large amount of seemingly equally
skilled applicants. The original article demonstrated the 'unlucky' solution,
I'm not sure what this article demonstrates since they still got too many
applicants and they're spending time responding to all of them.

~~~
yummyfajitas
_So they weeded out the really bad applicants and as a result were left with
so many good applicants that they're now in the same situation as the original
article._

The original article talks about finding 4-5 people worth talking to out of
100 people, 50 of whom can't fizzbuz.

We were in the situation of 20 interesting people out of 40 candidates, all of
whom could fizzbuz.

If it were truly unmanageable, we'd probably have used weak Bayesian filters.
E.g., degree, rockstar ninja, etc, all the things Raganwald originally
advocated against using. I gather this is what Google and Facebook do.

------
blhack
I fully support this mostly because I absolutely love doing the puzzles :).
(So make more of them!)[1]

One of my favorites from a few years ago was when reddit was hiring. One of
the more fun aspects of it was:

git clone our public repo and tell us the most commonly used word

Another fun one was (I think? This was a few years ago):

find me a right triangle, the length of which's sides all add up to:
$some_number.

[1]: In fact, this would be an _awesome_ website if somebody hasn't made it
yet: be a "recruiter", sortof, except you just post programming/math puzzles,
we get to solve them, and then you get to match us up with employers.

~~~
kami8845
see interviewstreet.com

------
kenrikm
The only jobs that caught my eye were the ones that offered some type of
challenge to apply (Apply Via API, Etc.) If I'm going to end up talking to
someone from Human Resources that's going to ask me the "Man Hole Cover" or
"Lightbulb Question" then forget about it, I'm not interested in working in
more stuffy corporate environments.

-(void)fizzbuzz { int i = 1;int t = 3; int f = 5;int h = 101; int z = 0;
    
    
        // here we do a for loop using i and incrementing it until it's equal to z
        for (i = i; i < h; i++) 
        {
            /* we check if both i/t is equal to z and if i/f is equal to z
             if both are true we log fizzbuzz in the console */
            
            if (i % t == z && i % f == z) 
            {
                NSLog(@"FizzBuzz");
            }
            //check if i/t = z and log
            else if (i % t == z)
            {
                NSLog(@"Fizz");
            }
            //check if i/f = z and log
            else if (i % f == z)
            {
                NSLog(@"Buzz");
            }
            // if none of them are true we just act like we're learning to count
            else 
            {
                NSLog(@"%d", i);
            }
        }
    }

~~~
Steve_Baker
Every time fizzbuzz comes up, someone has to post a solution it seems, but why
does everyone write such huge fizzbuzz programs?

    
    
      for(int i=1; i <= 100; i++)
        printf((char *[]){"%d\n","Buzz\n","Fizz\n","FizzBuzz\n"}[((i%3==0)<<1)|((i%5)==0)], i);

~~~
Tyrannosaurs
Some people value readability, some compactness / efficiency.

Why they do it one way or another is an occasionally interesting follow up
question.

~~~
kenrikm
I'm in the readability camp, I'm one to comment my code even if it's something
simple and I also like to use verbose variable names. I always keep in the
back of my mind that if someone had to pick up my code without knowing how
anything was implemented they might appreciate my efforts (even if it's just a
personal project that I don't plan on anyone seeing). You can't please
everyone I'm sure for as many people that like it compact there are people
that like the whitespace.

------
stephan83
It took me more than four months after college to find my first job as an
intern. Six months later, I got offered a better job so I quit. However during
the transition I was offered some well paid freelance gigs so I didn't end up
taking the second job. After 4 years of freelancing and moving out of a big
city, I got tired of working alone so I joined a small team in a non-tech
company. Six month later, I was getting bored so I looked for work in a major
city. I found one startup that did something that sounded interesting, so I
sent them an email even though they weren't advertising opened positions. An
hour later I got a response, and two weeks later I was working for them. The
past 3 months, I got offered 2 jobs and Facebook contacted me for the second
time.

This isn't even the Silicon Valley -- I live in Europe. Sometimes I wonder if
I'm smart or not. I know I probably wouldn't be able to answer all of these
trick questions I hear about. I think I'm much better at abstracting concepts
and seeing the whole picture. I love programming, but so far I haven't found a
place where I can have as much fun as when I am creating software on my own,
free of constraints and in control of everything. When I work on my own, every
now and then I stop coding, sometimes for days, thinking, sketching, until I
know exactly what I'm going to do and can explain why in plain English.

Anyways, I think younger programmers might enjoy taking time answering these
kinds of question, but I'm the kind of person that wishes he didn't have to
sleep so he could spend more time working on his own projects, so I don't
bother. If you want me to be interested, you'd have to find a way to convince
me that I would be working in an environment that feels like I'm working on my
own project. I want to work with motivated people that care more about their
product than getting funded. I don't want to work in an environment where the
number of closed tickets is a metric for productivity.

------
esonderegger
My field (classical music) has a similar application screening problem (lots
of applicants, very few good ones). Our chosen screening process, the
audition, is imperfect for many of the same reasons given for why puzzle
solving sometimes falls short when it comes to screening for developers:

\- The ability to perform on audition day doesn't necessarily correlate with
the abilty to perform consistently in concert settings. \- Great solo players
aren't always great ensemble players. \- It is very difficult to predict how
an auditionee will mature over time.

That said, our field has yet to come up with anything better, and I think the
organization I work for has developed a particularly good screening process:

On audition day, applicants are asked to perform some excerpts from literature
our organization plays regularly, a prepared solo selection, and to do some
sight-reading. This is pretty standard stuff for the clasical music industry
and this preliminary round is done from behind a screen to help combat biases
on the part of the panel. The excerpts are important because that's closest to
the work applicants will be doing on a day to day basis if selected. The
prepared solo helps show the candidates potential to excel in solo situations.
The sight reading is crucial because it is a basic and necessary skill and
correlates very highly with the applicant's quality of musical training.

The second round (generally 5-15 applicants from the initial pool of about
100) is where our organization does things a little differently. It consists
of more excerpts and sight-reading, but this time applicants are asked to make
adjustments to how they performed the selections (a little faster or slower, a
little louder or softer, etc.). Observing how the candidates respond to these
requests shows how nimble the musicians will be in a rehearsal setting, and
also shows how well-prepared they are and how well they will take direction.

In the final round (generally 2-5 candidates), applicants are asked to perform
alongside members of our organization. This demonstrates more responsiveness,
how well the applicant listens, and how well the candidate performs in a group
setting.

With that in mind, I often wonder why those looking to hire developers don't
use a multi-stage "audition" process.

Step one could consist of some thing along the lines of "Here's a piece of
code like what you might see in our organization. There's a bug in it. Find
and fix the bug and send the revised code back to us." Step one could also
consist of a short puzzle-type problem. Step one could be 100% automatic and
would require the same amount of time for 100 applicants as it would for 1000
applicants.

Step two could be where applicants are asked to show samples of something
they've coded in the past. This could also be where they are asked to perform
a more difficult bug fix and a more difficult puzzle. In this round,
applicants are told that their code will be read for documentation quality and
use of good style.

Finally a final round could be done in-person where an applicant is asked to
solve a problem with existing members of the team. This could be done in
conjunction with one on one interview. This would help tell if the candidate
is someone the team actually would enjoy working with, and how the candidate
would be able to contribute.

I know I wouldn't mind submitting to such an application process because I
could see how each step is relevant to what the company is looking for. It
also shows that the company takes hiring seriously. Hiring is supposed to be
hard. Knowing that every one else on the team has been able to excel in those
circumstances gives me greater confidence that I'd be joining a great team.

~~~
gcorne
Based on what I have seen, the turnover rate of musicians in a symphony
orchestra is much slower than the turnover rate within software development
organizations.

~~~
esonderegger
That is a very good point. Most top-tier orchestras have tenure for their
permanent positions and it's not uncommon for musicians to stay 20-40 years.

As a result, the consequences for a bad hire are extremely high. Many
orchestras have the audition winner perform with the group for a probationary
period of a season to help guard against this. That said, I read a lot on HN
about the consequences of a bad hire for a an early team member, so although
the timeframes are different, the fear is similar.

~~~
pmjordan
The long tenure (and prestige of the organisation) probably also contribute to
the willingness of candidates to subject themselves to such a process. Getting
someone who is currently employed to sacrifice multiple days of (paid or
unpaid) holiday for your hiring process is presumptuous, particularly if
they're actively looking for a new position and thus probably going through
this with multiple companies simultaneously.

As you say, it's all about making the relevance of the process obvious to the
candidate. Also, companies should be as accomodating as possible in terms of
timing of appointments, etc.

------
ceol
Do you have a link to your encrypted application?

~~~
DugFin
Yes, this is a bit annoying. First he relates a not entirely interesting story
about his HR problem, then says he solved it by including an interesting
puzzle. OK, cool story bro, NOW WHERE'S MY PUZZLE?!?!? I'm not looking for a
job, or looking to hire, but like most Hacker News readers I like puzzles. Now
I'm angry and have to find a puzzle somewhere else.

~~~
yummyfajitas
My previous blog post (submitted here yesterday) has two.

<http://www.chrisstucchio.com/blog/2012/leaving_academia.html>

But I'd suggest that except in special cases, posts with titles like mine are
probably not worth reading if you are uninterested in hiring or finding a job.

~~~
DugFin
Well, you never can tell. When someone says they've found a novel test for
weeding out bad applicants, that sounds interesting. When the test is
described with a handwave as an interesting puzzle that the best applicants
leap at the chance of solving before reading the rest of the interview.... and
then you don't even give us the puzzle, but rather launch back into the HR
aspect... well, that just seems cruel. It's like reading a murder mystery
where the detective details how he figured out who the murderer is, and then
never tells anyone. From my authoritative position as a random anonymous
internet jerk, I assert that if you allude to a puzzle, you ought to at least
include it.

I am apparently puzzle obsessed today. Slow day at work.

------
jakobe
I wonder how many high quality applicants he would get if he just described
what the fuck the job is about.

I often look through job ads out of curiosity, and it's remarkable how generic
most of them are. Maybe they promise "high profile clients" or "working with
exciting technologies". Maybe they even promise "a friendly atmosphere". I
don't understand why anybody would answer such a job ad.

I'd be interested how many well qualified applicants they would get for a job
ad that actually described what they are looking for: "We look for someone to
work on the back end of our fizz buzz cloud service. The task will involve
factoring numbers and counting digits."

------
iamleppert
Have you ever stopped to realize that there is a class of people who like
solving problems like these that have a definite, deterministic solution but
who are utterly incapable of shipping code or a working product?

------
ArekDymalski
I wonder when people stop searching for holy grail of simple, quick, universal
and always effective hiring method. Even probation doesn't provide you with
100% accuracy of performance prediction. And everything else you'll use will
be less effective. The real challenge is to make a proper, conscious choice of
the tool taking into account the job itself (and its context) as well as
balance between accuracy and resources spent. The discussions like this remind
me discussions about which programming language is the best. Is there answer
for such question? :)

------
aangjie
Hmm. I remember telephone interviewing for this position. As one of the
applicants in the first group(i.e No puzzle just a simple job description
posted :-)). I don't remember it verbatim, but the couple of questions i
remember was (1. Why did you decide to apply?(i.e: from the post) I mentioned
that the post referred to a entropy reduction algo and implementation. 2. Do
you like HPMOR? :-P Just a reference. I don't have a strong opinion about the
whole experience.

------
jisaacstone
This is fairly true in my experience.

I find solving puzzles at codeeval much more interesting than crafting custom
cover letters, and the ease of applying means I'll more likely apply to a
company there than elsewhere.

OTOH I don't think I'd classify myself as an "A" player, at least not yet. The
more I learn the more I find I need to learn &tc.

(haven't heard back for anyone yet though. Probably a bunch of good applicants
inundating their hiring team as well.)

------
kylemaxwell
I suspect this works even better in areas outside of traditional development,
like forensics, crypto, or data science. Hackers with specific skill sets,
especially where those skill sets usually get applied to private data sets,
often love the opportunity to mess around with alternate approaches on public
data sets. And if it means an opportunity to show off, even better.

------
ionforce
If you're simultaneously someone who regularly reads Hacker News AND cannot
FizzBuzz, shouldn't you pretty much kill yourself?

~~~
ArekDymalski
No, you should send yourself to the museum as an example of extremely rare
specimen ;) Or learn yourself on codeacademy.com

------
silentscope
I like this because it isn't a "gotcha" trick. That's the way this method
backfires (in my opinion).

------
cploonker
I found this very intriguing and decided to try. Here is my version of the
filter: <http://www.careesma.in/employer/joboffer/view/id/124996>

Any thoughts/feedback welcome.

------
prewett
If I have to spend a lot of effort just figuring out if I even want to apply
to the job/company, I'll probably just look somewhere else. On the other hand,
if I'm not looking for a job but like puzzles, then maybe I might do it.

------
oconnore
As someone who occasionally solves hiring problems with little intention of
applying, it was nice to hear that my hobby is less annoying than I had
imagined.

------
ezl
i remember doing this.

i mostly did the puzzle because it was a fun distraction.

another thing is that it self selects for people who consider problem solving
fun. its not necessarily the most predictive of job performance, but it is
better than nothing and you'll also be filtering for personality (no opinion
on whether that's a good or bad thing).

sure it might weed out some good people, but it probably improves signal/noise
ratio.

------
manishjhawar
How about putting a link to the real job post instead of "instructions on
where to send your resume" hidden in the application?

------
ralfd
What is this "Fizzbuzz" which only talented star programmers seem "able to
do"?

~~~
SatvikBeri
Fizzbuzz is a very simple problem described here:
[http://www.codinghorror.com/blog/2007/02/why-cant-
programmer...](http://www.codinghorror.com/blog/2007/02/why-cant-programmers-
program.html)

The point is not that everyone who can program FizzBuzz is talented. The point
is that programmers who can't solve FizzBuzz are probably extremely
untalented.

------
gtani
this was good:

<http://lemire.me/blog/archives/2011/08/15/better-job-ads/>

------
fowkswe
Great way to keep recruiters out of the process too.

------
robbywashere
OMG I LUV SEEING PEOPLE JUMP THROUGH HOOPS TO EAT. HAHAHA LOL, L4M3rz amirite?

------
joejohnson
What is fizzbuz?

~~~
joejohnson
Why are people so unhelpful? Is fizzbuz cryptography?

~~~
aangjie
It's an old blog post meme from jeff atwood
[http://www.codinghorror.com/blog/2007/02/why-cant-
programmer...](http://www.codinghorror.com/blog/2007/02/why-cant-programmers-
program.html). Simple googling for fizzbuzz programming finds it. EDIT: fixed
wrong author name . thanks lazugod

~~~
lazugod
Although that's Jeff Atwood, not Spolsky.

------
michaelochurch
I think companies need to do a better job of listing what technologies they
_use_ without making it sound like it's required to have mastered _all_ of
them. People who've taken Haskell or even Scala into prime-time production are
extremely rare. People who would be excited to try, and who could be trained
up into that role are a lot more common.

I'm getting to the point where I'm realizing that I'm only going to grow as a
software engineer if I'm extremely selective about the work I take. I'd go
into management before I'd work in Java or C++ full-time. (Of course, I'll use
these languages if small projects require it, but if you're a "Java shop" I'm
not interested.)

I wrote about this effect here:
[http://michaelochurch.wordpress.com/2012/01/24/how-any-
softw...](http://michaelochurch.wordpress.com/2012/01/24/how-any-software-
company-can-cross-the-developer-divide/) . It's long, but the TL;DR of it is
that it's very easy for a company to become good at hiring if it gets its core
technologies right.

~~~
eli_gottlieb
> _People who've taken Haskell or even Scala into prime-time production are
> extremely rare._

Not _that_ rare, actually. I have a hard time finding companies that want to
use Scala.

~~~
anonymoushn
I currently write Scala at work. As far as I can tell, it's a lot like C or
C++, except that you have to use while loops instead of for loops and you have
to perform unholy incantations to fake arrays of structs. If you don't do
this, you can either try to allocate a few hundred million tuples and
OutOfMemoryError, or you can allocate one array of a primitive for each field
of the struct and give up on the idea of having decent cache locality. It's
less capable of TCO than gcc, which is pretty hilarious for a functional
programming language. Generic programming in Scala is lots of fun until you
try to use an Ordering[Double] or an ArrayBuffer[Double] in a critical path.

Manageable parallelism is nice, though.

~~~
soc88
Almost every problem you mention is a problem of the underlying runtime. Are
you sure you bash the right thing?

~~~
anonymoushn
This is true. The only things that are trivial to fix are the absence of for
loops and the use of ArrayBuffer[AnyRef] to implement ArrayBuffer[Double]. It
might also be possible to make TCO of mutually recursive final methods work,
but it's probably more trouble than it's worth.

~~~
soc88
Aren't for loops already fixed since months? (see
[https://github.com/scala/scala/commit/4cfc633fc6cb2ab0f473c2...](https://github.com/scala/scala/commit/4cfc633fc6cb2ab0f473c2e5141724017d444dc6))

Regarding ArrayBuffer: It is not as easy as it looks like, because of the
conflicting nature of Arrays vs. Generics (yet another JVM problem). Amusingly
I tried using Manifests to do exactly that last week. I hit the wall because
the Manifests seemed to spread arbitrarily. Will try with TypeTags/macros when
they become available.

~~~
anonymoushn
I'm glad to learn that they're fixed for Range. Thanks.

------
its_so_on
Note: the below may apply more to contractors, but if you read through it I
think you might find it useful...

in order of confidence, here is how you hire someone for the job. (perceived)

100.000% - The job is done already, just like something you already had the
same guy do and it was perfect, now you pay him.

95% - The job is done to your satisfaction, you know the guy and he's solvent
and vouches for what he's done. He hasn't done just this, but maybe something
similar for you. You're not sure if this works as advertised yet...

90% - The job is done to your satisfaction, you know the guy and he vouches
for what he's done. He hasn't done this for you but did it for someone else
you know, who vouches for the guy.

85% - The job is done to your satisfaction, the guy comes recommended only by
people you don't know, but warrantees what he's offering and seems like he
cares about his reputation and about you, seems solvent and in a situation to
fix it if it's not what you thought you were getting.

80% - The job is NOT done (none of it is), but you know the guy (rest per
100.000%)

75% - The job is NOT done (none of it is), but you know the guy (rest per 95%)

70% - the job is NOT done (none of it is), but you ... (rest per 90%)

65% - the job is NOT done (none of it is), but... (rest per 85%)

60% --> the job is not done and you have a guy offering to do it who comes
recommended by people you don't know, you haven't worked with him, and he
doesn't seem like he's in a situation to warrantee what he's going to do or
care utmost about his reputation and about you. However, he brings proof he's
done it before (portfolio).

55% - the guy comes with recommendations from people you don't know, but
brings proof only of things OTHER than what you want him to do for you. He is
not in a position to guarantee what he's going to do for you.

50% - the guy comes with recommendations from people you don't know, and
doesn't bring proof of anything he's done.

BINGO. This is where you are if you get a stellar resume and cover letter with
references from people you don't know.

Now of these resumes, let's go through the hiring filters that work... 40% of
the time... 30% of the time... 20% of the time... 10% of the time...

Ready? Okay, you know what, let's not go through all that. If you are trying
to filter the hires so that you're getting the best of it like this, well,
what can I say. Good luck pal.

------
nirvana
A, B and C are never really defined in these discussions, so I'm going to use
my perspective:

C programmers can't do FizzBuzz[0]. B programmers can, and LOVE PUZZLES. A
developers are the ones whose idea of a "puzzle" is solving the fundamental
business problem via software. B programmers think that super elegant
engineering is great, and don't care about business so much.

These B players are also the ones who often will torpedo a candidate by asking
them trick questions (trick programming questions, irrelevant or obscure
interview questions, etc.)[1] These questions are great for making the B
player feel superior. Puzzles are a good way of doing this. This way if the
candidate is hired, the B player's ego is mollified by knowing he was able to
trick the candidate during the interview, and if the B player decides the
candidate isn't deferential, he can claim he doesn't have the engineering
chops for the job.

I sometimes wonder how many of these complaints about people "not being able
to do FizzBuzz" are from these B players. (I've never used FizzBuzz as an
interview question, I don't ask programming questions, but I've never managed
to hire someone who couldn't do FizzBuzz for a programming role.)

At any rate, one of the things that makes interviewing at a given place a
total roll of the dice is when you see evidence that they don't know what
they're doing. An HR person trying to screen you who doesn't know the
difference between java and javascript, for instance.

But another signal to me, is these puzzles. These puzzles scream to me "This
company is run by B players!"

For instance, this is how I knew Google was not a company I wanted to
interview with (even though they pursued me relentlessly for about 6 months,
and still pester me periodically)... before I had the inside info that
confirmed what the puzzles indicate.

I also don't believe in the "A people hire A people and B people hire C
people" mantra, because I worked at Amazon (A C organization if there ever was
one) and Bezos liked to say this (a D manager, in my book... but obviously he
thought he was an A.)

I'm sure to some people, I'm a B player, or a C player (certainly the puzzle
fanatics might think that) but I have seen a lot of companies get lost
spending all their time working on clever engineering and not solving business
problems. I like engineering problems and solving them, but the business
problems are the really interesting ones-- its amazing what software can do to
solve these problems, and often, non-engineering management has no way to
know, which is why specs often don't work because thy ask for only what they
think is possible.

So, I'm not being a snob here. I'm just saying- I use these puzzles as a
filter. I wouldn't apply to such a company (if I was still working for
others.)

[0] Theoretically, anyway. I'm a bit dubious about this claim-- all the
programmers that I've seen who were a bad fit and "couldn't' do the work" were
actually good programmers, just dealing with bad management. I have met a lot
of non-programmers who think they are programmers and who only can do things
that fit their pre-determined ways, e.g.: They're trained in a set of
foundation classes and can't solve problems without using those classes (which
have already solved things like sort, etc.) I assume these are the people we
hear about when someone says "can't program fizz buzz".

[1] For example, once I was interviewing for an internet infrastructure job
and got quizzed on Win32 API details (like "what are the parameters for this
obscure graphics call?" "I don't know, haven't looked at it in years." "It
says here on your resume that you know Win32!". Similar example- interviewing
for a Mac development position in the PowerPC era, mentioned I'd done assembly
in Z80 back in the day and had a little familiarity with 8086 assembly--
getting questions about the Pentium instruction set. I could answer broad
questions, because I had a high level exposure, but I only claimed to study
8086 back in the day, not be writing assembly currently, and it was for a Mac
development job, and he wasn't even talking about the PowerPC.

~~~
RandallBrown
I'm curious why you think Amazon is a C organization. They make fantastic
products. Their main store, AWS, and the kindle are all pretty great and well
ahead of their competition.

I've heard bad things about working for them, but as a consumer, I love
Amazon.

~~~
nirvana
I'm talking about the quality of the products and the management. The Store
does great because they have good customer service-- I don't fault them
there-- but it is still chicken wire and duck tape from an engineering
viewpoint.

AWS has gotten a lot of hype, as has the kindle, but I don't think either of
them are ahead of their competition. My point though, was internally.

Amazon is full of B and C players who think they are A players, and that goes
all the way to the top. (There are some A players there too, of course, but
they often leave within a year or two.)

------
wilfra
The lesson to be learned here is if you can do something different and get
people to look at and talk about your posting, more people will apply because
more people will see it. If the place all of those 'more' people see it is a
place frequented by A players (such as HN), more A players will apply.

In other words, this was a marketing/PR win - not a win in terms of how you
are asking people to apply.

