
Aaron Swartz: How I Hire Programmers - mqt
http://www.aaronsw.com/weblog/hiring
======
davidw
I'm not aware of all the details of Aaron's career, but I was not under the
impression that he has been in a position to hire that many people. Knowing
who he was hiring for what size organizations and in what capacity would be
useful information.

~~~
benhoyt
You're right, that would be useful information. However, Aaron was one of the
reddit founders and I imagine gained a few pennies from that. Perhaps the
project he's hired for is openlibrary.org?

From <http://openlibrary.org/about/people>: "Aaron Swartz, Former Project
Leader. The Open Library wouldn't exist without Aaron. He wrote the backbone
of the system you see today. He has also worked on specs like RSS, startups
like Reddit.com, and software projects like web.py. He worked from San
Francisco to architect the site, _put together the team_ , and attempt to keep
things organized, and we couldn't have come this far without his crucial
expertise."

~~~
davidw
"Gained a few pennies" goes without doubt, but that doesn't necessarily
translate into hiring. For instance, 'our very own' PG has probably _hired_
fewer people than many people here, although I'm sure he's interviewed way
more than most of us have.

------
staunch
The best "interview" I've ever had consisted of me spending an entire day
hanging out with the team. We essentially pretended that that it was my first
day working there. They asked me to keep coming back.

Even if they hadn't hired me I would have been happy to do it. It was
enjoyable and I learned a lot of interesting stuff. I know some people would
say it's unfair to ask this, but I really wish it would become the industry
standard method.

~~~
paulbaumgart
Did they have you sign an NDA?

It seems like it'd be hard, using this tactic, to protect trade secrets in
companies where that's important.

~~~
DarkShikari
Facebook makes everyone who walks in the door sign an NDA, including visitors
and people coming in for interviews. It literally takes 15 seconds to do.

~~~
chollida1
Really? How can you read a legal document that quickly?

If I was signing something that could get me sued in the future, I'd make darn
sure that I read it thoroughly before signing!

That's just common sense:)

~~~
randallsquared
Few people read contracts they sign any more. In most of our (U.S., at least)
culture, people no longer view a contract as an agreement between party A and
party B, but as a paperwork step that has to be signed as a formality. I think
most people don't even read their own employment contracts; they just sign
however many places it calls for and move on.

With EULAs and click-through "contracts" devaluing the term, the average
person probably signs hundreds of contracts a year, and reads none of them.

~~~
madebylaw
I mostly agree with you, but I think it also depends on the types of contracts
we're talking about. I definitely read all 20+ pages of my apartment lease
before I signed it, but not a word of the iTunes EULA when upgrading it.

------
anatoly
I think this is a terrible way to interview. I've encountered, again and
again, candidates who appear to be very intelligent, you can have a great
discussion with them, terrific rapport, they understand everything, they give
interesting suggestions, and when you ask them to write a simple piece of code
they absolutely cannot do it. Not because they're stressed etc., you can
actually see they can't write code. Sometimes the incongruity between their
confident appearance and convincing talk versus their inability to write code
is mind-boggling.

Now I consider any interview process incomplete and utterly unreliable if it
doesn't include writing actual code, however simple and straightforward.

~~~
tsetse-fly
What sort of programming problems are you giving them? Did these candidates
have experience with other projects and were they able to fluidly explain
their involvement with them?

It must be pretty rare to find a person who can describe a complex project in
detail but is unable to write any code during an interview.

~~~
anatoly
I thought it was rare too until I started interviewing a lot.

Yes, these candidates would typically be able to explain their involvement
with previous projects, to give a technical overview of what they'd done. They
would totally pass my bullshit filter. I thought I had a well-tuned bullshit
filter. I was wrong.

The programming problems are typically very simple, an order of magnitude
simpler than an algorithm or design question that might be a part of the same
interview. I don't want to give specific examples, but maybe implement one of
a standard algorithms on an array or a tree (I'm not talking balanced trees,
way simpler). Maybe do something that requires a nested loop and some bit-
twiddling, or careful thinking about entries in a two-dimensional array, or a
bit of smarts to hold some sparse data - but again, nearly trivial from the
algorithmic point of view. There are many possibilities.

What you want to check is whether the candidate can think through their own
code, whether they know to check for edge cases, whether they think of test
cases, how they deal with off-by-1 possibilities, how they debug their own
code, do they have a reasonably clear style.

What you tend to see when a candidate fails such an assignment is people who
don't remember the mask to fill a byte with 1's, who can't get a nested loop
right because they can't seem to keep two counters in their head at the same
time, who don't have the instinct to check for an off-by-1, who can't trace
through their own code visually with sample arguments, etc. etc. Sometimes
they had just impressed you a lot a few minutes before with a nice technical
summary of what they did in their previous project.

------
bioweek
I think this is a great technique. To all those who think programming IS a job
done under pressure, really think hard about your specific job and your
specific situation. We all like to glorify our careers (to the point of
fooling ourselves) and imagine scenarios where something mission critical
breaks and it HAS to be fixed in five minutes (something like you'd see on the
show 24).

But has that ever happened at your job? If so, instead of hiring high pressure
programmers, could you take steps to eliminate the high pressure scenarios,
e.g., unit testing, back up systems, roll back plans?

I know everyone will have counterexamples of this high pressure thing that
happens.. but is it really typical for your job, really?

------
waterlesscloud
My favorite hiring process I've been a part of was with a AI startup. The
initial phase was a list of maybe a dozen questions in an email. Fairly open-
ended questions about how I perceived aspects of intelligence, ontology, and
general philosophical concepts. Then a day at the company talking with the
founder and the team, walking through their concept, long two way discussions
about it, and a discussion about the direction of AI in general. Didn't get
into a lot of coding details, it was assumed if I could participate in the
conversation at a sufficient level that I'd be able to work the rest of it
out. Which was true. Great process, completely appropriate for that situation.

------
richcollins
Something else to consider ...

There are a lot of developers that "get stuff done" without accomplishing
anything of use to the company.

~~~
ojbyrne
I think that's mostly their manager's fault.

------
paulbaumgart
Inspired no doubt by Joel Spolsky's famous (in some circles, anyway) article:
[http://www.joelonsoftware.com/articles/GuerrillaInterviewing...](http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html)

Their objectives are basically the same. The difference is in the ways of
proving the "getting things done" part.

I never assumed the programming questions in an interview were really about
getting the right answer in ten minutes or however long you spend on it, but
more about giving the interviewee an opportunity to explain their thought
process and problem solving approach to the interviewer. I don't agree that
asking interviewees to solve programming problems in an interview is useless,
but obviously looking at code samples from other work is great, too.

~~~
jacoblyles
>"I never assumed the programming questions in an interview were really about
getting the right answer in ten minutes or however long you spend on it, but
more about giving the interviewee an opportunity to explain their thought
process and problem solving approach to the interviewer."

That's what they say. But do you know anyone who failed the questions and
still got hired? My experience has been that even when I nail the algorithmic
questions, I miss a few trivia questions and don't get called back. Nuance
cannot be replicated at corporate scale.

~~~
jerf
Well, I can say on the flip side I've been involved with hiring decisions
where the interviewee did not nail every single answer, but we hired anyhow.

It should be pointed out that I structure the interviews to try to avoid that
case anyhow. If your interviewee nails everything, it either means that you've
got a class-A programmer, or your interview difficulty tops out too soon. I've
become a big believer in gradated interviews. I've got a pretty good SQL
series that starts out where the answer is literally "SELECT * FROM
some_table" (though I find it occasionally trips people up to simply omit the
where clause) and goes up from there on this little two-table schema. I've got
some good HTML generation series that starts out with a simple "can you dump
out a table from a 2D array or equivalent structure?" (since I try to do
language-agnostic interviews) and go up through "do you know about escaping"
and "can you handle optional parameters" and so on. Over time I've been
selecting for the series that produce more gradation.

------
johnpignata
I'm surprised nobody has talked about pairing interviews in the comments yet.
I spent years clumsily hiring developers crapshoot-style. I shared the
misconceptions that people get so nervous during technical interviews that it
obviates their usefulness and that it doesn't really matter if a candidate
writes code or not during their interview. My current employer, Pivotal Labs,
hired me after a first interview that consists of interview-y conversation and
a short pairing exercise followed by a second full day of pairing with the
team on an actual project.

I know pairing is sometimes controversial around these parts, but for
interview purposes its benefits are pretty evident.

------
edw519
On one hand, I have always looked forward to and enjoyed Aaron's writing, and
was really curious about what he had to say in this post.

On the other hand, this is a subject near and dear to my own heart, and while
I appreciate a lot of his methods, I don't think his model scales well. In
order for it to work the way he wants, _he_ has to conduct the interviews
himself.

A few thoughts...

 _Programming isn’t typically a job done under pressure, so seeing how people
perform when nervous is pretty useless._

Not my experience. Actually, quite the contrary.

The single biggest difference I have ever seen (over and over again) between a
good programmer and a great programmer is how they respond under pressure.
Given enough time, lots of programmers can code something that works. The
problem is that there are many occasions where there simply isn't enough time.
Can you hit a deadline? Can you stay awake all night if you have to? Can you
resolve this problem that's holding everything and everyone else up? Can you
settle down that user or customer who's up in arms? Can you figure out what
wrong decision was made 4 steps ago that is causing big problems now? And can
you do it _now_?

Understanding how well someone performs under pressure (and whether or not
that makes them nervous) is hardly useless. I can't think of too many more
things that are useful to know.

 _So I just request a code sample and a demo and see whether it looks good._

There are too many cases where this doesn't work at all. What if they've never
contributed to an open source project? (Whether or not you like this has
nothing to do with their skill as a programmer and is another subject). What
if they're prohibited from sharing someone else's proprietary property? What
if they're bound by someone else's (illogical) shop standards? What if they
salvaged someone else's shitty architecture with great workaround code? If
they wrote something as part of a team, which part was theirs? How much of the
analysis, design, architecture, testing, and deployment did they do? For that
matter, how much of the actual programming did they do? Lots of posers share
something they "worked on" but could never build in a hundred years.

 _To find out whether someone’s smart, I just have a casual conversation with
them._

That's it? You make judgements on the fly? Using what metrics?

The idea of a learning from a casual conversation makes a lot of sense. But
what is your plan? Ten people having ten conversations will come up with ten
different assessments of the same candidate. Unless there is some kind of
standard and a plan going into the meeting. How will you know they are smart?
How will you know they can get things done? How will you know they can work
with others? Without a plan to answer these questions, you're just waving your
hands.

I suspect OP already has a plan of attack for his casual conversations but
it's hard to say because he never says anything about it.

 _Ask them what they’ve been thinking about and probe them about it._

Again, this may or may not be a good litmus test. I know programmers with IQs
of 160 who have written tons of brilliant code. If I asked tham what they've
been thinking about, the answer could be "who will win the Notre Dame game" or
"will I get laid tonight" just as easily as it can be about something truly
cerebral.

 _Finally, I figure out whether I can work with someone just by hanging out
with them for a bit._

Again, you can learn a lot about someone else this way, but sooner or later,
you'll need some kind of standard and metric for this method to be replicable.
Before you start, you must be able to say, " _This_ is when I will know what I
need to know." Until then this is just one guy's hand waving.

 _I’m amazed that so many companies use such silly hiring methods instead._

I'm amazed that so many companies use such silly methods for so many other
things as well. But there are just as many who hire very well. I know many
great programmers employed and still delivering great product 15 or 20 years
with the same company. The same companies that seldom make bad hires. They use
a lot of OP's strategies. But they also combine these strategies into a
comprehensive system that works _no matter who does the hiring_.

I like OP's thinking. I just don't see how it will work well once his company
becomes too big for him to do every interview himself. Systemitizing his
methods will make it scale.

~~~
Retric
I know it's probably an exaggeration, but _IQs of 160_ are the 99.997th
percentile so there should be something like 10,000 people in the united
states of all ages with 160+ IQ's. Suggesting those are the type of people you
want to recruit is mostly a waste of time because there are just not that many
people let alone people with that IQ let alone people with that IQ who are
looking for programming jobs.

~~~
davidmathers
As a programmer with an IQ of ~125, I'm doubtful that I would want to be a
programmer if my IQ was 160+. I have a hard time believing that there are many
professional programmers with IQs that high.

~~~
reeses
Having an IQ of 160 or above is not a life of sitting on clouds thinking deep
thoughts about how string theory is obviously wrong. You run especially hard
into the nurture trap so often discussed here.

That is, hearing,"You're so smart," so many times, from teachers, other
students, coworkers, bosses, etc., cultivates a sense of elitism and a shock
when something is actually difficult.

You're also bound by the same emotions as most other people. If you're a
particularly aggressive or irritable person, imagine being surrounded by
people who keep making mistakes because they're stupid. If you turn it
outward, you're impatient at best and a bully at worst. If you turn it inward,
you're stressed out, anxious, and depressed.

All of that said, being a professional programmer was the most satisfying
thing I've done. Unfortunately, I moved on precisely because of the above
paragraph -- I had managers I felt were particularly dim.

~~~
drewr
Always worth mentioning: "The Inverse Power of Praise"
<http://nymag.com/news/features/27840/>

~~~
dbz
I do not appreciate compliments anymore. I haven't for a long time. I have a
complicated view I don't really wish to explain right now (maybe I'll write a
blog post later) about why compliments are almost insulting, especially
insulting by those people who care about "you" and who give them often (Sorry
for that passive voice).

My thoughts all stemmed from the idea that compliments make people worse at
what they do. At BEST they make the person continue at the same speed with a
boost of moral. Imho.

------
storborg
I often hear people assert that engineering isn't done under pressure, so an
interviewee's performance under pressure doesn't matter. I don't think that's
necessarily true: lots of systems have to be designed for life-and-death
situations, web sites often have major crises/bugs, every team faces last
minute deadlines, etc.

I wouldn't want anyone on my team who fell apart under stress. Performance
under pressure isn't the only factor, but it is a factor.

That said, if anyone has a zero-stress engineering job, please tell me.

~~~
coffeemug
But this is a different kind of stress. Interviews are usually performed under
"social" stress - people normally get nervous when meeting someone new,
probably for evolutionary reasons. This "short term" stress is very different
from the stress you get because of a deadline or a critical bug.

~~~
storborg
I've never really felt it was that different, but for the sake of discussion,
how would you simulate the non-social variety of stress in a hiring situation?

~~~
kailashbadu
Having experienced it myself, I can vouch that such an difference exist. When
forced to write a complicated piece of code with a bunch of strange people
breathing down my neck and watching every step I take, my productivity tends
to slacken off when I hit a coders equivalent of writer’s block. It just
doesn’t come out as naturally as it used to. In contrast, same task in a more
familiar and comfortable setting looks like a piece of cake even if deadline
is drawing close.

~~~
tsetse-fly
Coder's block is a good way to describe it.

I've had phone interviews where I've been asked to solve a logic puzzle with
pen/paper while explaining my approach. Solving a problem you've never thought
about while being put on the spot turns out to be incredibly difficult. These
sorts of problems require deep concentration and quiet yet it tends to be
awkward to have silence between strangers. The situation really inhibits
serious problem solving.

------
e40
"Many brilliant people can seem delightful in a one-hour conversation, but
their eccentricities become grating after a couple hours."

That is so true. Most people can't keep up their "interview face" for very
long. It's tiring. At some point, if you put them at ease, they will revert to
their normal self. That is when you'll get to see what they're really like.
I've seen some pretty amazing transformations when waiting for this second
phase of the interview.

------
diN0bot
this is all pretty good advice, especially the reviewing past [code]
experiences and getting a sense for what it'd be like to work with the person.

i like staunch's "spend a day with the team" method a lot, though talking
about strengths/weaknesses and role playing certain scenarios is also a good
way to get a sense for potential clashes ("tests are for monkeys" or what not)

what i didn't understand was the "smart" part. i don't think it's possible to
get a sense for whether someone is smart--especially technical smarts as
opposed to, say, leadership or empathy smarts--simply by talking with someone.
in fact, the whole concept of smart is exactly what pressure-ful, fanciful
interview questions are trying to get at. i'm not even sure "smart" is a
single or useful concept.

specific interview questions do try to determine technical skills or thinking
process...but it a chat over coffee, a single, brief encounter, any different?
in a good way?

i'm nitpicking here, but i felt that the solution to smarts--assuming we
define smart to something useful--is kind of weak.

~~~
jurjenh
I suspect the "being smart" is in terms that the employer can relate to - do
they think about things in the kind of depth that I like / appreciate / can
work with.

From my personal experience, I've dealt with smart people and smart people -
one kind knows / understands a lot of things in a broad field, the other kind
knows a lot about things but is almost completely devoid of common sense. I'd
prefer the first kind in any situation where I or my friends would have to
deal with them over the medium to long term. I'm pretty sure this would be
reflected in all areas worth measuring for a company too.

So trying to define smart as a concrete measure may be missing the point, as
you will actually have to work with the person, and what you feel is what
tends to form your opinion / bias and affects your future interaction in such
a way that it generally tends to become a self-fulfilling prophecy.

------
look_lookatme
tl;dr version

Smart, affable, productive people with demonstrated ability make great
employees.

I'm not interested so much in _who_ to hire -- I think that's a no brainer,
frankly. I'm more interested in where to find these great people to hire.
Really. We are having hell finding great developers.

------
natrius
I'd imagine that googling for bug reports that one has been involved in on
either side would be a good heuristic for problem solving skills and
sociability. At the very least, you'll avoid hiring anyone who can be
identified by googling for "bugzilla asshole".

------
arithmetic
Great advice - I like the fact that Aaron keeps it real simple. I also like
how he doesn't like questions along the lines of "counting the number of piano
tuners in Chicago". I've been witness to interviews with similar questions and
I'm always left wondering as to what the point was.

That said, I'm not sure if _all_ of his advice can be followed by a reasonably
medium sized company/organization where hiring decisions are made by a
committee, rather than one person.

As a side note, I think that reading a resume can actually be a good starting
point to understand the candidate's background.

~~~
paulbaumgart
This thread has some decent discussion about the case-study type questions:
<http://news.ycombinator.com/item?id=940527>

------
kpaddy
I've personally felt that asking obscure tech trivia questions is meaningless.
There is no use in asking someone a question, the answer to which can be
obtained by a quick Internet search. Edit: Grammar

~~~
prodigal_erik
I'm not a big fan of trivia, but there are basic details you will necessarily
learn over time just by working with any technology, and not knowing these is
damning. E.g., asking how arguments are passed is a quick shibboleth to catch
the guy who thinks clicking "print to file" entitles him to put PostScript on
his résumé (a colleague has actually witnessed this).

~~~
kls
the problem is that these questions can devolve into a (you know) waving
contest in which the interviewer is, instead of interviewing, satisfying his
own ego by asking obscure questions that sever no purpose other than to
inflate the interviewers perception of his own intelligence. This is
particularly dangerous in technology professions where their is a high
propensity of intellectual self-conceit.

I understand the low hanging fruit, I just witnessed and interview the other
day for a Java web developer, and one of the questions was describe a
singleton pattern to which the interviewee could not respond. Given that there
has been a history of singleton / Java / Bad things web applications problem,
not understanding the pattern is a significant issue and I did thing it was an
appropriate question to use as a filter, but asking a web developer deep
questions about class-loader nuances seems like overkill to me and I have seen
that done as well.

~~~
count
I always loved to ask a single super obscure question that almost nobody would
know the answer to. I didn't expect, or desire the 'correct' answer, I wanted
to see how they handled the fact that they were asked something they almost
assuredly didn't know. Did they say 'I dunno' and just shrug? Did they make
some shit up? Did they try to work it out? That reaction to the
uncertain/unknown is, I think, a pretty important part of figuring out if
they'll be good to work with - who doesn't run into something unknown on a
fairly common basis?

------
w00pla
You should also check if the person is reliable (i.e. competent, responsible,
hard worker and isn’t an obnoxious diva). In a lot of situations you get
people who seem fairly smart, yet make an effort to bring their personal
problems to the workplace. This can be extremely counter productive
([http://www.reddit.com/r/reddit.com/comments/1octb/reddit_cof...](http://www.reddit.com/r/reddit.com/comments/1octb/reddit_cofounder_aaron_swartz_discusses_how_he/c1odyq)).

~~~
tsetse-fly
That's the second time you've posted that link on HN. Do you have something
against Aaron?

~~~
w00pla
Yes, he is an annoying self-anointed expert. This ruins what could be an
interesting debate (on any topic). It is ironic that a __bad employee
__somehow becomes an expert in hiring people.

I wouldn’t mind if people stop posting drivel from his site.

~~~
tsetse-fly
Perhaps you should look past the author to the content. There are some
interesting ideas in there and it seems to have spawned some good discussion.

------
blasdel
I think this a terrific interviewing method, but only if you have the
metacognition to actually know why you didn't like a candidate, and the
courage to tell them how they failed.

If you don't commit to that communication, you'll be an unsettling and
depressing interviewer. You can't just play part of the process by the book,
it really has to be all or nothing.

~~~
pmiller2
An interviewer has no obligation whatsoever to tell a candidate why he or she
"failed" an interview. In fact, I'd say you'd be hard pressed to find one who
would tell, for legal reasons. Companies hate being sued, so they're not going
to tell you anything that might potentially give someone a reason to sue them.

------
jbm
Perhaps off-topic, I was a bit put off at his definition of a friend. I
wouldn't want to befriend someone who keeps me around to solve his problems
for free while I'm at work.

Being that I've only interviewed at big-ish companies and in Japan (where
companies ape the large companies uncritically), I haven't had a chance to see
some of these techniques :/

------
kls
I thought he had a great point on the technical questions some of the best
developers that I have see, don't spend their time studying every trick
question for the array of technologies they use, just so, on the off chance
that if it comes up in an interview, they we be spot on. It is crazy, and it
reflect poorly on the ability of the interviewer to competently understand the
task they are trying to accomplish, which is hire a good developer that
integrates into the team well, not a jeopardy contestant.

------
ojbyrne
Yep.

~~~
ojbyrne
Ok, perhaps I wasn't clear. I love everything in this article, but it's hard
to say anything that doesn't increase the complexity of it, which is the whole
problem with people hiring programmers. So I'll say it again.

Yep.

