
Hiring and the market for lemons (2016) - ff_
http://danluu.com/hiring-lemons/
======
andrewstuart
When the topic of "great" developers comes up, I always challenge anyone to
define _precisely_ , in quantifiable terms, what a "great developer" is
exactly.

How do you know someone is a "great developer" if you can't, in tangible
terms, say exactly what a "great developer" is?

This is even more of an issue when recruiting because recruiting is an
assessment process to determine if someone meets some requirement. But just
like software testing, if you can't quantify the requirement, then you can't
quantify a test to determine if someone meets that requirement, and therefore
the interview/assessment process is simply voodoo recruiting.

Voodoo recruiting is when you have the appearance of scientific process in
recruiting. - a process, - a set of questions that appear good, - a coding
test that seems technical enough that it must be telling you something
meaningful about the candidate, - the opinions of many people on the
candidate. But in fact all of this is voodoo because you still can't directly
define the requirement and thus you never known if you ever found what you are
looking for, which is not actually known.

The final decision in recruiting a "great developer", as always, comes down to
some subjective opinion based on a range of criteria that can't be proven to
have had any meaning in the first place.

~~~
SatvikBeri
I often tell people that before they think about their hiring process, they
should think about their performance reviews. You're doing those with multiple
orders of magnitude more data, yet most managers will say their performance
reviews have a lot of uncertainty.

I spent a lot of time at my last job developing ways to measure performance
across my team. One big surprise was that I had to significantly change the
way we assigned work in order to make it at all measurable. Rather than giving
people well-defined tasks, I switched to assigning projects that had some
amount of (expected) business value. This meant the projects had to include
things like deployment, talking to marketing, and whatever needed to happen to
make it useful. Generally each one was 3-6 weeks of work.

This provided a good first-order measure of effectiveness. Importantly, it
helped people change their habits to contribute more business value. I found
massive variation in the junior engineers, easily with 5-10x differences
between the most and least effective. The differences in senior engineers were
smaller, but still ~2x.

Of course, there are other major factors you have to take into account, like
code quality and mentoring. I developed similarly detailed systems for
measuring those. Once again, you have to change the way you do things if you
want to make them measurable, often accepting local inefficiencies for global
insight.

With all that data in hand, I correlated that to our hiring ratings. The
results? Our two best engineers were both borderline in the hiring process,
and would have been rejected except for a strong vote of confidence by our
senior engineer, who, when we checked, turned out to have a basically perfect
track record on hiring judgments-much better than anyone else in the company,
me included.

~~~
majormajor
IME it's hard to precisely rank average devs who are in the middle. There's a
lot of clumping.

It's a lot less hard to identify overperformers or extreme underperformers.
You mention the big possible range, but I've never had to dive that deeply
into data to spot them - I've used the data to check my instincts, but have
generally been right. And if you don't trust your instincts, just ask your
team! They know whose code reviews they dread, who never offers helpful
suggestions, who seems to do just the minimum. It's sometimes tricky (people
often don't want to say bad things about their peers), but generally people
already know.

I know what I'm looking for in my performance reviews, and so I know what I'm
looking for in my candidates. A competent coding baseline, and skills in the
non-coding areas (knowing what _not_ to build, anticipating edge cases,
ability to distill business logic, ability to explain complex things
clearly...). But this list is quite different from the "standard" tech
interview loop: algorithms after algorithms with a sprinkling of "design a URL
shortener."

~~~
SatvikBeri
The data had a lot of other benefits:

* When I took over, I was a lot more permissive about letting people work from home. Upper management pushed back and was worried about productivity. I was able to show that people were getting much more done than before (though admittedly there were other factors, I changed a lot of things.)

* After a few months of monitoring, it became clear that one coworker had a predictable but volatile pattern: he'd be very productive for a few weeks, then very unproductive for a few weeks. So I asked him to take on urgent projects during the "up" periods and avoid them during the "down" periods.

* One particular person improved so fast it was almost unbelievable. They were already strong to start, but after 12 months they were literally getting about 4 times as much done as they had at the 3 month mark. I would never have believed it if I hadn't been keeping close track...and this helped me fight for promoting them twice in a year.

------
seattle_spring
Want to know a great strategy for hiring lemons? Ask only Leetcode-style
questions. Don't ask anything relevant to the job. Don't ask about past
experiences, architecture, tech-specific questions, or anything like that.

Only. Ask. Leetcode. Questions. If it's not a stupid algorithm disguised as
something cute like a dog jumping over a river, then don't ask it.

You will be in the exact situation this article describes: Passing up on good
engineers, and not being able to tell if the people you do hire can actually
perform on the job.

------
dvt
I hold Joel Spolsky in very high regard, but I just think he's fundamentally
wrong here. I've seen (first hand) _many_ times when great developers stagnate
at various companies and they leave. This can happen for a few reasons, but
we're deluding ourselves if we think every single job is "solving the big one"
\-- no, sometimes we need people to write the plumbing, the boring stuff, the
unit tests, and so on. Great developers, more often than not, get bored. If
they have any sort of intellectual curiosity, they _will_ leave. In most
cases, corporate programming is just not very interesting. So first, we have
the curious geniuses. I know many of these people that hop around from job to
job and they say "meh Google sucked" or "Facebook was boring so I'm at this
new startup now."

Second, we have the people like me. I hope my hubris isn't showing, but I
think I'm a pretty decent developer. I wrote a few books, had some
contributions to a few big-boy OSS projects, am in the top 1.5% of Stack
Overflow (even though I haven't really been contributing for like 4 years
now), etc. I also built things -- many, many things -- some have a few hundred
stars on Github, many have none. My goal in life is to "do my own thing" and,
eventually, get that FU money. That is _never_ going to happen working a
regular 9-5 job and, in my early 30s now, I've completely accepted that. So I
work for a few years (2-3) and then take a sabbatical where I dedicate my
entire time on a project or two, trying to get it off the ground. When it
inevitably fails, I go back to a "regular" job. The gap, contrary to popular
belief, doesn't hurt you. When you explain and show what you built, people are
more likely to be blown away rather than derisive.

Finally, we have people like Max Howell who famously got rejected by Google
even though literally everyone at Google uses his software. Max Howell is
_most definitely_ a great developer, and yet Google tripped all over their own
corporate shoelaces. Let me put it this way: if I did a startup, I would 100%
want Max Howell on my founding team. I mean just take a look at not only the
code quality, but also how PRs/reviews are handled in homebrew.

~~~
monocasa
WRT to the Max Howell thing, if you can't swap the left and right branches of
a tree, I have a hard time believing that you're a great developer.

~~~
thereare5lights
This kind of bias is a prime example of people valuing their own preconceived
notions of how the world works to the point where they are incapable of
updating that belief with hard data.

~~~
monocasa
How is 'something I expect out of a sophomore or equivalent' for a Google
developer interview an example of bias?

------
deviationblue
Everyone wants a Ferrari, but most work is ordinary enough that all they
really need is a Kia. It's very unlikely that you're the next Google etc.,
better to hire according to your needs and stature. Maybe you know what you're
doing, and if you need a Ferrari to get it done, get a Ferrari.

But let's be honest, most jobs out there are "sell this to people, faster".
How much intellect are we supposed to spend on that? I digress. All you really
need for that shit is a Kia.

Also, it's really hard to hide both competence and incompetence. I wouldn't
worry about passing resume screens or coding interviews if you can actually
make something. If you have the skills to make things, you can even start your
own business.

Most people are average skilled, though. To be honest, that's all you really
need. Let's not pretend otherwise.

~~~
perl4ever
I was really enthusiastic about a recent interview, because the process
started with a take home coding assignment that I thought I did well on and
got me invited for a 4+ hour interview. But then the in person interview
session ended with asking me to write a simple program in an unfamiliar IDE
and language while they talked at me - they called it "pair programming". I am
utterly unable to work anything out if I can't have _intermittent_ peace and
quiet. So, the verdict was accurate (in a tautological sort of way) that I
wasn't a good fit, but clearly they have preconceptions that are preventing
them from hiring useful people.

One thing to keep in mind though, is that whenever you're dealing with a lot
of similar trials of something, avoiding all the big mistakes is more
important than getting all the big successes. I think this applies to hiring
people, choosing investments, relationships, probably other things. So having
irrational reasons for rejecting people is less harmful than having irrational
reasons for accepting people. Which is probably an important reason why free
markets don't automatically eliminate the prejudices we consider unacceptable.

~~~
stevenwoo
Did they tell you it was going to be pair programming or was it a surprise? I
find things like this are better to practice beforehand or if one knows
beforehand one would prefer not to do it, to just reject it before it gets
that far along in the process.

~~~
perl4ever
"Pair programming" in an interview wasn't something I'd seen before, so I
didn't prejudge it. I'm still not sure what to think of the exercise - because
it may or may not resemble what is actually done on the job.

From a previous HN thread on pair programming[1]:

"Both partners must be fairly versed in the language and methodology of the
programming."

I got into the weeds with the minutiae of the editor pretty quickly. Figuring
that out _and_ the language _and_ the program in a few minutes while
maintaining a conversation about all of the above is far beyond my limit of
simultaneous tasks. It's not about the difficulty of any of them, it's about
the contention for locks, if you will.

Also from the article it links to: "Navigator knows the system well while the
driver does not" is described as a fairly good situation. This jumps out at me
because (presuming I was the driver and interviewers were navigating) in
_actual_ driving situations, I _really_ don't function well with someone
telling me what to do and carrying on a conversation at the same time as I the
driver am trying to exercise higher brain functions.

It may be there is a segment of the programming population that uses
mathematical processing brainpower for programming, and another segment that
uses verbal processing power, and the disconnect here is that I am in the
latter category. Thus, while talking is very helpful to me to work out a
problem, I can't talk about one thing while doing another because they both
require my verbal faculties. Maybe people who multitask better are using
different parts of the brain that are inherently more independent.

[1][https://news.ycombinator.com/item?id=714362](https://news.ycombinator.com/item?id=714362)

~~~
stevenwoo
I've done it before and it can be difficult to get into a flow state that is
more natural for me by myself but like all things requires practice to acquire
and maintain skills and also, familiarity as you say, with the tools and
problem space or it's not a very good assessment of your ability except for
that one very particular set of circumstances of going in blind on
tools/editor!

------
mixmastamyk
I do freelancing. Three times in the last few months I’ve had to sit in front
of coderpad to attack a problem I’m not familiar with and come up with a
solution—with one or two folks watching, the clock ticking, my family’s
livelihood on the line, next to ~two hundred grand in a suitcase. I write well
tested code every day but can’t perform under these conditions.

It’s basically the scene from Swordfish, and just as ridiculous.

Two of the three groups seemed to realize it’s a poor test, didn’t have high
expectations but went through with it anyway. One seemed to think I was
incompetent because I didn’t have his trick memorized about keyword args
making closures possible in a loop in Python.

I don’t write clever code on purpose, and certainly not enough to have it
memorized to use under the gun.

The 'shortage' is a myth and I blame time-wasting hiring practices such as
these. Am thankful a smart guy like Luu is highlighting the bullshit.

------
Tehdasi
> needing a weird combination of skills, can be solved by hiring people with
> half or a third of the expertise you need and training people.

I'll add that companies don't mention their internal tools/libraries/current
code base (which is most likely of pretty crap quality and poorly documented)
which is much more a hurdle than whatever language/toolset than they put in a
job ad as required experience.

~~~
FLUX-YOU
This is a frustrating part. Even if you ask about it in an interview (there's
almost always internal tools/libs/code), there's not really any compensation
if they lie about their code quality and then put you on that stuff full-time.

Rewriting that stuff is not great for a career. It's always been the stuff
I've chosen to leave out because I don't want to signal to anyone that I'm
willing to full-time rewrite your stuff.

~~~
jaggederest
As one of my better former managers put it, "Nobody gets a promotion for
reducing the size of a tire fire by 50%"

------
chrisbennet
_" Joel's claim is basically that "great" developers won't have that many jobs
compared to "bad" developers because companies will try to keep "great"
developers. "_

This is a not what Joel said. Joel said that great developers _are not on the
market very often_. They don't need to go on the open market to find their
next job.

------
thoughtexprmnt
Most teams should not waste their time searching for a “great” developer
(whatever that is, exactly). Instead they would be better off just hiring the
first candidate that comes along in whom they have reasonable level of
confidence will soon be a productive contributor to the team's goals and
makeup, and then move on with business. Everyone involved in the recruiting,
screening, and interview process should not lose sight of this as the
objective, and resist any temptation to get sidetracked by silliness like
“pedigree” and “trendiness”.

------
hpcjoe
What I've tried to do over my career has been to hire people who showed
ability to learn and grow, and then offer them the ability to step up to
challenges. What I've gotten has been a rather bad mixture of incompetence,
cluelessness, etc. They were eager to pass the interviews and get the job.
They were not eager to do the work under the timelines we needed.

This isn't just devs. This was sales, ops, and finance.

I don't believe in any hard and fast rules about good vs non-good. I do
believe that past performance is no indicator of future ability. That at least
has been my experience.

Even more relevant, pedigree is of no value in estimating potential success or
failure of an employee at their mission. The best dev I'd ever worked with had
a GED (high school equivalency diploma), and never went to college. The worst
had a top-of-heap pedigree.

Hiring is hard. You have to be an optimist. In the case of a small company, a
bad hire can be a fatal mistake.

~~~
PostOnce
If everyone you hire in every department is "not eager to do the work under
the timelines you need", then perhaps your timelines are not realistic and you
either need to adjust them, hire more staff, or both?

A commander can blame his soldiers for losing him the war, but is it more
their fault or his?

------
rdlecler1
This is sort of like the venture capital market. Entrepreneurs will sort rank
their investor list based on perceived value of that investor and will work
their way down. Most VCs simply never see the great companies, which is why
top firms consistently outperform while the overall asset class has fared
poorly. In short, deal flow is everything.

------
taurath
I think there's a really key point with nontraditional developers. I'm also
terrible at interviews because of both anxiety and lack of formal education (I
can usually get optimal answers, but come across as "not confident" in them),
but consistently get really high ratings and reviews from wherever I work.

------
tabeth
I've been thinking about this for a while and have an interesting idea --
feedback welcome!

Imagine a fully open company -- code, tools, library, marketing data,
strategy, etc. Literally anything about the company you can think of is
accessible on the open and ready to interact with, except for the underlying
database data.

Following along? Cool. Now, imagine all of this stuff being on some open
repository (not necessarily Git, but for the sake of example let's suppose
so). As employees create their backlog of items they add four properties:

\- A 1-100 score on how much difficulty _they_ would have completing the item,
100 being extreme, 1 being none

\- A 1-100 score on how much difficulty they think an entry employee would
have with the item. An _entry employee_ would be defined clearly as the lowest
level for their ladder, e.g. Software Developer I.

\- A time estimate on how much time _they_ would have completing the item.

\- A time estimate they estimate on how long they think an entry level
employee would have.

So, knowing on this. Would the perfect hiring process simply not be taking the
items, picking things that should take a reasonable amount of time (less than
8 hours) that are in the highest reported entry difficulty possible and then
scaling that by the average self difficulty? With this system you could even
anonymize the dummy interview pull requests and have employees rate the
quality of it. Differences in quality and difficulty could be a positive
signal (e.g. high quality for low difficulty item is good, obviously, but high
quality for low difficulty item that was estimated to take less time than the
average estimate, by employees, is even better).

\---

So the purpose of this is less to identify who is objectively great, but
rather to figure out how good the candidate is relative to people in your own
organization. Personally I think finding an objective measure is an
intractable problem.

~~~
BWStearns
Domain knowledge is the uncontrollable here. A junior is still probably a
junior 3 months in but they are a lot faster for their domain knowledge
(codebase, business rules etc). Day 1 (I.e. Your proposed interview) there
would be little observable difference between various experience
levels/quality of engineer since the overhead of domain knowledge will dwarf
anything they already have in their head. That is, unless the problem you pick
is so isolated that there is no domain knowledge required. In that case you
have expensively reproduced fizzbuzz without the mass memorization of code
golf style answers.

------
rurabe
I think it's possible for both of these to be true if you back off of the
extreme positions.

1) Great developers are seldom looking for new jobs, but 2) There are a ton of
inefficiencies in hiring that making finding great developers possible if you
think outside the box.

------
mattsfrey
The problem with this is that so many companies (especially startups with
young ceos) have _no idea_ how valuable an employee is. They negotiate hard
and underpay often because that's just a general prescription in business.
From my personal experience "great" developers change jobs all the time.

------
jamestimmins
Does anyone know what he's referring to when he talks about Matasano's unique
hiring strategy?

~~~
antoncohen
[https://sockpuppet.org/blog/2015/03/06/the-hiring-
post/](https://sockpuppet.org/blog/2015/03/06/the-hiring-post/)

------
nstj
The article paraphrases Spolsky incorrectly: great developers don’t look for
jobs more than 4x in their career not because they stay in their jobs longer
(ie: career length/4) but because they are _headhunted_ for most of their jobs
so don’t have to look themselves

------
PaulHoule
The "Bob" scenario happens all too often. Sometimes the "slow and sure"
developer gets a lot of grief from pathological management that doesn't
realize that developers who seem faster as short tasks might go in circles for
years on bigger projects.

------
johngalt
Dead on.

There is also an environmental component in play. You'll find a lot more noise
in the hiring pool when the tech field is booming.

------
naveen99
That’s why the best time to hire is out of grad school, college or high
school.

------
itsdrewmiller
This is just not persuasive - here's a key point from it:

"If we put aside Joel's argument and look at the job market, there's
incomplete information, but both current and prospective employers have
incomplete information, and whose information is better varies widely. It's
actually quite common for prospective employers to have better information
than current employers!"

Ok, sure, maybe there are employers who don't know what they've got. But just
based on the raw amount of information on either side, all else being equal
current employers should have a _much_ better idea of actual performance.

