
What happens when you stop relying on resumes - appamatto
http://blog.alinelerner.com/what-happens-when-you-stop-relying-on-resumes/
======
tikhonj
First of all, I definitely agree that resumes are woefully overused and
overrated. They have structural problems and incentivize people to use
"negative selection" criteria where people are eliminated based on _not_
having a specific feature rather than selected _for_ excelling on something¹.

This article neatly demonstrates that resumes are _not necessary_ and that not
using them can unlock new sorts of candidates.

However, I don't think there's a conclusion to be made about the actual method
used here. I suspect that it worked because it was _different_ , not because
it carried a fundamentally strong signal. If everyone did this, project
descriptions would be gamed even more than resumes—it would select for people
who prepared _for the selection process_ ² more than anything else.

This reminds me of various captcha strategies I've seen used by small forums
to great effect—solving some math, typing a word into a text box, choosing a
popular character's picture… etc. They all work, perfectly. But only because
spammers don't care about the small fry: it's not worth their time to modify
their bots for your little site. If any given captcha becomes used widely—or
your forum grows big enough—they will bypass it trivially.

Now, an essay like this isn't quite as bad as a captcha, but the idea is the
same: it works because it's new and different. If everybody used it, it would
probably be a step back.

Ultimately, I think the real moral is that more companies should do _their own
thing_ , even if that thing is not great in the abstract. Being different
carries a value of its own, and it breeds biodiversity that's healthy for the
system as a whole. (Of course, many of the things companies try are really bad
for various reasons, but that's a different story…)

¹ In particular, most people have a bunch of "red flags" they look for with,
at best, cursory rationale—everything from passing on people who didn't go to
the right school to those who have breaks in their work history, based on
"common sense" or "experience" rather than anything meaningful. Most of these
criteria seem counter-productive.

² I also think this is really true for college admissions and especially the
admissions essay. A project blurb for hiring is more or less the same idea in
a new context.

~~~
analog31
_However, I don 't think there's a conclusion to be made about the actual
method used here. I suspect that it worked because it was different, not
because it carried a fundamentally strong signal. If everyone did this,
project descriptions would be gamed even more than resumes—it would select for
people who prepared for the selection process² more than anything else._

I find myself almost irresistably drawn towards the conclusion that people
will game the hiring process, _because it is a game_. It has winners and
losers. Everybody is trying to make themselves look better than everybody
else, even to the point of making themselves look better than they really are:
Job candidates, employers, internal and external recruiters, everybody who is
selling a hiring product, etc. Everybody is trying to emphasize their
positives and hide their negatives, while searching for the negatives of the
others.

A drawback to "do your own thing" is that it is enormously inefficient for
candidates, and the results may turn out to be based on the luck of a
particular candidate guessing a particular employer's game is, or being
generally better at games.

I don't know a solution. What strikes me is that as broken as the system
seems, we still manage to hire good people most of the time.

~~~
cbhl
The process being broken isn't a deal-breaker for companies, which can still
hire from the 90% that can deal with the broken system. It's a deal-breaker
for the 10% that are bad at playing the game / jumping hoops / resume writing
/ interviews, because they can't get a job.

~~~
marcosdumay
It's also a golden opportunity for anybody able to hire those 10%.

Maybe most companies shouldn't be different. But there's plenty of value in
not being like everybody else.

~~~
cbhl
Yup -- I'm excited for the Starfighter CTF by patio11, tptacek, et al.

------
soham
Thanks Aline, for yet another well researched article.

I have to say though, that in my experience, these experiments in sourcing
work quite well when your hiring is small. The moment you hit some sort of
scale, it becomes very very difficult, if not impossible to run and rely on
such experiments.

E.g. in the first growth phase at Box, we were tasked with hiring 25 engineers
a quarter. At that scale, the company deals with too many resumes and too many
stakeholders in the hiring process. And at that point, you also have a group
of people explicitly looking at resumes, less involvement from actual hiring
managers, deadlines to meet, land to grab etc. Not saying one thing is better
than the other, just that hiring at scale is an entirely different game.

The other thing, which is implied in the article, but may get lost if the
reader isn't careful: regardless of how a candidate is sourced, the interview
bar still remains the same. i.e. AJ also must have had to clear same or
similar technical interviews like other engineers that got hired there.

~~~
leeny
Hey Soham. We have to stop meeting like this :)

Yeah, scaling this stuff is hard. I do think there's a big danger in scaling
it by offloading filtering powers to non-technical people because then you
have to rely on proxies. Proxies aren't inherently bad, of course, but the
ones we have now (school and past employment) are pretty bad, and if
ultimately, we're getting things wrong more than we're getting them right, it
outweighs the temptation to cut costs and time.

For a company like Box with a super strong engineering brand, it's OK to have
a pretty high false negative rate, of course. You can reject a lot of good
people and still have a revolving door of others who want to work there.
However, smaller companies often take their cues from big ones and adopt the
same processes without realizing that they may not work the same way.

And yes, thank you so much for calling that out. AJ had to meet the same bar
as everyone else. Fortunately, he killed it.

------
roymurdock
So, essentially the company that was hiring (KeepSafe), allowed applicants to
submit highly flexible cover letters, _and then they actually read the cover
letters_.

It would be great if the majority of companies used both the resume and the
cover letter effectively. It feels like most companies that require a cover
letter only do so to screen out the laziest 10% who can't be bothered to write
up a generic 1 page essay filled with ass-kissing and vague jargon.

The cover letter is just a relic from the olden days when the application was
slower and more formal. There were less applicants for each position so HR
probably had more time to read/screen.

This study presents an interesting alternative: Let people submit some text
along with their resume on any topic of any length, and see how their
personality comes through in the writing. Probably wouldn't work extremely
well at a large company, but it seems like it served KeepSafe quite well.

~~~
autokad
like the above commented, people would game the text, hire coaches, etc. I
don't think there is anything fundamentally wrong with the cover letter /
resume process. i just don think companies use resources properly at hand.

I am biased, i think most HR are useless and most tech people aren't great at
picking candidates either. they seem to have their own biases that they cant
get over (for example: putting too much weight on technical skills and none on
soft skills).

honestly, if i were a company i'd send select employees to school. either take
classes or better yet teach a class. you'd get a semester long interview
process and the employee would get something out of it.

------
a-dub
Resumes are fine, it's really in how you treat them. When I read resumes I
generally ignore where people have worked and gone to school and instead look
for what they have done. If there's either a good match between the general
type of stuff they've done in the past and what the role is, or if there's
stuff on there that is interesting enough such that I'd enjoy hearing about
it, I give a thumbs up.

When I interview, I tend to spend most of the time asking in depth questions
about the projects I find most interesting on the resume. What was easy? What
was hard? X sounds like it would be a problem, how did you solve it? What was
fun? What was headbangonthewall miserable? Generally this gives a sense as to
whether or not there's any bullshitting going on, and gives a sense for
whether or not the candidate has a good head for thinking about hard problems.

Finally, I'll ask a few questions to probe for "difficult-to-work-with" red
flags and finish with a few fairly easy "technical challenges" that offer
opportunity for the candidate to either walk about having solved the problem,
or walk away having solved the problem and demonstrated understanding of the
solution from top to bottom.

~~~
__z
What kind of "difficult-to-work-with" questions do you ask? I'd like to filter
out these people but I'm not sure what sorts of questions to ask to suss that
out.

~~~
a-dub
Basically I goad them into complaining about past bad work experiences and
then pay close attention for subtle clues that may indicate that they are
systemically disrespectful or unwilling to compromise on things that don't
seem worth fighting for. Were they bothered by people, situations or outcomes?

Tell me about the most frustrating time when you needed a thing, or consensus
on a thing, and you had to go through way too much to get it.

Tell me about the most frustrating time when you needed a thing, or consensus
on a thing, and no matter how hard you pushed, you never got it.

And, of course: The fastest way to be shown the door around here is to be an
agitator of your colleagues on the grounds of race, sex, religion or any other
attribute that has little to do with work. Nobody here is in the business of
policing behavior and nobody wants to be. This requires perhaps more
discipline than other places as the hammer falls harder and quicker here if
things go awry, so it demands either heightened discretion or a heightened
sense of self awareness and awareness of those around you. Do you think you
would be able to work under such conditions?

~~~
JoshTriplett
I'm not quite sure what you're trying to ask with that last question. I can't
tell if you're saying "you're out if you behave badly" or "you're out if you
complain about other people people behaving badly"; "Nobody here is in the
business of policing behavior and nobody wants to be" comes across as the
latter.

~~~
spacecowboy_lon
That was my thought would you be fired if you stepped in to stop say racist or
sexist behavior in the workplace.

------
codeonfire
When people abandoned resumes like this it's because there is some corruption
in their hiring process and lesser skilled management is attempting to hire
down for political control. Their only goal is to go through the motions and
stay employed. Some people don't want experts. They want to have the illusion
of a functioning business unit. People with no formal training that have
projects that sound impressive to the layperson are not going to make waves or
quit when they find out that management really doesn't know what they're
doing. Just as one can build a model airplane in their garage, those same
people will never be able to build an airliner. It's not a good idea to hire
hobbyists if you're in the airliner business. When you have layperson managers
judging what is a strong candidate and what is important technology, you are
going to get hobbyists skilled in popular tech and your company is going to
get worthless hobbyist tech for your organization. For example, the article
refers to an 'open source Android animation library.' To many people that
sounds like a massive, great achievement. On the tech scale of importance it
is a 0 or 1 out of ten. No business can be formed around an animation library,
it's not a difficult or uncommon thing, and there are thousands of
alternatives.

~~~
codeonfire
In a way, the entire premise of what a software engineer is has been
destroyed. Imagine if mechanical engineers were hired based on their ability
to design a simple bracket. Imagine if electrical engineers were hired because
they amazed management by building a flashing LED circuit. After many years,
the entire criteria for being an engineer is dumbed down to "are you able to
make a flashing led or simple bracket or something equally impressive?" That's
what the people chosen to manage are familiar with. That is what has happened
to software engineering.

~~~
Nacraile
I'm not sure I buy your implicit assertion that other engineering disciplines
are somehow better at hiring than the software industry is. As far as I can
tell, mechanical and electrical engineers are hired based on relevance of work
experience, some discussion of that experience, and maybe a few specific
technical details which should be known by anybody with the experience they
claim to have. All of which is isomorphic to "Worked at Google, could explain
his project in depth, and coded tree inversion on a whiteboard". The only
difference I'm aware of is that a Mech or EE would probably be subjected to
bullshit situational interviews by HR.

This kind of feels like the general "software isn't rigorous like the real
engineering disciplines" angst, which as far as I can tell is also bullshit.
In reality, the "rigorous" old-school disciplines still manage to make
colossal messes of complex, unprecedented projects, in the same way that
software companies often make colossal messes of complex, unprecedented
software projects. Sure, civil engineers can build normal roads and bridges
and buildings reliably, but then again, software engineers have no problem
throwing up CRUD apps and wordpress blogs. There are plenty of bridge
collapses, exploding batteries, stalled tunnel-borers, and so on to match all
of software's spectacular failures.

It turns out that complicated things are really hard to build, regardless of
your field.

~~~
codeonfire
Engineering firms won't even talk to someone claiming to be a ME or EE with no
degree, no certifications, and/or no license. And, the premise of the article
are that they are ignoring both education and experience, so I don't see how
your comparison fits.

This thread isn't about who is more likely to fail. We are talking about
hiring and who is likely to succeed. My point that companies have dumbed down
what they call success so that managers can have jobs has nothing to do with
engineering catastrophes. What is being called a success, I call a failure.

~~~
Nacraile
> Engineering firms won't even talk to someone claiming to be a ME or EE with
> no degree, no certifications, and/or no license.

I don't think software is "dumbing down" by ignoring credentials. We're
ignoring credentials because we are finding that they have no predictive power
whatsoever for competence. Most software companies do still pay attention to
experience (at least while sourcing), on the belief that it is not useless as
a predictor. This article is interesting because it provides strong data
indicating that experience (as presented by the candidate on their resume) is
also useless as a predictor.

The point is that people with good resumes are often incompetent, and
competent people often have thin resumes. This article presents data
indicating that filtering by resume is no better than filtering by coin flip.
Thus, if OP's data and analysis are correct, paying heed to credentials and
experience is irrational.

~~~
jlarocco
> We're ignoring credentials because we are finding that they have no
> predictive power whatsoever for competence.

I disagree with that actually.

It's my own anecdotal evidence, but from what I've seen, people with degrees
write generally better code. They have a better understanding of algorithms,
are more aware that what they're writing in a high level language isn't
running by magic but is being translated into lower level constructs which may
or may not be very efficient, they're more likely to realize there's an
existing algorithm for what they're doing, etc.

I guess I would sum it up that people without degrees tend to work harder and
not smarter.

------
wambotron
I worked at two places where people had resumegasms over "brand name" schools.
Neither of them hired anyone who was ever any better than completely average
(and yes, I include myself).

I don't really care where anyone went to school. It doesn't mean anything.
Really, going to school at all doesn't mean much. I need to see what you've
done outside of that to make any meaningful evaluation. It doesn't matter if
it's a huge project. You can give me a couple 10-line things that do something
useful and I'll still get to see how you name things, format code, use built-
in libraries, etc. Then we can chit chat about project management and how much
you love or hate it.

------
s_q_b
I agree we need an alternative to resume-based hiring, and the hiring process
in general.

For example, I don't do well in whiteboard interviews, which is odd because I
normally don't have a public speaking issue. It feels like there's some muscle
memory attached to coding that isn't well replicated with poor handwriting in
a room full of people.

Whiteboard lines of code are simply not the manner in which developers work
once hired. That is the reason for the disconnect between speaking well about
projects (easy to verbally explain and sketch) and the programming portion
(bizarre.)

Right the industry is doing the equivalent of interviewing lawyers by asking
them to write a legal brief on a white board.

We're testing the wrong thing: a proxy for the work, when we we could easily
test the work itself.

I much prefer work sample tests rather than whiteboard Q/A as it better
replicates the actual job. Give me a few hours with problems I would actually
face on the job, my dev environment, internet access, and a set of problems
that truly reflect the work, and I find it much more natural.

Is it too much to ask that an interview measure skills the job actually
requires, in an environment that emulates the work?

------
appamatto
I think this is very interesting counterpoint to TripleByte's data which
implied that talking about passion projects was lower signal than their coding
quizzes.

The benchmark was performance in a long form coding interview for TripleByte,
whereas Aline's is the final offer, so not exactly apples to apples.

~~~
leeny
Aline here. Fwiw, I don't disagree with TripleByte's findings... anecdotally,
after having interviewed somewhere between 500 and 1000 people in my career, I
think they're right -- I, too, have observed a disconnect between how polished
people sound when they talk about their work and what happens when they
actually have to write code.

What we're actually comparing here, though, isn't coding vs. describing
projects. It's describing projects vs. resumes. And I expect that, there,
resumes provide a lower signal.

~~~
msandford
> I think they're right -- I, too, have observed a disconnect between how
> polished people sound when they talk about their work and what happens when
> they actually have to write code.

I've done a lot of work that people would probably find very interesting and
useful. But I tend to choke on whiteboard code interviews because they're so
high stakes. Any time spent thinking about the problem looks bad, so you have
to talk a lot. But I can't really think and talk at the same time. So I end up
talking rather than thinking and I do poorly.

Now obviously I'm going to push to move the status quo towards something that
doesn't put me at a competitive disadvantage. So we both know that I'm biased.

But the idea that people can talk about what they've done and answer any
questions that you have (about what they did and programming in general) but
still screw up on the actual "coding" part might mean that the part where you
make them write code is more noise than signal.

The problem is that you never find out because if someone bombs the coding
part you simply chuckle and say "well that person is clearly a liar, or
something!" and they don't go any further in the hiring process. So they never
get hired, and because they're never hired, you can't evaluate their work
performance. Which might be excellent when they're not being actively
scrutinized by multiple people all at the same time in a high stakes
situation.

Unfortunately to try and get some objective data on this you'd have to hire
several people who talk about their projects well but don't do well on the
coding part. An understandably impossible task unless your client is a Google
or Microsoft and they know it's just a big experiment regarding hiring.

But until someone does that and reports back (and they won't because it'll be
a competitive advantage) it's tough for me to swallow the "talks good but
can't code so NOPE" that I tend to see bandied about.

Putting someone in a pressure cooker and then measuring their performance will
only tell you how they perform in a pressure cooker. Which is usually quite
distinct from what they're going to do day-to-day.

~~~
leeny
I think what you describe is a real problem and unfortunately one that getting
data around is really tough, for the reasons you describe.

For what it's worth, when I observed a disconnect between how well people
spoke about their projects and how well they coded, it was generally a
situation where someone had perfected a pretty polished self-pitch rather than
a situation where I drilled down deeply into what they had done, asked them
what they'd have done differently if we varied up certain constraints, etc.
And when they fucked up on coding, it was on warmup problems that was
something you'd reasonably expect anyone with some experience to be able to do
(e.g. explain why you might want to use a hash table over a linked list for
certain scenarios, reverse a string in place).

That said, one of the reasons I'm really psyched about interviewing.io (the
thing I'm working on now) is that we're getting a lot of comparative interview
data, i.e. where the same person gets interviewed a bunch of different ways.
Excited to see if we can draw some good conclusions about what works and what
doesn't.

~~~
jrapdx3
Actually, the experiments that need to be conducted wouldn't be that hard to
do. In fact I'd be surprised if research of this kind has not already been
done.

The problem of poor performance under scrutiny, where there is pressure for
superior performance (as in a job interview) is well-described as a form of
social anxiety disorder (or social phobia), a common condition affecting ~10%
of adults in the US (lifetime prevalence). Of course, there's a range of mild
to severe symptoms, nonetheless it affects a significant population.

The implication is that in a not-so-pressured setting candidates might perform
very differently. Furthermore, writing code, solving software problems are
generally incremental processes more akin to watching paint dry than putting
out fires for all the externally visible action there is to see. A
"whiteboard" exam likely isn't a good model of the real requirements of the
job.

There's an enormous amount of research on testing methodology, testing is a
_huge_ industry. Ironically enough, one that is extremely reliant on software
for analyzing test data in order to determine what is a good test of sets of
knowledge or actual abilities. Seems like there's a clue in there somewhere
about how software enterprises could find out who is really good at creating
software.

------
blfr
Would a Northrop Grumman engineer with a GitHub full of cool projects really
be overlooked in a recruitment process? Not once, that can always happen, but
regularly?

~~~
soham
Yes, regularly. Valley has a very short attention span. NG isn't in the list
of hip sexy companies.

Same goes with profiles from Cisco, VMWare, Intel, Synposys, IBM, and pretty
much all the big companies that were pinnacles of business and your career at
one point, but they are not considered hip anymore.

Looking at Github profiles, however much people talk about it, also doesn't
regularly happen. It's a chore to type it out in the browser, if you're
looking at a paper resume or if it's not hyper-linked. And again, if you
worked at NG, what possibly interesting things you could have done? It's not
Pinterest or Uber.

~~~
nether
> And again, if you worked at NG, what possibly interesting things you could
> have done?

Applicant: Well, I wrote a code to optimize a UAV flight path to avoid enemy
radar and minimize fuel consumption, and debugged another controls code to
prevent people from dying. Did some debugging on a computational
electrodynamics code for radar simulations...

Recruiter: Oh. You know what's cool? Photo sharing.

~~~
vonmoltke
On top of that, hope your title was never "Systems Engineer"[1] at one of
these companies. If it was, prepare for a whole lot of clueless recruiters and
managers to misunderstand your experience.

[1] At Raytheon, if you were the one developing radar avoidance algorithms[2]
and code, that was probably your title.

[2] "Algorithms", another word that has slightly different meaning leading to
major misinterpretation.

~~~
Jemmeh
I frequently put my title on resumes as whatever the company I am applying for
is calling it. There's so many names for the same thing. If we call it
"programmer" at this job and they call it "software developer" at the place
I'm applying to-- well, I can read the requirements in the ad. Usually it is
similar enough to not matter. So I just put software developer even if that's
not the title on my paystub. Gets me past computer and HR filter, but when
they call me/past employer no one balks at the slight difference.

~~~
jghn
I sometimes put both, but never the full switch like that. I'm too paranoid
that I'll come across someone who does balk at the slight difference.

~~~
Jemmeh
I've never met someone who was like "Programmer" and "Software Developer" are
too different! Same with "Coder". "Software Engineer" in some countries would
be a no-no but many places it's interchangeable. Further, I've done some
research on the psychology behind it and usually "software developer" and
"software engineers" is a better way to put it for negotiation if that's
possible because you're seen more as someone who makes something rather than
"really expensive computer guy".

~~~
jghn
I've seen companies that are real sticklers for that sort of thing claiming
that if one falsifies that what else are they about? Yes,nit is asinine but it
does happen

~~~
Jemmeh
Well if anyone can be a stickler it'll definitely be that programmer that was
once burnt by a rogue semicolon.

I'm sure it happens sometimes. I just haven't personally seen it with the
roles "Software developer", "coder", and "programmer". That is effectively the
same thing everywhere I've seen it. Now if I'm a programmer and say I worked
in primarily QA or database position we have a problem, but there's no real
nuance between those three titles.

------
fecak
Another great article from Aline. Past employers, schools, and GPAs can
obviously generate false positives. I like this idea overall and would be
interested in seeing the results of others. 400 applications to 1 hire isn't a
great result, and I was somewhat surprised that only about 5% of applicants
were even interviewed.

A few months ago I launched a side project, doing (of all things) resume
review and revision services. When my clients want a review of a resume that I
know won't get results, and I ask "Give me more to work with", the types of
things I hear are eerily similar to the "awesome stuff" quotes in this post. I
try to incorporate those things into the resume when possible.

Is it the resume itself that is the problem, or is it that candidates are just
less inclined to include additional details (that may seem irrelevant) that
could differentiate them from others? Some resumes will list accomplishments
that make it rather clear of their qualifications, but everyone doesn't have
that luxury.

When a candidate doesn't have a long list of work accomplishments, do they
think to include this type of content that might get our attention?

~~~
brettlangdon
I can speak from my own experiences reviewing engineering candidates.

Sometimes when you see someone's resume and they have great work
accomplishments, like "last 4 years at [successful, engineering focused
company], building their [product everyone knows about] platform", it makes
deciding to interview that person fairly easy. However, there is that issue
when someone is either just starting out from college, who doesn't have a lot
of work experience or who hasn't had the best of work experiences (usually not
their fault). For these people the main thing that I look for and try to
evaluate is their ability to learn on their own. The main thing to consider
when reviewing a candidate like that is, if they aren't as up to speed on the
technology we need them to, how long do I think it will take to on-board them
and get them self-sufficient on our platform. If someone does not have the
best resume, but their Github profile is full of projects, even half finished
projects, of them trying out different languages or frameworks or maybe making
(or trying to make) contributions to open source, it usually makes it easier
to say "lets give them a chance," especially if they have been playing around
with the languages/frameworks we use.

Hope that helps answer your question.

~~~
fecak
Agreed that the ones you are definitely going to speak with are easy to
identify. I like to err on the side of giving anyone borderline at least a
conversation or request for more info.

The "especially if they have been playing around with the languages/frameworks
we use" comment hit home a bit. Part of that may be just the asset of exposure
to those technologies the hiring company uses (hitting the ground not quite
running), but I've had a few clients that seemed to use interest and curiosity
in their languages/tools as an indicator of a likemindedness which was viewed
as a strong positive. Particularly in the FP world.

------
Harj
We have the same belief in the limited usefulness of resumes at Triplebyte. We
found talking about projects with candidates both more enjoyable and
interesting than looking at words on a resume. Especially when people are
enthusiastic about what they built.

The difficulty we had was not seeing a strong correlation between talking
about projects and doing well at programming during an interview.

~~~
lk145
What is your programming interview like? If it's whiteboarding algorithms,
it's likely to be eliminating a lot of good people who get nervous, and
selecting for people who are good at studying interview questions and coding
under pressure in an artificial environment. There is good evidence it doesn't
predict job performance well, see: [http://www.wired.com/2015/04/hire-like-
google/](http://www.wired.com/2015/04/hire-like-google/) and
[https://twitter.com/mxcl/status/608682016205344768](https://twitter.com/mxcl/status/608682016205344768)

~~~
Harj
We completely agree. We do all of our interviews via screen share, allowing
candidates to use whichever language and environment they're most comfortable
with. We also give a selection of problems, one of which is algorithm based,
allowing candidates to choose which they prefer.

Ultimately though people are aware they're being watched and assessed under
timed conditions so it's going to be somewhat stressful. If we think someone
is so nervous they're clearly not able to code at all, we'll offer a take home
test as well before making a final decision.

~~~
tjradcliffe
As a matter of interest, does your company also ask managers to go through a
management test as part of the hiring process? Let them run a team for twenty
minutes or an hour and evaluate them on that basis? If not, why not? :-)

I think "take-home" style tests are the only reasonable way to evaluate this
kind of technical competency. They are the closest to realistic. Throw people
a small problem of the kind they'll actually be asked to work on. Let 'em deal
with it--including documenting their solution!

------
vonmoltke
> While AJ’s government work experience gave him a good amount of cred in the
> public sector, he found that making the move to industry, and startups
> especially, was near impossible. It wasn’t that he was blowing interviews.
> He just couldn’t get through the filter in the first place.

...

> It was AJ, a candidate that Zouhair Belkoura, KeepSafe’s cofounder and CEO,
> readily admits he would have overlooked, had he come in through traditional
> channels.

This was the story of my job search three years ago. It still kinda is.

~~~
pzxc
I'm going through this right now as well. I ran my own business for 7 years,
making content websites monetized through google adsense, building games, and
freelancing, among other things. The primary source of revenue was adsense,
and it paid the bills enough for me not to need a "real job" for years. 18
months ago, the adsense revenue started to dry up, and in an effort to make
some additional income, I started looking for a normal job again.

I ended up reluctantly taking a job working for the State of California,
mostly because the schedule was flexible and it was a 5 minute commute. In the
last 18 months I've gotten two promotions, including one four months after I
started which is unheard of in state service (and which my bosses had to fight
HR to get). I spearheaded the acquisition and implemented of a version control
system, which no one hear had ever used (they were just FTPing files all the
time, risking overwriting of others' work etc), and got buy-in from all the
developers who now say they couldn't imagine working without it. I also now
wear a variety of hats besides programming including sysadmin work, dba work,
architecture design, etc. My bosses rely on me more and more everyday just for
my opinions and advice, let alone the work I do.

Yet now, when I send out my resume, it's almost always crickets. A year and a
half ago, when the first item on my work experience list was "failed
entrepreneurship", the response rate to my resume was about 80% (not even
talking about interviews, just getting a response at all). Now it's more like
20%, all because the first item in my work experience section is my current
job working in public service.

I admit, many of my coworkers probably deserve the reputation that public
sector work has. A significant number of them are clock-watchers that the
bosses don't even try to assign anything important to, because they know they
don't give a shit and can't easily be fired (union), they are just filling a
chair waiting for their public pension to accumulate over 20-30 years.
Nevertheless, I also list on my resume all of the above, including the
pioneering (for us) work I've accomplished here implementing version control
etc. I never intended to stay here more than a couple of years, but I also
never intended for this job to have such a negative impact on my job
prospects.

I think I am probably going to revise my resume to leave this job off
completely, and just say I've been working for myself for 9 years instead of
7. I predict sadly that this will return the response rate back to what it was
before I had this job.

~~~
sageabilly
Depending on how you write your resume, I think that your public sector job
could be a draw for smart employers. Hell, if you pitched it on your resume
the way you pitched it here I'd absolutely want to get you in and talk to you
about how you A) handled the failure of your business, B) pivoted to having a
real job and C) overcame the crazy obstacles in public sector work to make
real impact.

The fact that you can and did push change in public sector is huge, HUGE to
anyone with half a brain. It's all in how you tell your story.

------
sjtgraham
I screened and interviewed a lot of developers in my last position and it
stood out to me that résumé quality seemed to be inversely correlated with the
candidate's actual ability. I distinctly remember the absolute best developers
that I hired also had the most atrociously bad résumés. The candidates with
résumés that literally almost knocked me off my chair failed miserably at the
most basic programming task.

~~~
rokhayakebe
Frankly I cannot see an expert in anything other than marketing sitting down
and crafting their resume. "Appearance starts where performance ends."

Let me illustrate:

Growth Engineer One line Resume: "I was employee 7 at Snapchat when we had 6
engineers and 400,000 users. During my tenure as the only growth engineer, our
userbase grew to 10 million users over the next 6 months."

SEO resume: "I joined XYZ when it was ranking at page 10 for major industry
keywords. 9 months later. Google the following keyword BDHDUYD, which accounts
for 40% of your market. If you find the company in the top 3 results, we
should schedule an interview."

Software engineer: I do not know, but I am sure you can insert a short
paragraph here.

~~~
rspeer
I am not really sure what you're illustrating with these examples. But I'm
guessing you're saying that anybody worth hiring will have a stellar
accomplishment where you can look them up by name.

This is really not the case.

I've found that some people's greatest accomplishments are trade secrets that
they can't show you. Some people's greatest accomplishments are in hobbies you
don't quite understand. Some people are good at things even though they don't
have a heroic narrative of great accomplishments building up to their job
application.

Meanwhile, people who come in to an interview boasting "Look at this! I did
this!" are often just taking credit for what their co-workers did.

~~~
vonmoltke
> I've found that some people's greatest accomplishments are trade secrets
> that they can't show you.

My greatest accomplishments in the domain I _want_ to work in are trade
secrets of the US Government. :) I can talk at length about my current domain,
though, which I want to get _out_ of. :P

------
tempestn
> Resumes don’t have an explicit section for building rockets or Minecraft
> servers, and even if you stick it somewhere in “personal projects”, that’s
> not where the reader’s eye will go.

That's true, but a good cover letter can go a long way toward helping with
this. _Most_ cover letters are generic, bland, and obviously copy-pasted from
a template. (Or more often from a previous application, sometimes with info
about the previous company left in!) A cover letter that talks about something
exciting you've done recently, and ideally how it might be related to the job,
or even just how it demonstrates skills you'll use in the job (and describes
exactly how), is _awesome_ in comparison. A letter like that would absolutely
get you an interview with me, almost regardless of experience. One of our
current co-op students actually had almost no programming experience on paper;
he had actually switched out of a theatre degree iirc. But his cover letter
was awesome (the theatre degree probably not being coincidental). Got him the
interview, which got him the job, and I haven't regretted it. Just a co-op of
course, but the point stands. The cover letter is probably the most important
part of your application. Take the time to write a good one.

------
andrewstuart
Everyone should just give up on trying every sort of new angle to "identify
great developers". It's purely subjective, there is no meaningful universal
definition.

It comes down to whether or not the people doing the recruiting all have the
same subjective opinion.

------
sopooneo
All these discussion of how hiring is broken. Doesn't this imply an enormous
market opportunity for the recruiting agencies or companies that can properly
capitalize on either undervalued candidates or better knowledge?

~~~
shanemhansen
Recruiting companies aren't in the business of finding good candidates, they
are in the business of finding candidates their clients like.

Companies have the same problem the military does. They hire a certain
stereotype because they've always hired a certain stereotype and the people in
charge match that stereotype.

If there's any opportunity for disruption, it would be at tiny companies where
there are people that 1) just need talent and 2) know that they process that
made them successful is broken.

------
caublestone
We use an alternative screening process where we prompt prospective
interviewees with 2 extreme customer service complaints (this product sucks,
it didn't arrive on time, i hear it's poison etc.), ask them to provide an
answer directed to the customer and create a plan to prevent the issue from
happening in the future. It's fascinating to see how people approach the
problem of making people happy now and preventing dissatisfaction down the
road. If they pass (1/10 do), we use a 30 minute phone call to identify
interests and motivations which ends up being the strongest indicator of value
add. This might work for B2C companies only but I'd love to see any company
identify people that are truly passionate about making people happy.

Edit: Clarifying that this is for non-engineering roles.

~~~
kansface
I don't suspect strong correlation between the desire to make customers happy
and productivity writing software - maybe you can tell me if my intuition is
correct.

~~~
caublestone
Ah, I meant to say we do this for non-engineering roles. For engineering we
provide coding challenges as a pre-screen.

------
dropit_sphere
Is it just me, or does this still seem like a crazy hard problem? There were
still __415 __not-resumes to wade through for one offer.

------
nickpsecurity
Great article. A good move. I think one of the reasons the method is
successful is that it asks applicants to keep it real. The traditional
channels want people to come off a certain way. People also know about their
filtering rate. So, the incentive for them is to tell companies what they want
to hear and in a way that conveys unreliable information.

Seems your example changed the incentives, got useful information in return,
and that led to a positive result. Unsurprising in hindsight. I'm going to
send your article to a few people to see if I can get any to try that
approach.

------
vultour
Something else piqued my interest in this article.

This lady claims that the company is fighting for candidates with Google,
although the only thing they do (if i read it right) is provide an encrypted
version of Dropbox. How does this require world class engineers? I've coded a
file syncing app quite fast as a personal project once, and I don't think I
could call myself even a regular developer. I do not believe such application
would be even remotely as complex as anything Google does.

------
7402
I've always found resumes to be quite useful in figuring out who to bring in
for an interview - BUT, most of the people I've been involved in hiring have
had between 5 and 15 years of experience, usually at at least 2 or more real
companies. I can well believe that if you're only interested in people right
out of school, then it's harder to figure out candidates from their resumes.

I am a little puzzled, though, about why others seem to find resumes so
opaque. It seems like resume-reading is a lost art. A resume is usually a
document that someone has spent a lot of effort on to make themselves look
good. If you learn to read them, that can tell you a lot about the author.
(Note: searching for buzzwords is not "reading.") A resume should not be
regarded as simply a collection of facts - of course you'll be misled if you
do that; a resume should be regarded as a document of self-expression. After a
while, you can see useful patterns in what people put in resumes - a least for
more-experienced applicants. Almost every resume suggests a bunch of next
questions, which can be asked in a phone screen or interview to get a pretty
good idea of what a person is about.

It's worth recalling that absolutely all software engineers at all software
companies in the world from the first ones around 1955 up to 2002 were hired
without benefit of LinkedIn, StackOverflow, Github. Almost all of these
engineers submitted resumes, which were reviewed prior to offering interviews.
Yes, there were hiring mistakes in the old days, but I don't see a huge number
of people taking about how the hiring process now is so much easier, smoother
and more foolproof than it used to be.

------
bsder
> His GitHub, full of projects spanning everything from a Python SHA-1
> implementation to a tongue-in-cheek “What should I call my bro?” bromanteau
> generator, hinted at a different story, but most people never got there.
> While AJ’s government work experience gave him a good amount of cred in the
> public sector, he found that making the move to industry, and startups
> especially, was near impossible.

Huh? The companies this woman hires for don't look at github? Not looking at
public code that someone has published is more broken than relying on resumes.
If someone has published code and it doesn't suck, I'll probably bring them in
for an on-site, period. I may even tell them that "We're going to talk about
"file foo.c in your code where you implemented feature Z. So be prepared."

And, I suspect with startups it was more a case of "How many years were you in
government? That would makes us so unhappy that we would leave. Why didn't
you?" That's a different way of asking "Is this really the place for you?"

As a hiring manager in a startup, when I knew I only had 9 months of runway
without more funding, I'd feel _REALLY_ bad about taking someone with a family
away from their very stable job. As someone who has recruited employee single
digit, I often have made a point to meet the family when recruiting someone--
even if I have to fly to them. I need both the prospective employee _and_
their partner to understand that the big probability is that the company _won
't be around_ in 24 months, there won't be any payoff, and a new employment
search is likely to be the result. Yeah, there is a small probability that
we'll survive and an even smaller probability that we'll get some money. It's
a really delicate balance for me, at least, to properly sell the company
(Startup! Options! Novel!) and reality (Bankrupt! Flameout! Layoffs!).

I'd say I'm batting about 50%. For every employee I scare off, I absolutely
convince one to join. Funnily enough, every single one who didn't run away
said the same thing: "My wife told me I had to work with you." They were
stunned that someone so important (Hah! Management in a startup is a good way
to understand how _unimportant_ you are really quickly ...) would take the
time to make sure the family was informed _properly_ about the risks and
rewards.

------
chris_wot
My submission would be:

 _I was a help desk pleb at a well known inkjet /scanner/camera company 15
years ago and this company "extended" their clipper database to record third
party cartridges, but recorded them in .ini format. That's right, one file per
record, in key=value pairs. I was bored and accidentally mentioned to the guy
whose job it was to copy and paste the data from each of all 90,000 ini files
into an Excel spreadsheet that Perl could do it, and I'd even use references
to hashes to do it. He had no idea about that last bit, but I did it for him
on the proviso that he didn't tell anyone, and reduced 10 weeks of work to 30
seconds. They unfortunately made me employee of the quarter but neglected to
tell me so I missed my awards ceremony._

------
AndyNemmity
I just went through a job search, and never created a resume. I only submitted
my linkedin. If anyone required a resume, I immediately responded that we
weren't a good culture fit.

Worked out really well. Not sure it's to be duplicated, but for me it went
fantastically.

~~~
puredemo
What if I've never had a linkedin because of all their repeated data security
issues?

Your comment sort of sounds like a shill post, fyi.

------
Mimu
I fail to see how someone building games at 14 or own one of the most
successful Minecraft server could not pass the resume filter.

Not saying the process described in the article is bad, even though I believe
anything can be gamed, but I don't really see a big difference. Main change is
the way recruiters looked at what they got, resume or essay wouldn't have
change a lot I think.

Maybe off topic but if companies want the best people, maybe THEY should write
the essay explaining why people should join instead of sitting in their high
tower waiting for minions to come.

------
pbreit
What surprises me is how many people dislike cover letters. When I go through
applicants, a decent cover letter demonstrating some enthusiasm about the
company and a unique point or two is appealing.

~~~
m3talridl3y
It surprises me how many employers think that any of their engineers give a
crap about the actual business. The better ones have some form of passion for
creating good code, the actual business is an implementation detail.

~~~
vonmoltke
The domain, product, or use case is what draws me in. But then, I'm of the
"classic" engineering mindset; a maker. Code is a means to an end, not an end
in itself.

------
lordnacho
The "new hiring process" experiment comes up here quite often, and I do
appreciate it.

However, how can we conclude anything from a procedure that only examines the
hired population and none of the unhired?

~~~
fsk
The only way you can truly be sure of your hiring process is to pick a random
sample of people who fail, and hire them anyway.

I see a couple of issues with this:

\- Could it get you in legal trouble?

\- Are you capable of evaluating who are the best employees AFTER they are
hired?

\- Only a large corporation would have the resources to gamble on hiring a
random sample of people who failed their interview process.

Here's one half-hearted way to do it. Pick a random sample of resumes you
reject, and give them phone interviews anyway. Pick a random sample of people
who fail your phone interview, and give them an on-site interview anyway.

------
dools
I do something very similar in my hiring processes on odesk. I ask each
candidates opinion on something (articles usually). This allows me to
eliminate 90% of applicants. Then I review the 10 applicants who's answers
weren't complete gibberish and give the best 2 or 3 a programming task. Works
pretty well, I've only ired one guy turned out to be inadequate for the role
(out of about 20 hires over the past 4 years for php dev work).

------
sytelus
You can do much more easier, automated filtering: Ask candidate to submit link
to any of the followings:

1\. Github a/c

2\. StackOverflow a/c

3\. Their blog

4\. Anything they made online

If candidate fails to submit link for any of above then just don't interview
them. I would guesstimate this simple check filters out 70% of the junk resume
and probably 20% of the good resumes. It can scale like crazy and expanded
even more (for example, use APIs to get their profile information and rank
resumes).

~~~
throwaway12309
* 20+ years of development experience 15 of those professionally * published author of various books * worked in systems that most startups would shit their pants with the requirements * some of the big name companies have tried to poach me based on the products I've worked * used professionally: C, C++, F#, C#, Ruby(Rails), Swift, Obj-C, Kotlin, Haskell, Scala

I would never be called based on that criteria as I have no inclination to
spend my free time doing stupid shit online for hipster new developers that
think GitHub is the end-all.

ps: to be fair, I probably wouldn't want to work in a company that has this
mentality, so maybe that really does work

~~~
Cyph0n
So you're saying that the people who spend their free time developing projects
that you most probably use in some form or another are doing “stupid shit”. I
wouldn't hire you either.

~~~
throwaway12309
No, I mean stupid shit for people that think looking at a GitHub page is all
they need to know about a developer. I'm happy folks work on Open Source
software and Github has helped a lot of these projects with visibility and
infrastructure, but as the gp basically stated: "you either have a web
presence or we don't interview you", then yeah, I don't care about that and
having been on the interviewing side a lot, 95% of the github links in resumes
(and yes, I check them) are crap and I can honestly said, done so they would
have something to show and not because it had any real value (sorta like the
4043th JS framework, you do it because it brings you a bit of status, not
because you are actually solving a problem or improving on a solution)

------
ageofwant
This is very bad news. I have always highly valued resumes as a very effective
candidate filter. It works as follows: I take the pile of resumes, shuffle
them thoroughly an divide roughly in half. The pile to the left goes in the
bin. I repeat this process until I have the luckiest candidate's resume in my
hand. This is the type of guy I want to associate with: one on whom fortune
smiles, repeatedly.

------
ammon
Interesting post. Talking and writing about yourself well, even when not
matched with programming well, probably helps get job offers (and in many
cases helps be a good employee). It's difficult to tease these two thing
apart. I imagine looking at speaking and writing can still miss great people
(as can every filter), but I can believe that it's much better than looking at
resumes.

------
moises_silva
GapJumpers seems like a good alternative to resume screening:
[https://www.gapjumpers.me/](https://www.gapjumpers.me/)

Even if not using GapJumpers itself, you can follow the concept by requesting
solving a problem or submitting a piece of original technical content along
with the resume.

------
ilaksh
I think allowing people to submit a link to a demo or portfolio makes a lot of
sense, especially since everyone has computers and the internet now. Welcome
to the amazing new world of hyperlinked multimedia, hiring people.

------
dzhiurgis
So you've turned down 399 people to employ 1. Not sure if that is any
different when employing via resume, but that sends shivers to my spine. 399:1
ratio says to me that there is oversupply of engineers.

~~~
dsymonds
It means there's an oversupply of people who want engineering jobs. Most
people applying for software engineering positions aren't able to program even
the most basic of things. That doesn't mean there's too many qualified
engineers.

------
paulgayham
For what position? How can you judge candidates without a position to judge
how well they fit?

I can just see this guy going out to Web devs, System devs, DBAs etc. and them
all disagreeing because they're looking for different things (and value things
differently).

------
mpenn
The writer claims to "rely heavily on data," but the punchline of the article
is purely anecdotal that 1 person at 1 company got hired, was good and would
have been overlooked. I am sure there were also many candidates with good
resumes who were now overlooked.

This article starts out with an air of science and ends with a completely
unproven conclusion.

While I do agree in my gut that resumes are not an amazing filter, she has
completely failed to present evidence that her alternative interview process
is better.

And in fact, while KeepSafe still has the no resumes option open, they are now
accepting resumes again -- I do not great confidence that the alternative
system was anything more than a PR move by the company.

~~~
roneesh
Shortly after she says she's come to "rely heavily on data", she says "This
post, however, is going to be a bit of a departure. Rather than making broad,
sweeping conclusions based on a lot of data points, I’m going to narrow in on
one story that happened".

I think she did a great job of doing exactly what she set out to do, and since
this is just one anecdote, any qualitative or numerical data she presents
won't be worth much, all the more reason to omit it and just share the story.

~~~
mpenn
My bad on that -- I definitely misread the beginning. After rereading, she
presents the beginning and end fairly based on an anecdote and not data.

I'd still love to know why the company didn't switch to this full throttle.

