
Abolish The Reference Check - nireyal
http://www.nirandfar.com/2012/04/abolish-the-reference-check.html
======
onan_barbarian
There are many, many bogus things about this post.

First of all, I'm ready to stab myself in the eye every time I have to see
someone they only hire "A" players. This is getting tiresome and unbelievable
- I don't know of any startup that has nothing but "A-player type work" to do.
Almost every plausible startup has some work that will be comfortably
addressed by a range of abilities and the idea that you're going to stuff your
startup full of absolute super-stars (and do well as a result) is just
posturing.

I am also not entirely sure that someone whose references are, I dunno, busy
doing actual work or "on holiday" deserves to be penalized with respect to
someone whose references are all hanging around in their office with nothing
to do but promote the living shit out of each other in some kind of mutual
admiration society.

Speaking from experience, I have seen many such cliques of utterly mediocre,
self-serving people who would unhesitatingly tell happy stories of how
wonderful everyone in their clique was to anyone who asked - even if a more
conventional response from further up the management chain would have given a
very tepid review of the actual performance of said people.

Finally, if you're going to contact a bunch of people out of the blue, adding
self-promoting line about your blog is not appropriate. Under the signature
line, maybe, but putting it the body text is a dick move.

~~~
incongruity
Yes. Thank you. I'm glad I'm not the only one who wants to retch when I read
posts like that. The underlying ego and assumptions are laughable, at best.

Beyond that, however, the poster needs to learn about ROC curves
([http://en.wikipedia.org/wiki/Receiver_operating_characterist...](http://en.wikipedia.org/wiki/Receiver_operating_characteristic))
– for just a moment, let's assume we're genuinely only hiring "A" players –
then the challenge is a signal detection problem, pure and simple – sorting
the signal from the noise and avoiding both false positives and false
negatives.

While I have _huge_ issues with the post's method, it's clearly only focused
on avoiding false-positives – avoiding hiring someone who's bad but is sold as
good/great. Unfortunately, there are real issues with false negatives as well
– you may miss out on a fantastic candidate, first and foremost...

But, of course, decisions aren't that simple. Anyone who understands decision
making knows that decisions depend highly on context – including the context
provided by the other options... and if you systematically fail to account for
eliminating some number of options, you may well be (foolishly) limiting the
range of options and the context of your decision. In short, "A players" are
relative...

And that idea goes even further. Many stellar employees, innovators, coders,
etc. don't start out that way. Sure, some do, but those aren't the people
applying and giving you references.

Your job, when hiring from a resume pool, is precisely to find those diamonds
in the rough – those false negatives.

~~~
mwd_
This is particularly interesting because it is common to hear complaints that
(A) it is hard to find "good" people, and (B) it is hard to go through huge
quantities of resumes. If those are both true then clearly it's time to worry
about false negatives. Maybe failing to hire isn't such a big deal? Maybe
there's an element of bravado in claiming that nobody's good enough? Or is a
lot of tech hiring just horribly misguided? I'm not sure.

Personally I've got to say that I also find the "rock star" rhetoric to be a
huge turn-off. When I see it in a job ad I think of how horrible it would be
to have to work with a club of self-described "A" players.

The gruelling interviews are another perverse system that I think has a
negative effect. If you can easily get lots of interviews are you going to
bother with the five stage interview with four hours of quiz questions and
"homework"?

~~~
incongruity
Well – the one thing I'll say is that interview processes like that screen out
people who are both good and know what they're worth – which is one of the
subtle subtexts in much of the hiring processes being discussed/described here
– everyone wants the _cheap_ rockstar – not the rockstar that expects rockstar
prices and rockstar access... or well, few people do and those people don't
use these sorts of hiring mechanisms.

------
hncommenter13
A long time ago, I was a VC/growth equity investor. Reference checks were
_incredibly_ important, as a startup's chances of success, are, on average,
very much correlated with the strength of the team in a variety of areas.

First, obviously, we were looking for red flags. We had ways of phrasing
questions designed to get people talking (esp about negatives). Also, more
importantly, we were looking for names/contact info of other people who worked
with the person we were checking that weren't on the reference list they
provided to us. We'd do the same thing with customer references.

Second, we didn't know most of the people provided as references. A random
stranger's opinion is interesting, but not that useful, especially when
phrased in terms of "good/bad" or "exceptional/not exceptional." What was more
valuable was to find out what _kind_ of person we were dealing with, which is
something that you really can learn from a few random strangers. For example,
in past lives, has the CEO been an effective salesperson? Does s/he need a
strong COO because s/he tends to let details drop here and there? The line of
questioning went something like: "If you were going to build a team around
this person, who else would you want on it? What qualities would the other
team members have?"

In other words, references are just another data point, and I believe their
value is not a binary determination of that person's ninja-rockstar-astronaut-
whatever-ness, but a more nuanced read on that person's abilities as they
contribute to the team overall.

------
j_baker
Is it just me, or does the person who wrote this sound like a snob?

* "In a startup, you can’t afford to hire B-players."

* "...a technique I’ve come to call it the “average-need-not-apply” method."

* "You saved yourself from hiring a B-player, or worse."

* "...employers can make better hiring decisions and make the hiring process better for everyone involved – except maybe the B-players."

~~~
edandersen
This is the language of "TopGrading", a hiring methodology.

~~~
amcintyre
Oh dear. Is this a thing now? Thanks for the heads up...looks like I have
another set of snake-oil buzzwords to watch out for.

------
jsight
I'm not really fond of references for a couple of primary reasons:

1) IMO, they are relatively unreliable indicators for technical candidates

2) They leak the candidates active job seeking status to potentially untrusted
colleagues

1 occurs primarily because references are self-selected to be people that the
candidate has friendships with rather than people who will give the the most
honest assessment. In large companies, this can often be especially easy for a
candidate to game. The technique in the linked article will do nothing to
resolve this, IMO.

And 2 is made worse by this person's desire to interview people who aren't
even listed as a reference by the candidate. I've worked for companies before
where I would have had to turn down a restriction like this.

------
PaulHoule
The obsession with finding super-employees is the flip side of the obsession
with finding the perfect process.

Does pretending you hire only exceptional employees help you believe that
you're exceptional? Or does running an exceptional workplace help you hire
exceptional employees? Would an exceptional employee REALLY be able to make a
difference where you work? Or would they just ground down by an ordinary
organization?

~~~
xsmasher
My definition of "A" people is "can do the job without being taught how to do
the job." This is a necessity at a start-up where there is no one else to
teach you how to perform your role, because your role is new.

"B" people are fine for large organizations where there is someone to show you
the ropes, but smaller orgs need people making fast and correct decisions
without organizational tradition/experience to support them.

------
hapless
This technique exposes you to a new pool of mediocrity. Sure, you're
guaranteed _exceptional_ candidates, but what how are they _exceptional_?

Did they, in the past, work for _exceptionally_ abusive employers who need
them back? (The new guy quit after his fourth month of unpaid overtime!)

Did they work for _exceptionally_ dimwitted individuals who can't measure
their new workers? (The new guy doesn't understand the old system! Fred said
it was SO EASY!)

Did they do _exceptionally_ niche work, where almost no other candidate will
do ? (e.g. managing physical linotype inventory -- no one ever got our 'e' and
't' counts in the same place as Fred! His database management was magic!)

Using former bosses as an uncalibrated barometer isn't very much better than
phoning them up and trying to draw inferences from their tone of voice.
Garbage in, garbage out. (But I bet it saves time on collecting your crap
data.)

------
candeira
My answer, not really, but what that part of me that flies off the handle
feels like answering:

Dear interviewer,

This is _not_ a reference. I don't respond to douchebags with manipulative
reference requests.

Thanks for letting me know that Jane Doe is in the job market. In the case
that Jane is indeed exceptional (and she might be for all you know, or you
wouldn't be annoying me at my place of work), I have done the following:

I have just written to all my friends in a position to hire developers to let
them know there is an exceptional candidate doing the rounds, and they should
call her for an interview asap.

In fact, I may have done this even if I don't think Jane is exceptional. By
definition, only a few people can be exceptional. Some developers are just
solid and dependably conscientious, creative and productive, and I also hire
and recommend those whenever I can, because in doing that I help two people:
the developer and the employer.

Which is also why I would never give you references for good people. The
industry needs good people, and good people need good places to work. Which
isn't your shop. Good developers deserve better than your manipulative sports-
metaphor elitism.

Regards, etc.

[edit: took out an unprofessional bit of invective at the end]

------
vellum
This method falls victim to self-selection bias. Consider this scenario: Tom
is an mediocre developer but he has a great personality. People who don't like
him or his work won't answer. But his friends will.

------
michaelochurch
This deserves a downvote.

Horrible idea. Most companies have their policies of not giving references
either way. Lawsuits over good references (for people who turned out not to be
good) have occurred, and even though the probability of a plaintiff victory is
low, who wants to deal with that pain in the ass? Most people are fine saying
that someone was essentially decent and worked hard, but aren't going to go
out on a limb to say that someone who no longer works for them is
"EXCEPTIONAL".

Also, what if the person doesn't remember his boss's email or gets it wrong?
What if his boss is three companies away by now? Lots of false negatives here.

I think reference checks are not essential for junior hires-- not having
references is a warning sign, but I'd take a chance and decide fast on someone
if necessary _if_ he wasn't going to be in a managerial role-- but I'd require
references from past subordinates (2 per company) from anyone being hired into
a managerial role. I actually think subordinate and peer references are far
more useful than managerial ones: if you want to see someone's true character,
ask people he had power over.

~~~
SpiderX
It was written by some dude who "sold two companies". Quite out of touch with
the working man who needs references. Safe to ignore in that case.

~~~
michaelochurch
Oh, come on! If you're just going to be an employee, possibly _for ten years_
making money for other people, instead of taking some initiative and having
your dad set up a lunch between you and the VCs he prepped with and having
them teach you how to work for yourself and join Those Who Have Completed An
Exit, well then... then you deserve the mediocrity you get!

------
droithomme
Ah the good old stalking technique. I remember the first time a potential
employer tracked down and telephoned my mother.

Sure that would have been OK if this was an FBI background check. But it
wasn't. So the answer to the job offer was "Sorry friend, but go to hell."

------
ghshephard
I think most employees (and perhaps many managers) fail to understand the
purpose of a reference check. It is _not_ to find out whether the employee
will be any good - that can only be determined by observing the employee in
your new workplace for some probationary period.

The hiring decision is traditionally made on the strength of a hiring-
manager's conviction and only the hiring manager - the interview pool can
inform, but never make a decision. I've seen many situations in which the
entire interview group said no to an individual who was hired, and became a
star player. I've also seen scenarios where everyone said yes, and the
employee was terminated for non performance within a year.

The interview is just a numbskull filter - you usually can't identify great
people, but usually (particularly for technical positions) - you can eliminate
the people who don't know anything.

So, what then, is the purpose of the "Reference Check" - it's to determine if
this employee you are hiring burned down, assaulted, or otherwise was
convicted of some horrible crime at a previous company. In general, with
larger corporations, all you'll ever get out of them was "He was an employee
from XXX to YYY in the position of ZZZ" - but, if he was terminated for
killing a coworker, you might discover he was terminated for cause. (unclear
whether they'll tell you _what_ cause, but do you really need to know?)

I've done over a 100 reference checks for employees - and, over time, I've
realized that's all I'll ever get. And, when an employee does not give you
_any_ prior bosses (particularly if they've had a lot of them) - that's a
warning sign as well, that you are allowed to discriminate on in at least
California.

------
Navarr
Am I the only person here that is offended by "Don't let the candidate return
the call" or was that meant metaphorically?

I might not be "A-level" yet but as a potential candidate I would be furious
if my employer kept my options away from me, especially if he's likely to toss
me away if everything isn't going perfect for him, as the language of this
post seems to dictate as a very real possibility.

~~~
plorkyeran
I assumed that meant "Give a good enough offer that the candidate doesn't
bother returning the other call", as a truly exceptional candidate is likely
to say no if you press them to decide immediately.

------
rdl
I'd probably ignore the mail specifically because of the marketing effort for
the blog at the bottom (and would have already been a bit hackle-raised at the
attempt to define exceptional as top-10%. One, that's wrong. Two, it's
patronizing.)

There is something reasonable related to this, though. Let people recommend
with different weightings. Say, a way to say "no, this person is a bozo"
without directly saying it.

It's like you could come up with a simple web UI to do this. Assign -1 to
"this person is a bozo, avoid", 0 to "I am only doing this out of social
obligation", 1 to "weak rec", 2 to "good, competent", 3 to "indeed, far better
than average", and 4 to "you are an utter moron if you don't hit this."

Maybe add a comment field at the bottom.

The key is to give the recommender a social out -- they can tell the candidate
"yes, I recommended you", even if it was a 0 or 1 (or, in some cases, a -1).

Facebook and Google do "recommend" (no-op) vs. "strongly recommend" (actually
significant), but adding more levels is good.

------
acgourley
This seems like a good strategy but I would only try it if you had a few
different good references to check. The danger that the email gets ignored
because the person let it slip through the cracks is fairly substantial.

Maybe it could be improved to be super short and only ask for a simple "yes"
answer so they can answer it right away instead of flagging and moving back to
it.

~~~
michaelochurch
I wouldn't respond. I would just find it douchey.

You should already be convinced that the person is exceptional (in talent) or
at least good enough to hire before you start messing around with references.
The purpose of a reference check is to make sure the person isn't either (a)
lying about his background, or (b) seriously harmful. For (b), I don't think
references are the solution but I can't think of anything better.
Unfortunately, there's no character test that can be done in a few hours of
interaction.

~~~
backjudge
Not only would I not respond, I'd let my (former) colleague know what kind of
potential employer (s)he was considering.

I don't know that I've encountered such an ill-conceived idea in quite awhile.
Forget all the false negatives this method would be sure to "detect", I
wouldn't feel comfortable spamming the candidates contact list.

~~~
michaelochurch
Yeah, all this nonsense about "we don't hire B-players" is ridiculous.
Narcissists always think their loyalists are the best people in the world and
everyone else is an idiot. Classic sign.

If we're going to ignore the douchebag baggage ("douchebaggage"? 'tis a new
word) associated with "A-player" and "B-player", I'd say I _really_ want the
person who turns B-players into A-players.

------
sopooneo
I've done some hiring of math tutors for a small tutoring company I used to
run and I was extremely diligent about reference checking. It made a big
difference. I never got a bad reference, but I got some where there were
pauses on the other end of the line that made me curious. I was naive then and
hired the people anyway. In one case it was worth it, but in both cases, it
became clear early on what might _give pause_. Other than that it helped
because a lot of people with cheerful demeanor and astonishing resumes simply
disappeared when it became clear that I really did make calls and verify every
degree they listed. That happened with _a lot_ of people.

------
tnash
I'm not sure this is the solution. The problem is hiring (or accepting for YC,
College, etc) in general is a complicated process where the generally accepted
solution doesn't work as well as we want it to.

The only real solution would be to get to know the people better, which takes
too much time. We resort to things like the reference check because it's the
only option that doesn't require massive amounts of time. Unfortunately there
is no substitute for the time it takes to get to know someone.

~~~
benaston
Companies complain and complain about the hiring process, with people like the
OP even resorting to subversion in an attempt to "de-risk" the process.

"The only real solution would be to get to know the people better..."

I totally agree. Read the resume/CV, run the technical test (if required),
conduct the interview. If they seem good enough, hire them.

All the jobs I've ever had (perm and contract), had a probationary period
during which I could be fired at will.

If a company cannot handle the financial strain of hiring and subsequently
firing someone within the probationary period then perhaps they shouldn't be
hiring someone in the first place?!

------
veguss
Hire base on employee referrals have always worked the best.

------
georgieporgie
_Congratulations! Looks like you found a star_

Uh, why are you looking for a star? Why does everything think they need or
even want a star? How about just someone who gets things done?

As for reference checks, they're a great way to assure that a past, toxic work
environment will continue to haunt a candidate indefinitely.

This article is just more of the "I eliminate _X_ out of _Y_ candidates, so I
know I'm hiring the best!" B.S.

