
We can't judge another programmer's abilities in a 60 minute interview - passenger
https://www.linkedin.com/feed/update/urn:li:activity:6528766848526278657
======
andrewstuart
I am both a recruiter and a highly experienced developer and software
designer.

I once sent one of the very best programmers I know for a job interview.

I have met a small handful of outstanding programmers in my > 30 years in IT
and this guy was in the top five, near the top.

I had personally worked directly with him for 5 years and he was
unquestionably one of the most talented programmers there is, and a nice guy,
easy to get along with and great to work with. Incredibly productive, and
capable of the most incredible feats of software development.

The client interviewed him and said ....... no thanks.

There you go - I have just explained everything you need to know about
technical recruiting. Give it some thought - the more you think about it, the
more it reveals the deepest truth about YOU and your software hiring
processes.

NO-ONE - no-one ever thinks they make the wrong call when they reject a
candidate - including you.

If you want to know what happened after that - I called the CEO who I knew
quite well and essentially said to him "What the fuck are you doing? I sent
you one of the best programmers there is - and I know this from first hand
personal experience over five years - and your interviewers rejected him?".
They employed him because of that and he became their chief technical
architect pretty quickly.

~~~
mattrp
I’ve seen this so many times as well where candidates under sell themselves or
key accomplishments never get brought up. Even Elon Musk in the most recent
Autonomy day had to say to Karparthy “introduce yourself and don’t be
bashful.” Even then, I’ve noticed, I’ve noticed that people don’t “hear”
things until a second or third conversation. The idea that you can absolutely
hear everything a candidate has to say in 60 minutes is just unreasonable. At
the same time, I think you did the right thing: you helped your client make a
better decision but not letting them rely on a first look exclusively.

~~~
andrewstuart
You assume that the reason the guy did not get the job is that candidate
undersold themselves.

It is mostly the responsibility of the technical interviewers to determine if
the candidate is suitable.

------
rsweeney21
Post author here:

My new company helps companies fill contract software positions. We're a lot
like a recruiting or staffing agency, but trying to do it in a new, less
spammy way. It's been REALLY frustrating working with hiring managers. I used
to be one of these frustrating hiring managers so I guess I had it coming.

I had a hiring manager tell me that he can tell in the first 15 min of an
interview if someone is a great engineer or not. I remember thinking the same
thing. I felt like I had this special intuition that I had developed over the
years. That's just total BS. There is NO WAY you can know that and you don't
have a "special gift".

I don't have a great solution yet, but for senior engineers I've come to trust
previous work experience the candidate has much more than my gut, my culture
fit questions and my stupid coding questions. We had a hiring manager that was
looking for a strong back end engineer turn down the engineer that built the
search backend for yelp because he didn't like his answer to "how would you
program the code in an elevator."

So my new company Facet (www.facetdev.com) is basically built around helping
senior engineers, with strong experience building products at scale, find
contract work.

~~~
ronilan
I agree, a 60 minute judgment may just be a random gut feeling, but...

Your new service offers to connect ex-Google, ex-Facebook, ex-YadaYada with
work. It’s all pedigree. Pedigree in my face, on page and in the website title
tag.

And... I’m none of those.

So... I personally prefer a world in which I have a chance to be misjudged to
a world where I have no chance at all. Maybe it’s just me though...

I think you got the observation right. The solution, no so much so.

~~~
rsweeney21
Yeah, we're not trying to solve the hiring/interviewing problem yet. We've
completely punted on that by only working with people that have a very
specific type of experience. We know that leaves out a HUGE number of
excellent engineers.

Right now we are focusing on a small niche to try and build a sustainable
business. Then we'll have the resources and impact to solve the broader
problem.

~~~
henryfjordan
What problem did you attempt to solve? You are essentially offloading the
vetting problem to all the companies who's former employees you work with. The
only value I can see that you create is in liquidity in the market.

Sure, there's some benefit to solving market inefficiencies and making some
money along the way, but don't pretend for a second that this isn't just
another recruiting firm using pedigree as a proxy for actual skills.

~~~
alias_neo
I had written basically this same response fearing I was being overly
judgemental before seeing your response but my gut was the same.

The posts read like sales pitches for someone solving a problem of making
money for themselves by limiting their portfolio to candidates for whom
$company had done the hard work of vetting by having previously hired.

Click-bait is a generous summation, I feel.

~~~
rsweeney21
Some people find our service valuable. There are devs that would prefer to be
freelance but don't want to spend the time doing sales to make sure they have
consistent work.

------
MikeTaylor
Obviously interviewing isn't 100% accurate. Almost nothing is. You definitely
can't tell in 60 minutes whether someone will be a success at your company.

But you often can tell, with very very nearly 100% confidence if someone will
_not_ be a success. And that is the purpose of the interview. To filter out
the "definitely not"s — the programmers who can't write fizzbuzz. And yes, I
have seen plenty of these.

And interview is not the be-all and end-all of recruiting: of course it isn't.
But I'm seeing a pendulum swinging a long way in the other direction now, and
a lot of people talking as though interviews are worthless. They're really
not.

~~~
ignoramous
And yet the discussions on the candidate post-hiring interviews mostly
revolves around whether they confirm to the company/tech/what-have-you
'culture' which, imo, is simply a euphemism for ageism, in-group bias,
prejudice, hubris, and one-upmanship more often than not. The world would be a
better place if people started accepting the fact for what it is: There
wouldn't be so much despair around the tech interview process if not
_everything_ was broken, surely?

Refs:

[https://news.ycombinator.com/item?id=9166501](https://news.ycombinator.com/item?id=9166501)
(On Secretly Terrible Engineers).

[https://news.ycombinator.com/item?id=19541617](https://news.ycombinator.com/item?id=19541617)
(How Not to Hire a Software Engineer).

~~~
scarmig
And yet when a company tries to reduce a candidate to a consistent set of
objective criteria as best it can--meaning, the ability to understand
programming problems and solve them in a structured interview--people get
upset because it doesn't account for the "big picture" of the candidate and
unfairly excludes people who'd otherwise be a great fit.

The reality is that any interview procedure is bound to have some false
positives and some false negatives, and there will always be people who see
themselves as false negatives (correctly and incorrectly) who'll complain
loudly about any procedure.

~~~
sanderjd
If it were only true or false negatives complaining, that would be one thing.
But lots of the complaining also comes from true positives. I've gotten every
tech job I've interviewed for since college, and have usually been quite
successful in the companies I've worked for. I think all of those interviews
were evaluating me on the wrong metric. They evaluated something I also happen
to be relatively good at - I have a CS background and I'm ok at contrived
little programming problems - but I believe the companies just got lucky that
I'm actually even better at the kinds of things that are important in my work
- problem analysis, solution design, consensus seeking, teamwork,
communication, writing, debugging, research, detail orientation, automation,
process definition, etc. etc. etc. - none of which has any real overlap with
writing little code snippets.

~~~
scarmig
All of those things, though, are inherently hard to test in a way that's fair,
representative, and not easily cheated. Anyone who had a reproducible,
consistent way to evaluate which candidates are good at those things would be
able to make a whole lot of money.

~~~
sanderjd
I don't disagree, but that also doesn't mean coding interviews like we do them
now are the best alternative.

------
nocitrek
I conduct the technical part of interview in my company. There are many
candidates which charm our director and HR. They recite the buzzwords
flawlessly and their resumes are top notch. When they come to technical part
of interview, they charm us as well. Until they start coding. We ask two easy
coding tasks (one programming, one data science) and so many candidates have
issue with transfering algorithm they conjured on the spot into code; or are
unable to solve problem on abstract level (and not hard-code the values for
problem given to them). It is not uncommon that we identify sweet talkers
which would have negative productivity for the company (meaning we would spend
more time/money into fixing their code than they would bring value). I am firm
believer that software developers must pass technical part of the interview.

------
sagitariusrex
Still relevant:
[http://www.aaronsw.com/weblog/hiring](http://www.aaronsw.com/weblog/hiring)

~~~
acct1771
Thanks for sharing this.

------
0x445442
One of the biggest issues is the industry as a whole operates on a completely
different hubris level. I don't think any of my professional references has
ever been contacted with the purposes of gaining significant insight into my
prior accomplishments or to get information about what it was like to work
with me. Nope, all we need is some fancy data structure and algorithm
knowledge and a white board to find that great candidate.

And those that do care about past accomplishments seem to be more interested
in public Github repos.

However, it's my guess there's some sort of inflection point because I doubt
Pike or van Rossum did any white board problems when they were hired at
Google. My guess is their interviews were much closer to the way the rest of
the world, the world outside software development, conducts interviews.

------
larrik
It's better to let good/great people slip away then to get a bad apple onto
the team. Or worse, a mediocre one who never does anything _wrong_ but also
isn't contributing positively either.

~~~
glitchc
If everyone only wants “great” people, where are the mediocre people supposed
to work? Think like this long enough and soon the have-nots are plotting a
revolution.

Companies with defined, well established revenue streams need to build
mentoring programs that turn mediocre developers (solid C) into good (solid B)
developers. Those employees tend to be the most loyal and are capable of a
great amount of menial to boring coding tasks without complaint. Why do we
devalue the grunts? They are the bedrock of every organization.

It’s not like Google’s Adwords is undergoing a complete code rewrite every
quarter. Neither is Microsoft Word. Most code changes are incremental, and
people need jobs, so why do we denigrate them so?

Plus A players are only usually A in their requisite subdomain. A crack coder
of financial systems is going to struggle mightily when writing a
morphological filter or a seam carving algorithm. There’s real value to the
business when an individual acquires domain knowledge, especially when the
domain is not fundamentally exciting (e.g. finance, geology).

~~~
pytester
>Companies with defined, well established revenue streams need to build
mentoring programs that turn mediocre developers (solid C) into good (solid B)
developers. Those employees tend to be the most loyal and are capable of a
great amount of menial to boring coding tasks without complaint.

In general, when coding, if you're doing a menial or boring task you're doing
something wrong. E.g. I've worked with a ton of mediocre developers who don't
automate the menial tasks and end up doing the task manually, sloppily and
slowly.

>It’s not like Google’s Adwords is undergoing a complete code rewrite every
quarter. Neither is Microsoft Word. Most code changes are incremental,

Incremental changes are hard. Incremental changes to large and unwieldy
systems while leaving them in a maintainable state are very hard (actually,
from what I've heard, Google Adwords' code base _is_ a mess).

>Plus A players are only usually A in their requisite subdomain. A crack coder
of financial systems is going to struggle mightily when writing a
morphological filter or a seam carving algorithm. There’s real value to the
business when an individual acquires domain knowledge

It's vastly better to have a really good product owner/manager who knows the
domain working with a really good coder than a mediocre coder with mediocre
domain knowledge.

~~~
glitchc
>In general, when coding, if you're doing a menial or boring task you're doing
something wrong. E.g. I've worked with a ton of mediocre developers who don't
automate the menial tasks and end up doing the task manually, sloppily and
slowly.

That’s total BS. Development is full of boring mediocre tasks such as fixing
spelling mistakes in UI controls, updating help messages for dialog boxes, or
porting the product from one build system to another. In many tasks, there’s a
huge gap (a chasm sometimes) between manual intervention and complete
automation. I’ve lost count of the number of times a clever programmer has
tried to automate the process and completely mucked things up, forcing me to
go in and fix things manually.

>Incremental changes are hard. Incremental changes to large and unwieldy
systems while leaving them in a maintainable state are very hard (actually,
from what I've heard, Google Adwords' code base is a mess).

That’s exactly where a great product manager can mentor a mediocre developer
in making a sensible change. It’s a great learning opportunity that we’re
depriving people of because software founders want to retain all the revenue
for themselves.

>It's vastly better to have a really good product owner/manager who knows the
domain working with a really good coder than a mediocre coder with mediocre
domain knowledge.

Of course. It’s also vastly better that to be born rich so one doesn’t have to
work at all. Ideally, all developers in the market would be great developers.
But they’re not. So I ask again, what are the mediocre developers supposed to
do?

~~~
pytester
>That’s total BS. Development is full of boring mediocre tasks such as fixing
spelling mistakes in UI controls, updating help messages for dialog boxes

I would say this counts for much less than 1% of what I have to do. A lot of
this stuff is in files which I can give product owner control over, too.

>or porting the product from one build system to another.

And this isn't boring. Build systems are tricky and filled with pitfalls. The
worst thing you can say about build systems is that it's not a prestigious
task.

>I’ve lost count of the number of times a clever programmer has tried to
automate the process and completely mucked things up, forcing me to go in and
fix things manually.

Yeah, and fixing things manually in that case _is_ boring. As I put it above
"if you're doing a menial or boring task you're doing something wrong" and you
precisely described "something wrong".

>That’s exactly where a great product manager can mentor a mediocre developer
in making a sensible change.

Product managers are not the right people to teach this kind of thing.

>Of course. It’s also vastly better that to be born rich so one doesn’t have
to work at all.

Unlike being born rich, which is out of your control, you are able to exercise
some degree of control over whom you work with.

------
movax
Daniel Kahneman, a Nobel laureate noted for his work in judgment and decision
making, stated in an interview earlier this year that interviews are terrible
at choosing the best candidates. Unfortunately, he didn’t articulate
alternatives.

~~~
apple4ever
The alternative is certainly the big problem.

The other big problem is not even giving an interview to a great candidate
based on things like a resume and possible a cover letter.

~~~
SeanAppleby
I feel like this is the much more complicated and secretly much worse problem,
not just for businesses, but for society as a whole.

We lean on heuristics like pedigree, both corporate, which is reflexive on
having already past a hiring filter before, and academic, which is ludicrously
expensive, both in time and money, is itself largely reflexive on having been
born into high socioeconomic status, and seems to have dubious value outside
of providing an easily digestible signal to get past these filters constructed
in the labor market.

Maybe if we could design a better interview system and more accurate and
precise cheap signals for who to throw into that (expensive to scale)
pipeline, we could save young people from having to start their lives 4 years
later and behind the starting line, reaping the productivity gains of the
labor market gaining those extra 4 years and a wider pool of talent, while
also creating more socioeconomic mobility spread more widely across society.

------
mjevans
Maybe making the 'extended interview' process more formal would be a good
thing.

Formally have an internal 'contract' worker process where anyone they might
want can be hired in to a trial pool, and projects can bring in the trial
worker for small things quickly (maybe writing unit tests, or a fresh set of
eyes, etc).

Every month review performance, make a decision to offer job as X, keep on for
another month, or let go.

As a different part of this, alternating weekends might also be a more focused
part of this, let someone that ALREADY has a job have a similar trial process
on just some weekends, maybe with a more remote-work focus.

~~~
phonebanshee
Why in the world would anyone be interested in doing this? Companies are
competing for talent, not the other way around.

~~~
kadoban
Because there are people out there looking for work. It's not trivial for all
programmers to find a job. I am in particular really awful at the entire
applying and interviewing process (except funny little algorithms problems, I
can kill those). It's a stressful, amorphous blob of everything I hate.

What the parent of your comment describes sounds a lot like a specific kind of
contract-to-hire, which if it's done well can be really effective. I get to
show off that I can actually develop software well, a company gets to judge me
on more than just an hour (or maybe a day) of questioning.

Unfortunately it can be pretty exploitative if done wrong. It's easy for it to
become an unending period of low-paid work where you're just hoping that one
day it'll turn into a real job.

------
thtthings
It's more accurate to say that the current format of asking algorithm
questions is not the best way to judge programmers ability. All it tests is
how much has the programmer memorized for the interview and not really how he
codes in the real world.

But i dont see this changing as most interviewers have been through the system
and are likely average programmers themselves.

------
stratigos
I am approaching 1.5 decades as an employed software engineer. Interviewing
culture seems to change a little over the years, but usually in other wacky
directions that leave me feeling entirely disengaged from the industry. I also
very much love my vocation, though it alone is not enough to keep me enduring
through the insanity of technical interviews , "cultural fit" assessments, and
the violence of linear reductionism.

Articles and discussion threads like this are pretty much the only reason Im
willing to interview at all (and thus remain in the industry). So thanks for
sharing, Im glad to know other people are aware of how bonkers this is.

The long version takes years of explaining.

The short version is: I am highly allergic to cognitive dissonance.

------
llamataboot
I'm interested in where the break-even point is. With some interview processes
now stretching over 15-20 hours or even more, it seems at some point the
process is not helping make a better decision and just demanding extra
resources from both the hiring company and the interviewees.

------
starpilot
This is why onsite rounds last 4-7 hours instead, while speaking to 5-10
engineers.

------
bdcravens
The underlying assumption is that Daniel would have been a great fit. Can this
be assumed? Even as a good engineer, his claim to fame is founding the Prime
Air team. Would there have been a "What started as an idea over coffee with
Gur..." moment at Netflix? Would he have just become a random developer
instead? Worth nothing that Daniel actually went from Microsoft to Amazon
after not being hired at Netflix, so perhaps he would have similarly churned
at Netflix.

This really isn't your typical "but whiteboard interviews suck!" situation.
(not to say that they don't ....)

------
gumby
I feel that the best we can do is "does this person look good on reference
and/or paper? Then if so, in the phone screen and then interview: is there any
reason to believe the stuff on paper might not be true?" (that latter happens
a surprising amount).

Also: "is this person a jerk?" though you have to account for the awkward
things someone might accidentally say when they feel they're in a high-
pressure situation.

------
vram22
It's interesting that some people find that many so-called "programmers"
cannot write FizzBuzz (see links in post, apart from a fun FizzBuzz version in
Python):

FizzBuzz in Python with nested conditional expressions and a generator
expression:

[https://jugad2.blogspot.com/2018/12/fizzbuzz-in-python-
with-...](https://jugad2.blogspot.com/2018/12/fizzbuzz-in-python-with-
nested.html)

------
julienreszka
Sometimes I think we would be better off hiring people on the spot at random.

In France we have a legal maximal 4 months trial period for employees so...
This gives enough time I think.

[https://www.service-
public.fr/particuliers/vosdroits/F1643](https://www.service-
public.fr/particuliers/vosdroits/F1643)

~~~
phonebanshee
In the United States we basically only have trial employees. There's nothing
at all like French job protections.

~~~
julienreszka
Really? Is that in all states? So hiring someone should be at least as easy
no?

------
stunt
It really depends. When you are hiring senior programmers, don’t set the bar
very high, then you can filter out majority of bad applications in a 60mins
interview. For both hard skills and soft skills.

Then you need to spend more time for the remaining ones to find the right
people.

------
baconmania
What's interesting is that most thoughtful people in the industry agree that
our conventions around interviewing make for an almost hilariously arbitrary
hiring process, but very few large software orgs have actually changed the way
they hire.

------
combatentropy
Portfolio, portfolio, portfolio.

I could talk to someone for hours and still misjudge whether I would enjoy
coding on the same project with them.

Show me reams of code that they wrote, and I will know immediately.

It's too bad that usually isn't possible.

------
alkonaut
We can’t. But we also can’t interview people for weeks. So we do one hour and
we go with our gut feeling. It’s a terrible way to hire, possibly the worst,
except for all the other ones.

------
EliRivers
No worries. There's always that probationary period in which either party can
recognise it's not a match and either do something about it, or just give it
up. How long will it take you to decide? A week? Two weeks? A month?

~~~
mental1896
It's not a "no worries" situation if you just left one job for another only to
find out the culture isn't a good fit.

~~~
joker3
Particularly if you relocated.

~~~
commandlinefan
Had that happen to a co-worker once. He moved (on his own dime) from Chicago
down to Dallas, then got let go just three months later. He wasn't even bad,
he just wasn't performing at the level I guess they wanted him to. I never
imagined back when I was in school that programming, of all professions, would
be as cut-throat as it is.

------
LeicaLatte
Of course we can. Its just that we can't do it right.

------
segmondy
You can judge programmer's ability in 60 minutes. Can someone program? 60
minutes is way more than enough. How great of a programmer are they? You're
going to need more than 60 minutes, unless already had a chance to look at
their prior code base.

What's difficult to judge is, are they organized? are they hard working? how
do they handle stress? how's their communication and attitude? do they write
great documentation? do they test their code? are they a great team player?
how creative are they?

Due to time constraint of an interview, someone might be more sloppy than
usual. Their might be the interview stress which is different from regular
work stress, interview stress can impact their communication, etc. But if we
focus plainly on technical skills, you can flush that out in 60 minutes.

I'm a hiring manager, and I have never failed in hiring someone who didn't
have the required tech skills. Technical ability I can always hit spot on, the
tough one is determination and soft skills. Some of the folks I took chance
on, who displayed terrible soft skills during interview turned out to be
champs, communicate great, work hard, etc. A few that were very eloquent
during interviews were a disappointment, sloppy in development, poor
communication in practice, etc.

People also grow, so someone that was a bad fit/interview for you, could in a
few years grow and become great. Potential is also difficult to spot.

~~~
ken
> You can judge programmer's ability in 60 minutes. [...] What's difficult to
> judge is, are they organized? are they hard working? [...] do they write
> great documentation? do they test their code? [...]

You answer in the affirmative, but then go on to clarify that you're excluding
95% of the job. I would simplify this to say "no, you can't".

~~~
taormina
> Can this person code at all? How well?

He's trying to draw the distinction between these two questions.

~~~
enraged_camel
The distinction is meaningless. No company wants someone who can merely code
(as determined by their ability to solve fizzbuzz-type stuff). They want
someone who can code well.

~~~
hayksaakian
I think you're missing the point.

The purpose is to Filter out people who can't code at all.

If you've ever hired developers you know how many people show up to interviews
and genuinely cannot program to save their life.

~~~
enraged_camel
>>The purpose is to Filter out people who can't code at all.

You shouldn't need a 60-minute in-person interview to figure out that they
can't code.

