
We Prefer Coding Challenges - ddispaltro
https://gravitational.com/blog/coding-challenge
======
tptacek
Unfortunately, the idea behind coding challenges has been cargo culted into a
lot of weak hiring cultures:

* The challenges often don't actually matter, and are simply part of the recruiting hazing ritual.

* They're totally subjective and just a proxy for the same bullshit friends-and-resumes process people were using already.

* No consideration is given to the amount of time the entire recruiting process is taking.

* The challenges themselves are totally divorced from the day-to-day expectations of the job, so people are doing CS102 algorithms problems for jobs that will mostly be about moving bits from one place to another.

I understand why people don't like them. But that doesn't really matter,
because code challenges are _so much more effective_ at qualifying people that
it would be crazy for any competent shop to stop using them.

For a couple years (and, of course, for many years before at Matasano) we've
been hiring resume-blind and with almost no interviewing (we do a final on-
site, but it's a formality with no tech-out, which is something we tell
candidates). We calibrate our challenges so that they'd ideally, for a
qualified candidate, take less time than interviews and phone screens would,
and have the advantage of being scheduled at the candidate's convenience (all
in one night, spread out over a couple weekends, whatever).

We have candidates coming out of our ears (volume has been the single biggest
problem with our recruiting practice), and the quality of those candidates is
just as good as anyone else's comparable funnel.

This is the right way to hire technical staff.

~~~
cdavid
Since I have been in a position to do so, I tried to push for coding
challenges, and base tech IV on the assignment itself. Your blog post about
hiring has been my biggest inspiration on doing hiring. I hired 10+ people in
1.5 years this way, and I would rehire 90+% of them for most roles. I do see
difficulties about practicing it though:

1\. When you don't control all the HR process, you can't guarantee that the
tech assignment is the main filter.

2\. Because resume are generally so useless, if you look for a lot of people,
you need to be able to review assignments very quickly. Also, to convince
people to do the assignment, I make sure to have a phone call _before_ the
assignment, which is time consuming since there is no filter.

3\. Writing good assignments is hard. You want something that does not take
much time (half a day max), is quick to review, interesting enough for the
candidate.

I have also not been able to make sure the assignment is the sole filter. I
still filter at the (single, < 90 mins) tech IV. But that is at least
partially because of my domain (ML engineering), where profiles are more
diverse than pure software engineering.

~~~
tptacek
We did a company based on this approach, and problem (1) is the main reason it
failed.

------
nscalf
I don’t do code challenges anymore. I have a full time job, I’m working on a
startup on the side, I do some extra contracting, and I have an ever growing
list of potentially profitable opportunities that I want to build. I would
love to get paid more for my full time gig, and I should. I’m solidly a senior
engineer, but I lack the title or pay. But I don’t have the time to actually
spend 8 hours doing some silly app. I get that it makes sense for the hiring
manager to give a coding challenge, but I don’t care. And the reality of our
world right now is that I don’t need to care. There are plenty of other
companies desperate to bring on good people.

My skills give companies extreme multiples of ROI, and no other fields have
nearly as ridiculous a hiring process. Have someone come onsite for a day and
pay them, we can both decide if it’s a decent fit. But I don’t know you, your
company, your team, the work process, the supporting teams, your funding,
etc... honestly, one of those things are probably really awful. I’m not
jumping through hoops for free. You’re not google, you won’t pay me $150,000+.
Pass.

~~~
tptacek
Spending a week doing off-and-on phone screens and then a day on-site doing an
interview loop is a bigger hoop to jump through than banging out some code on
your couch. But, I mean, people have all sorts of reasons not to apply to a
job. Some people don't want to touch Java code. Some people don't want to work
for a company in Santa Clara. You don't want to work anywhere that asks you to
complete a code challenge. That's totally fine.

~~~
runT1ME
If I am doing a phone screen + an all day onsite, sure it sucks but at the
very least I know I'm a serious enough candidate that the company is willing
to burn one engineer day to evaluate me. With a coding challenge, I have no
way of knowing! Maybe they sent the challenge to ten other candidates, maybe
twenty, etc? I don't even know if the code will be reviewed. Make me jump
through as many in person hoops as you want, I refuse to do take home coding
challenges. _shrug_

~~~
nscalf
I think ultimately the problem I haven’t isn’t that I have been assigned a
problem to see my code skill. I think that’s pretty fair. The problem is that
it’s not a great indicator if your beyond a staff level, it offloads all of
the cost of finding and filtering an employee to the applicant, and like you
said, it has no indication of their commitment to the code challenge while
demanding an open ended commitment from the applicant.

~~~
wildrhythms
I actually turned down a job, whose hiring manager was very excited (or
desperate) to have me, after the recruiter sent me a link to a coding
challenge. I felt devalued, that they didn't value my time, and I felt turned
off realizing that this coding challenge was the only hurdle for engineers on
that team. Note that I wasn't a fresh grad or anything, I had years of
experience at this point from, what I assume was at the time, one of their big
competitors.

Since the company can't make time for a potential "experienced candidate"
(their words) during the hiring process, I wonder what else they can't make
time for? Red flag for me.

~~~
busterarm
You're in the rare camp that realized this. Thanks for doing me and everyone
else in this industry a solid.

Companies won't change until this is enough of a negative signal for them.

------
deeg
I do not want to work for a company that does NOT test my technical skills; if
they don't verify my skills that means they don't verify others and I'd prefer
to work skilled engineers (with exceptions for entry-level or junior
engineers).

Most people seem to agree that white-boarding is a poor way to judge technical
skills. Coding challenges seem a good compromise, assuming they don't require
more than a few hours of time.

~~~
didibus
I find as both an interviewer and an interviewee that I prefer whiteboarding.

But, it depends who you need to hire. If you need someone to just do the job.
A coding challenge is fine. If you want someone who's above the rest in their
CS knowledge, and problem solving skills, than a whiteboard interview is best.

Just my opinion. I have no data for this.

A whiteboard interview is pretty much a coding challenge where I can observe
the candidate's methodology and their general approach to tackling problems.
That's super valuable information.

From being a candidate myself, I've found that having someone watch me and
being in a room with the stress and all motivates me to show my best. Whereas
coding challenges I really lack the motivation and I end up rushing them,
doing them last minute, and really not showing my best.

Ultimately, I think they both aren't ideal, but we don't have anything better
short of an internship (and even those don't always pan out, since interns are
treated differently and provided more hand holding).

In the end, it's just hard to reduce work that in practice spans multiple
weeks per project, and involve all kinds of side process, to something you can
test for in under a day.

~~~
shantly
> From being a candidate myself, I've found that having someone watch me and
> being in a room with the stress and all motivates me to show my best.
> Whereas coding challenges I really lack the motivation and I end up rushing
> them, doing them last minute, and really not showing my best.

OTOH, whiteboards or live-coding under interview conditions is a great way to
watch me do worse work than I did when I was less than a year out from writing
my first hello world. And yeah, I'm cool & collected under normal
shit's-broken pressure, or client's-pissed-off pressure, on the actual job,
which is totally different.

~~~
gremlinsinc
Same, I can't code while being watched, but I can do a take home assessment.
My favorites have been real world stuff like ... build x by creating your own
MVC w/ x, y, and z as a base... like x router, y templating engine, etc...

~~~
shantly
Someone on here posted something about letting candidates pick an open issue
on an open-source project and prep a PR for it, to go over. Easily my favorite
version of a take-home assignment I've ever heard of—it's not throwaway junk
or contrived bullshit, and I get to claim the PR if it gets accepted, whether
or not you hire me, so while it's not paid it's also not a complete waste of
my time if the job doesn't work out—but it's not repeatable the same way for
every candidate (which is kind of the point? I mean selecting the issue to
tackle is _part_ of the assessment, which, as long as there's a little
guidance about the sort of thing that's considered good-enough so you're not
entirely in the dark re: expectations, seems ideal) so a certain kind of
hiring manager or interviewer will think it's necessarily crap.

~~~
gremlinsinc
That's genius. Almost sounds like something that could be it's own
Startup/SaaS. A marketplace of code 'assignments' that OSS projects need done,
employer can 'tag' assignments that they approve of, then you submit your work
to the employer/oss project simultaneously so both can do a code review of it,
if you lose the job but it gets accepted by the OSS then at least there's
that. -- The employer would be the only part paying $$ for the platform, but
it could be tax deductible as a 'donation' to OSS perhaps.

------
0xDEFC0DE
Unlike the other code-slinging gods who walk amongst us in this thread, I
prefer coding challenges because I'm pretty mid-level and average and I do
actually need a job. And I hate whiteboarding. No one should do whiteboarding
for code.

~~~
shantly
Agreed. Make me whiteboard or live-code if you wanna make me look and feel
like I've not actually had over a dozen years of productive experience
designing systems and shipping code and fixing other people's broken crap, and
you'd love to watch me stress out and forget how to write a damn loop so you
can complain on HN about how I was clearly lying about my experience &
abilities and that's why you have to make people live-code or whiteboard.
Challenges are, at least, preferable to that, for sure.

------
vorpalhex
> They have so many offers that writing code for 4-8 hours is just not going
> to work for them.

Yeah, I don't typically work for free either. Are you paying candidates for
this time?

"Hey, I need you to come in and stock shelves for four hours, you know, just
to see if you're a hidden gem for shelf stocking"

~~~
macintux
On the other hand, I’ll spend 4-8 hours on a coding challenge for free. Give
me an interesting problem and why not?

I was asked to write an AWS integration tool once, and I declined to proceed
with the interview process, because I didn’t feel like spending hours learning
the Amazon API and re-learning a language with a useful library (I’ve since
come to appreciate Python and Boto3, even if I’d rather be Erlanging).

It also wasn’t a sufficiently interesting problem to make the time worthwhile.

~~~
tptacek
Which, for the right job, can be a perfect example of the system working: the
firm wants to hire someone super comfortable with AWS, for that person, the
tool might just take 30 minutes to bang out, and so it works as a filter.

------
iliaznk
I’ve passed through a screening procedure recently that consisted of three
steps:

1) They asked for some publicly available samples of my code «I’m proud of».
Okay, I prepared three samples from different areas with detailed descriptions
and comments, explaining what that code did and why it was good.

2) After that, as a coding challenge, they asked me to write a simple app
using the Unsplash API.

3) Finally we had an over an hour long interview during which I was again
given three simple coding problems.

And then they rejected me. The question I have after all that is do you really
need a multi-staged procedure like that to determine a candidate is not a good
fit?! When exactly did they decide I wouldn’t fit? If that was after step 1 or
2, why didn’t they stop right there? If that was only after step 3, then
what’s the purpose of steps 1 and 2? They don’t mean anything if step 3 can
just cancel them out. Then why didn’t they start with step 3 not to waste my
and their time?

------
etxm
I’ve posted on here before about when I’m in a hiring role how I prefer to do
code challenges. I enjoy doing BYOC challenges.

Let a candidate pick a ticket from an open source project and come in to pair
on it (or shoulder surf if uncomfortable with pairing). I don’t see the issue
until the day we are pairing. Goal is to let them be the expert and as
comfortable as possible.

They get some code on their GitHub, open source benefits, and I get a peak
into their engineering skills.

If you’re thinking, “hey that’s not apples to apples,” well people aren’t
fucking apples.

I had someone come in once that didn’t have an “issue.” She had been working
with a library that had poor docs and wanted to work through the library and
document it. THAT! That was an awesome experience. Best engineer? I had no
idea. Willing to bang her head against a wall to make the experience better
for others. Yes. Fuck yes, get on my team.

------
deedubaya
Coding challenges as your barrier to entry are an optimization for a certain
type of employee - low demand workers who don't have the leverage to by-pass
who start hiring funnels that start with coding challenges and recruiter
calls.

You may be gaining access to hidden gems, but you're definitely turning away
proven and experienced developers from even entering your hiring funnel.

It all depends on what type of employee you're wanting to hire.

~~~
alexk
Author here. Even seasoned developers have to go onsite and spend 1-2 days
doing interview.

In this scenario, we are asking for similar amount of time, but with no
travel, time that works for everyone and has low stress with an interesting
task folks can review beforehand.

Besides, before even going through this challenge, any engineer can make up
their own mind whether it's worth for them because we present compensation
brackets upfront on the call.

~~~
jakebasile
> Even seasoned developers have to go onsite and spend 1-2 days doing
> interview.

No, they don't. Like most people that champion these types of nonsense "code
challenge" interviews you present a false dichotomy of "you have to either
torture someone in person for multiple days, or torture them with a bullshit
made up test for hours!"

You can talk to them for a while over the phone (or video chat), split up over
a few days, and get to know them just fine.

~~~
tptacek
Yes, this much is true: in weak hiring cultures, which are very common, people
get their friends (or people with dazzling resumes) hired without any real
rigor. What's more, in that culture, it's straightforward to generate a
dazzling resume: you "fail up", because it takes longer to get fired (or for
it to become clear that it's time to get a new job) than it does to get enough
time on the job to look good on a resume.

~~~
jakebasile
I really don't think this issue is as widespread as commonly held, and I think
it is used to perpetuate the horrible hiring culture that is common in our
industry.

I think it's perfectly reasonable to hire people based on resume, references,
and conversation. If they can't handle the job once hired then they should be
let go.

I do not think that the idea that all candidates are complete idiots unless
they can be "proven" to be good engineers (by faked tests that rarely cover
the actual job requirements but do have a way of discriminating against those
who don't enjoy brainteasers, didn't go to college, or are older) is good or
healthy.

~~~
shantly
I wanna know who the people are who can talk confidently and intelligently
about the details, decisions, and purposes in the systems, technologies, and
workflows they've worked with or built, to essentially arbitrary depth, but
can't actually do the job, and also actually manage to hold down jobs without
huge gaps in their work history. I find it hard to believe there are so many
of them that it's worth defending against. Smooth talkers exist, sure, but
c'mon, they can handle a friendly but detailed discussion about stuff on their
résumé but can't write code? God, learning to write code would be easier than
prepping for that kind of thing _without_ actually knowing how to do the work,
surely.

~~~
tptacek
Serious question: have you done much hiring? Because this was a huge component
of my lived experience as a hiring manager since the late 1990s, and every
hiring manager I've talked with directly since has had similar experiences. If
you do a lot of hiring, and you're honestly surprised that there are people
who do really well in interviews and turn out not to be good hires, I'm
curious to hear more about your experience.

~~~
shantly
Yeah, both as just another part of larger hiring panels and in tech lead roles
with the biggest say in who gets in. Maybe I've been rejecting people who'd be
fine and am committing the same sin but it _seems_ pretty easy to get a sense
of "this person has been working 5 years but is incredibly junior and barely
understands the stuff they're working with" or "this person has been working 5
years and really knows their shit" from like 20 minutes of conversation, and
I've not seen a true bullshitter get through hiring despite a lack of heavy,
time-intensive tech-challenge-based interviews anywhere I've worked. Closest
we had one place was a really easy screening project, which we didn't make
anyone not-obviously-junior bother with, and I bet a few minutes on the phone
with the right person could have replaced that with no real harm to the
process.

I have, however, on two occasions probably _personally_ convinced someone they
had dodged a bullet by passing on me when I bombed that sort of quizzy
interview—they were, I am entirely confident given the job descriptions, my
work history, and actual feedback I've received from managers and peers, quite
wrong, but I bet they were _very sure_ I was useless and nowhere near being
fit for the job.

[EDIT] I guess I should add that I have a history of not realizing that
something I'm good at is not actually easy or obvious to others, so possibly
I'm just unusually awesome at assessing developer suitability through
conversation—I really doubt it, but maybe that's what's going on. I've also
not found a way to make this fit with any sort of "we ask everyone the same
set of questions because we want spreadsheets at the end" process, as you've
got to tailor the interview to both the position and to the candidate, as
presented on their résumé and related material. So if you want spreadsheets
and quantifiable-everything then I'm not sure how you match that up with my
preferred style.

------
kemiller2002
I get why companies like them, and there is nothing wrong with them. The truth
is that I probably don't want the job enough to actually do it. It takes time
away from my family, and my friends and most of the time, it's not that
stellar of a place to work anyway. If I'm desperate for a job, I could
feasibly see doing it, or if I thought it was a really amazing opportunity.
The truth is that I'm probably not going to be in either of those positions.
What really does irk me is when a company says you have to do it before
they'll even talk to me. I'm definitely not going to spend several hours on
something without even knowing anything about the company.

~~~
rdelval
The asymmetric time investment required from candidates is what bothers me the
most about these coding challenges.

My personal experience has been that it's a lopsided process with the
candidate putting in hours of unpaid work and the company spending one, maybe
two engineering hours if we're being generous, reviewing the submission.

If the reviewer decides to reject the challenge, the loop closes, HR sends a
nice e-mail, and the candidate has nothing to show for it.

------
UweSchmidt
Another one of those, huh?

The only way to find out what kind of salary is possible is to ask for numbers
that get rejected a few times. Additionally, there are many companies out
there, with a surprising variety of job parameters. Finally, many people are
not good or comfortable with the application process, so you need to practice.

This means you have to apply to many companies and have to have many job
interviews. Long, multi-round interview processes and take-home assignments -
reasonable or not - make this very difficult and expensive for the employee.

These regular company blogposts seem to respond to the pushback of applicants
and try and establish this kind of application process as a norm.
Unfortunately this would decrease the market transparency for employees even
more - many people go through their working lives underpaid and in subobtimal
working conditions without knowing any better. This procedure also feels
asymetric - in a time where software people are supposedly in demand, it's the
employee who is supposed to jump through hoops to get a job.

~~~
tptacek
If you reached out at Latacora and asked "is there any way you'd be able to
pay $X" before you did any of our challenges, we'd get you a quick, straight
answer. Why wouldn't we? If you're outside the ballpark for us, working to
recruit you isn't a good use of our time.

------
jgwil2
The process outlined in the article shows respect for candidates in many ways,
but especially by providing _feedback_. I have no problem going through an in-
depth interview process, because I can always look at it as a learning
experience, but it can be very frustrating to go through a process and then
not receive feedback, positive or negative, on your performance. I understand
that all employers are busy, engineers' time is valuable, and that often
communication is managed by HR people who don't have in-depth knowledge of the
reasoning behind decision-making, but I think that making constructive
feedback an expectation for the hiring process would improve the experience
for candidates manifold, and ultimately would result in less arbitrariness in
hiring decisions.

------
mooreds
I didn't see "pay candidates for their time" as a section even though I saw
several mentions of how invested the company was (CEO commenting on PRs, team
investment).

Saying "We will pay you to do this exercise" is a simple (and cheap!) way to
show you value candidate's time.

------
jonfw
> “I’ve been hiring people for 10 years, and I still swear by a simple rule:
> If someone doesn’t send a thank-you email, don’t hire them.”

I was contacted by a recruiter a short while ago asking my availability for a
phone screen- and I replied simply with my availability. The recruiter still
hasn't gotten back to me, and I think that it may be because I didn't go out
of my way to thank them for reaching out to me.

Are all of the niceties in your email inbox really that important to people?
It seems pretty manufactured and boilerplate when people send me emails that
they "hope find me well".

I may just have to get over it and start letting people know how "excited I am
to speak with you and learn more about this opportunity!"

~~~
olooney
Yes. They're not testing your knowledge of etiquette, people skills, or
friendliness though. They're looking for a more basic and far more widely
applicable skill: following best practices. If you were configuring Apache for
the first time, wouldn't you google for a list of do's and don'ts and spend a
few minutes implementing the low-hanging fruit? Not doing that puts you at a
huge risk of leaving a huge security hole or massive performance hit, right?
It would be amateurish, and if a problem occurred because you didn't take even
a few minutes to do the most basic and widely known stuff, you would have no
excuse.

Likewise, while nobody really cares about thank you emails, _not_ doing that
suggests you didn't even bother to Google "Interview Tips" and implement the
ones that took less than a few minutes. That's not a good signal to send to
people that are going to have to rely on your professionalism and quality of
work for the next few years.

~~~
jgwil2
This would make sense except that the job isn't interviewing for jobs. You
shouldn't hold people to the expectation that they're going to approach
communicating with recruiters in the same way they would approach actual work
they were actually getting paid for. This is a great way to filter good
candidates out based on something that has nothing to do with the work you are
hiring for.

~~~
jonfw
This doesn't just have nothing to do with the work you are hiring for, this is
testing for something that's actually a negative trait in your work.

Technical writing should be as concise and clear as possible

------
kureikain
This seems got some negative review.

Let me shared my experience.

Alexander is fantastic as an interviewer and co-worker. I was interview by
ALEXANDER a few years ago. I ended up not getting an offer. It was my fault.

The challenge is the hardest thing I have to research on my own to implement
because google found no result for me at the time. I learned a lot when
working on his challenge and the idea of being able to solve it is more
imporant than passing to interview to me.

The way they conduct interview is really great. You are given help by their
team.

All I can say is Gravitational takes hiring to their heart and setup for
success of candidate.

TO Alex, I'm Vinh if you remember me :-).

~~~
alexk
Hey Vinh, thank you for the kind words and I'm glad that you got a positive
experience with our challenge and the process!

------
sandGorgon
> _Split your code submission using pull requests and give the team an
> opportunity to review the PRs_

> _Not communicating. Candidates who submitted all the code to master branch,
> which does not give us the ability to provide feedback on the various
> implementation phases. Because we are a distributed team, structured
> communication is critical to us._

how do you ask your candidate to split your answer using PR ? We would like
this during interviewing but it ends up being a single PR. I dont want to
impose create separate branches and a PR for each small incremental feature
thing.

I'm assuming you are using Github

~~~
alexk
Good question! We usually recommend to submit a fully complete logical part,
e.g. auth middleware, client, server. We recommend against submitting
incomplete functionality in a PR because it is hard for our interview team to
understand whether it's incomplete or complete and has defects.

~~~
sandGorgon
so would each complete logical part be a separate branch with its PR ? and
then branch from the previous branch to create the next PR?

I'm just wondering if you found there to be any cognitive overhead here.

------
pickle-wizard
My current job had a coding challenge and I felt they handled it pretty well.

It was done one site as part of the interview process. It was related to the
work I'd be doing, however it wasn't super complicated. They expected that
you'd be able to finish it in about 30 minutes, but you had as much time as
you needed. I finished it in about 20 minutes. Since it was over lunch time,
they bought me lunch. I figure lunch for 20 minutes of work was a fair
reimbursement.

------
etxm
They’re bull’s excrement, this industry’s interview process is broken and
arbitrary.

I don’t know if we’ll ever fix it, but code challenges certainly aren’t the
answer.

I’ve been developing professionally for 20 years.

I take about 4 interviews a year when _not_ looking for a job to stay
practiced. Held DoE, Staff, and Principle roles.

Anecdotal evidence from this year:

\- given a decent sized MVC app with bugs and missing features. Tasked to
green the test suite. PR comment from a pretty established engineer was “this
is perfect, I wouldnt change anything.” I used 1 of my 4 hours and thought my
work was poor

\- given a “simple”[1] sort and map/reduce a large log file and didn’t finish
after 2 hours because the interviewer kept interrupting to ask me about the
details of how internals of the language and GC worked - things I was familiar
with, but I can’t focus on code and answering questions at the same time. I
was ranked as “junior” and not a fit for the role, cute.

This last one rolls up into one of the problems I have (and assume others
have) in interviews: “work through the problem as you would, but talk out loud
about your process”

If I’m talking, I’m not in my zone. Let me shut the fuck up and solve the
problem and we can talk about it afterwards.

[1] “simply” parse this 500k line CSV file and aggregate some results, no
dependencies. CSV file was actually a STDOUT log file that multiple containers
were writing to with different formats, JSON, CSV, TSV, SSV, and random ass
unstructured logs from a prod system.

------
rileymat2
I do not mind a coding challenge in the hiring process if it is the last step,
not the first step.

------
rajacombinator
I’ll only do these if 1) company pays me or 2) I’m exceptionally interested
and think the comp will justify it. (ie. Almost never.) Spending several hours
on something that might not even get reviewed - no thanks.

------
IshKebab
Great article, although 4-8 hours is clearly _way_ too long. 2 hours is much
more reasonable.

------
a0zU
Coding challanges are used pretty often to scam programmers to do work for
free before the company rejects them.

~~~
twic
I don't believe this actually happens. Do you have any concrete examples?

~~~
mamon
There's this infamous company from Austin, Texas, which hires programmers for
remote work, and they typically give you coding "challenges" that require a
week to complete, and are complete (although not very complex) applications.

