
How we interview engineers at CircleCI - maliapowers
https://circleci.com/blog/how-we-interview-engineers-at-circleci/
======
akmiller
I interviewed at CircleCI many months back. Overall I thought the process was
done very well until I got to the stage before the onsite and I was simply
sent an email that said that while everybody like me and thought highly of me
they were not going to move forward at that time. I followed up asking for
more specific information and never heard back.

This process is never perfect, but I think when you don't hire someone it is
always nice to provide something solid the interviewee can go off of. Even if
the response is simply that we felt we had stronger candidates for whatever
reason.

~~~
rrdharan
This has been covered in previous discussions, but basically anything they say
can be used against them in a lawsuit:

[https://www.quora.com/Why-dont-companies-give-interview-
feed...](https://www.quora.com/Why-dont-companies-give-interview-feedback)

~~~
spike021
Why isn't there a waiver process so an interviewee can sign off that they
won't use the feedback in a lawsuit?

~~~
paxys
Waivers aren't legally binding. The company gets nothing by providing
feedback, and even if there's a small chance of a lawsuit, why take the risk?

~~~
spike021
I'd argue that the company does get something back, potentially. Many
companies will say to come back after 6 or 12 months to re-apply. If they like
a candidate well enough to really mean that, wouldn't it help the candidate to
learn what they were otherwise weaker with so they can work that out before
returning?

But that's a fair point about waivers not being legally binding.

------
hamandcheese
> Out of any group of 1000 applicants [...] Fewer than 3 will pass the on-site
> interview and receive offers.

> Our ultimate goal is to have the best people here, doing their best work.

I feel like everyone is so afraid of hiring a 0x or -1x programmer that they
don’t want to take a chance on a mere 1x programmer.

If I ran a company (I don’t!) on the scale of hundreds or thousands of
developers I’d be very choosy about who I let into senior/leadership type
roles, but be rather accepting of individual contributors, the hope being that
many of them would grow and develop their skills.

Am I just vastly underestimating the overal strength of a random sample of
1000 candidates? Are there really only 3 worthwhile developers?

~~~
rrdharan
No, I don't think you're underestimating their strength - I think it's more
that you're vastly underestimating how expensive it can be to make a bad hire
and how difficult it can be to fire someone in California, even at the
individual contributor level.

A few companies do manage to do it, but as far as I can tell you have to set
up your corporate culture, HR department etc. from the outset to be prepared
and ready for firing people frequently, and that's not something many startups
(other than maybe Netflix?) were/are really well prepared for...

~~~
JBlue42
Um, is it really that hard to fire someone in CA? It's an at-will state and
you can get rid of someone that day for any reason under the sky (that isn't
'discriminatory').

Firing doesn't seem hard but I think your point does stand that the cost of
having to do so is kind of great. Find, hire, onboard and train a new
employee, one that you hope does better than the bad hire.

------
protonimitate
How would you feel about there being some sort of standardized credential
certification, similar to a BAR or trade skill licensing, that proved a
candidates technical chops ahead of time? Do you think this would help reduce
the number of candidates that gets turned down during the initial screen?

It seems to me like if there was some standard of proving technical skills,
both parties could save a ton of time and go back to regular culture-fit style
interviews.

Of course, for the highest paying/most demanding jobs interviews would have to
be tailored - but I don't really see why junior/mid level roles need
interviews that last 4-8 hours on site plus any pre-on-site screening that
needs to happen.

~~~
sgslo
This exists in the form of the NCEES Professional Engineer exam
([https://ncees.org/engineering/pe/software/](https://ncees.org/engineering/pe/software/)).
The exam specifications for software are here: [https://ncees.org/wp-
content/uploads/2015/07/SWE-Apr-2013.pd...](https://ncees.org/wp-
content/uploads/2015/07/SWE-Apr-2013.pdf).

The PE exam is traditionally taken by people in the physical engineering
disciplines; mechanical, structural, civil, and the like. By passing this exam
you earn the right to use the 'Professional Engineer' title. Many (all?)
permitting jurisdictions require non-trivial construction projects to be
reviewed and approved by licensed PE's - this is not a joke certification.

In the physical engineering world (structural, mechanical, civil) the PE exam
is seen as a mark of a competent engineer, and usually comes along with a
significant pay increase, even if the engineer is not using their license to
review and approve projects.

Seems to me like the PE exam would be an appropriate exam filter for software
hiring.

~~~
twic
The trouble with this is that to have an exam to be an fooist, you have to
have an agreed canonical body knowledge that constitutes the craft of fooing.

Mechanical engineering has that because it's been going on for a very long
time, and has had time to converge on such a canon, as have many other
professional disciplines. But programming hasn't.

This is why the few successful certifications in our industry are issued by a
body which controls a specific canon - Cisco can certify you as a network
associate on its switches, Microsoft can certify you as a solutions expert on
its development platform, etc.

------
anonfunction
> Candidates pair with members of the engineering team, and work together on
> problems/projects in the actual CircleCI codebase.

Are they compensated for this work?

~~~
rrdharan
I feel like this is a disingenuous question. Are you asking if they should be?
Do you believe that CircleCI has some nefarious plot to increase engineering
team productivity by mining spare workcycles out of interview candidates?

If I were betting on this I would wager that something like 95% of the time,
the existing employee would be able to solve the project or problem much
faster without the candidate's input at all, but the point is to try and
approximate a more realistic work experience and environment.

I think that's a reasonable goal and I don't think CircleCI should be vilified
for it.

~~~
Analemma_
> I feel like this is a disingenuous question.

Not at all.

> Do you believe that CircleCI has some nefarious plot to increase engineering
> team productivity by mining spare workcycles out of interview candidates?

Your tone suggests that you think this is impossible, but it happens all the
time:
[https://news.ycombinator.com/item?id=16661338](https://news.ycombinator.com/item?id=16661338)
That was on the front page three days ago.

The GP's concern is legitimate; job interviews should not be an excuse for
unpaid labor.

~~~
rrdharan
> Your tone suggests that you think this is impossible, but it happens all the
> time

That's fair. To be clear, I don't think it's impossible, and especially given
what I've heard from friends who've worked in freelance web development
there's a huge long tail of scummy opportunities out there.

I more meant to suggest that it was unlikely this was CircleCI's motivation.

Also an important distinction between these two situations is that in the
example you point out, someone is being assigned a specific task to produce
some new IP which the company can then simply steal, whereas in the CircleCI
case I am assuming the "problem/projects" they have you pair on are something
in the neighborhood of e.g. "let's go analyze some page load performance data
together and see if we can fix some low hanging fruit" or "let's figure out
how to set up an alert in our Slack channel for this signal we have in some
logs".

Yes it's still true that a candidate could come up with some brilliant insight
that saves CircleCI real money in this scenario, but it just seems unlikely
and not something they could profitably build a development strategy around.

------
fein
> Our ultimate goal is to have the best people here, doing their best work.

Yet you have to tailor your ad language so more women apply because they get
scared by words like "objectives"?

I don't even know where to start here.

~~~
nilkn
I do worry that this is deeply circular logic. Words like "challenging" are
considered strongly masculine by the Gender Decoder they linked to. I don't
see how calling that word masculine is doing anything other than entrenching
gender stereotypes even further.

~~~
Techonomicon
I'll take a company trying to be cognizant of such things, and over adjusting,
over not at all.

I think it's great that Circle is trying to do their part in being as
inclusive-minded as the can be from the research they've collected.

If at the end of the day that is somewhat misguided, I'm sure they'll
readjust, given that they have already proven they will, where many others
have either explicitly not cared, or never even thought to.

------
GiorgioG
> 2\. Micro-skills take-home problem. The candidate completes a coding problem
> and reviews it with the interviewer. Talking through the problem with the
> candidate helps us assess ability as well as the candidate’s thought
> process, and how their priorities and values in writing software line up
> with ours.

This is generally a lousy experience. I've been places that want me to build a
'simple' (single entity) fullstack CRUD app (angular w/ASP.NET core backend.)

So what is it that they want prioritized? UI skills? Backend dev skills? I can
do both, but I'm not going to spend a week making it enterprise-production-
ready. The hiring engineers all have things they are looking for and these
things are never well defined for you up-front, so no matter what you do
(unless you spend way too much time on it,) someone may use it to disqualify
you (without you ever knowing it was important.)

~~~
paxys
Your experience isn't the norm. All the take-home exercises I have ever done
are specific to the position, and take a few hours to finish.

~~~
GiorgioG
"full-stack developer" is fairly norm these days in my experience.

------
mychael
The fact that they use a pseudo scientific "Gender Decoder" is a red flag. It
leads me to think Circle CI has unfair prejudices against men and masculinity
in general.

By the way, "Objectives" and "driven" are neither feminine or masculine,
they're just words. Can you imagine if every word in the Oxford English
Dictionary had a gender classification?

~~~
titanomachy
If it's true that using "female-coded language" results in more applications
from women, without reducing the number of men who apply, what's the harm?

If it's not true, then I would agree that this is probably a waste of time.

There's at least one peer-reviewed study (Gaucher et al, 2011) which supports
the idea that these words make jobs sound less appealing to women. Do you know
of a study which challenges these findings?

------
Willson50
CricleCI has 100-250 Employees, which at 1000 applicants per 3 employees means
over 30,000 interviewed applicant? Seems high.

~~~
daenz
Seems like bullshit, to be frank. It smells like a culture that has
progressively thought more highly of itself over time than it should.

~~~
fro0116
Or more likely, it's just a process that was born out of necessity due to them
getting more and more applicants for a limited handful of positions. How else
could any company handle that situation other than making their process more
and more selective?

And it would seem self-evident to me that not all of their 100-250 employees
were hired at that 3/1000 rate, as number of applicants will grow over time
due to increased brand recognition, increased spending on sourcing/outreach,
etc.

------
marcinzm
>Candidates pair with members of the engineering team, and work together on
problems/projects in the actual CircleCI codebase.

This seems like it'll bias to those who can quickly ramp up on a new code base
in an unfamiliar environment. Which, in my mind, is a distinct skill from
being able to contribute quality code to a code base you already know. I
wonder if this is known and acknowledged bias or not at CircleCI.

~~~
bhasi
While this is true, one could come up with similar arguments for any task put
to the interviewee. Not to demean your point, but the interview process _is_
one of elimination after all, and biasing the task to favour certain traits
and skills is but one of the ways you can go about it.

~~~
marcinzm
I totally agree that everything introduces bias, I was more wondering if this
was intentional/understood bias or accidental bias.

------
mnemonicsloth
There's a basic problem with posts like this. Finding the right people to
associate with is a hard problem.

Finding friends is hard. Finding a significant other is (ideally, anyway) a
once-in-a-lifetime challenge. Finding people to hire is even harder: you're
still trying to establish chemistry, only now there's a lot of money involved
and the actors are big groups of people with complicated internal dynamics.

So I think what's wrong with this post is that CircleCI seem so self-
satisfied. They seem to think they have a handle on this problem, when anyone
who's been through a job interview in the last year or so will tell you it's a
shitty, capricious, random experience even when they take you.

------
yodon
CircleCI’s hiring process had 1000 people enter their funnel as new candidates
and 3 emerge as employees. Others have tabulated the rate of
candidate->screen, screen->onsite, and onsite->offer. If we make a
conservative estimate of 1 hour of company time per considered resume
(averaged across all steps, higher for those who get far into the process,
lower for those pruned earlier), they burned at least half a head of employee
time to hire 3 heads. That’s way too high an interview cost per hired head.

------
daenz
Only 3/1000 will receive offers. Does this strike anyone else as an extremely
flawed and stringent filter? They claim to only want to hire the best, but in
my opinion, they will end up hiring the lucky, with similar opinions and
skillsets to the interviewers.

~~~
MarkSweep
After reading this, it sounds like applying to CircleCi would be a waste of
one's time.

~~~
jmtame
That seems a bit harsh, even if CircleCI scores below average on some steps in
their funnel. You know going into any company's hiring process that they're
going to extend an offer about 1-2% of the time from the very top of their
funnel. That doesn't mean you'll go around talking to 100 companies before you
get an offer, but the numbers on the company's side will almost always look
grim to a candidate.

------
AboutTheWhisles
If you get 1000 applicants and only make 3 of them offers, those better be
some very good offers.

------
nartz
How many positions were you _actually_ hiring for though?

