
How to Interview Entry Level Software Engineers - dsil
https://technology.cloverhealth.com/how-to-interview-entry-level-software-engineers-daaecf9db97
======
kinkrtyavimoodh
Every few days an article of this type is written and posted on HN. And all
the comments are people either proposing another (same old) approach to hiring
or people replying to those comments explaining why they'd hate that approach.

This is then done for every possible combination of approaches (homework
questions, whiteboard interview, pair programming, fizzbuzz questions,
algorithmic questions, work-history questions, career-progression questions,
mini-project, magical incantations, and so on...)

It may sound cynical but at this point it feels like no one has any new ideas
about it and the same content keeps getting recycled in the form of yet
another article and yet another comments section.

Half of these articles are from company blogs which exist largely as just a
marketing tool.

Since there is no actual new thought, everyone just cargo-cults the Google
approach.

~~~
auxym
Why though? Why is programming seemingly the only field where companies feel
the need to put up these super interview processes with mutiple interview
rounds, technical tests, white boarding, etc?

Is it the high salaries? In that case, do you see such testing in law, or
finance?

Is it the technical difficulty? In a previous life (read: up to a few months
ago) I worked as a mechanical engineer, in a pretty technical field (numerical
simulation of structures). Additionally to demanding reasonably deep knowledge
in a few areas, the job came with the responsibility of making sure that your
designs do not kill or injure people.

I had never had a technical test in an interview before I interviewed for my
current IT position. Nor have I heard of colleagues having such a test. Is it
because most engineers in Canada are licensed (myself included), and most jobs
require you be licensed? If that's the case, it's pretty funny, because at
least in my province, the test you have to pass for the license is entirely
about ethics and professionalism, nothing technical.

Part of the code of ethics though, is that it's on you as a professional to
ensure you have the necessary competence to do the work.

~~~
travisjungroth
I've been both a professional pilot and software engineer. The process to
become a pilot was more intense in time and cost than to become a software
engineer. The interview process for a job as a pilot is much easier than the
interview process for a job as a software engineer.

The barrier to entry for software engineer jobs is all at the interview.
That's why it's so hard, tedious, and complicated.

~~~
quickben
Imagine if this article was intended for pilots and it said: "How to interview
entry level pilots for A380. On the job training provided if you are fresh
graduate"

I'm aware it's a regulated profession, but imagine if it wasn't? Whats there
to stop management from actually fast tracking the backup pilots as that?

~~~
travisjungroth
That's kinda how it works in Asia. You sign a seven year contract, they send
you to a flight school where you learn the basics, and they stick you as like
fifth-in-command in an airliner. At some airlines you have to work for the
company in some other capacity first.

------
gopalv
> Also, we chose to completely scrap whiteboard coding and assigned homework
> instead.

For an entry level engineer, this removes a bunch of variance and also because
it gets you another scenario which happens way often in actual work.

You show up with the code and it gets reviewed & discussed on how/why of the
implementation.

The most talented entry level people I've worked with have struggled with
criticism instead of grading (i.e "grade the code" through a bunch of test-
cases is different from "why did you do this and explain").

~~~
taurath
I’m a senior engineer with an anxiety disorder that has trouble with white
boarding. It’s honestly probably the most cruel environment possible for me
because when I work I’m on the same team.

~~~
socanxdev-thrwy
My social anxiety makes interviewing very difficult. I freeze up to a single-
track-mind. Every spare bit of processing available is used up on guessing
social situation around me.

I recently had an old boss try to bring me on to a new job with him, but
wasn't able to get me through the interview phase, despite knowing I'm
capable, because my interviews with the rest of the team went so bad.

I've been a software engineer over ten years. All of the positions that I have
been offered have come from especially kind/empathetic teams, and the
interviews usually focused more on conversation than quizzing.

It requires an unfortunate amount of rejection before coming across my more
selective criteria, but I guess that is what it is.

~~~
zip1234
Our coding test is over Skype and a collaborative editor. Is that a scenario
when you would freeze up? Genuinely curious as there were a few cases where it
seemed that people were way nervous.

~~~
socanxdev-thrwy
I personally would. Any situation where I'm being judged by strangers in real-
time sets off social anxiety.

Some things that help me in that situation are when there's some
light/friendly conversation preceding the test, or when there's a limited the
us-vs-them size, like only one or two interviewers at a time.

------
drakonka
I was pretty nervous at my interview; no test was mentioned until the day, at
which time (after the rest of the interview) I was presented with two sheets
of xml-style configuration code for the company's proprietary build system.

The task was super simple - basically just duplicating some of the
configuration and renaming some values to create another job, but at the time
I just froze and couldn't see it. I managed to get some sort of answer out,
but wasn't happy with my response. They claimed that they were meant to give
this to me in advance as a homework question and somewhere the ball got
dropped, but I had a feeling they wanted it to be a more on the spot thing
considering the problem was so simple.

Thankfully I had the wherewithal to ask if I could take the problem home with
me after the interview. When I got home and looked at the problem with fresh
eyes I facepalmed so hard. I wrote up the correct answer and emailed it over
to the company recruiter, hoping she would pass it along to my interviewers,
but didn't get my hopes up.

I guess she did, since I ended up getting the job. I still cringe a bit about
how I just blanked on that stupid question. Thankfully I got ramped up really
quickly once I actually started the job and had no problem figuring things out
after that.

------
makecheck
This is generally fine but “homework” is a terrible idea (trend?) because it
isn’t looking from the perspective of the new hire:

1\. Unemployment can be extremely stressful and there is a _very good chance_
the person needs a job _NOW_. _Every_ little thing you do to dilly-dally
instead of making an offer is another hurdle that may send the applicant
elsewhere. Act as soon as possible when you see a résumé you like (get on the
phone, find out what you want to know, make a decision; if you bring them to
interview and they’re struggling, give feedback immediately; don’t have a
system where you wait 3 weeks to tell them yes/no).

2\. Any effort you expect an applicant to spend will be _greatly_ multiplied.
That person is _not_ talking only to you, they’re looking at LOTS of job
postings!! If every potential employer starts assigning “homework”, it becomes
a major hurdle and you can bet they’ll look a lot closer at an employer that
might provide a paycheck in a couple weeks instead of “homework”.

3\. You don’t need complex problems to understand if somebody can code. Let
them use whatever language their résumé claims they know, give them something
to do for a few minutes and see what happens. If they get stuck, give them a
solid hint and again, see what happens. You’re trying to assess things like
(a) if they appear to know what their résumé claims, (b) if they are the kind
of person you can interact with to get somewhere on a problem.

~~~
tptacek
I don't think you've thought this through carefully.

What do we think the median interval from "established intent to interview" to
"offer" is at medium-to-large software companies in SFBA? If I had to guess,
it's on the order of a month. Nobody's hiring homework process puts a dent in
that timeline. On the other hand, when done well, "homework" makes it easier
for engineering teams to make confident decisions, which expedites offers.

How much effort do you think it takes a typical candidate to get through an
interview gauntlet at a tech company with more than (say) 50 employees? It's
at least multiple full days worth of aggregate effort, almost always including
one _actual, on-site_ full day. To the extent that homework offsets on-site
interviews, it _reduces_ the amount of effort candidates need to put in.

~~~
walshemj
Is it your ignoring the fact that some good candidates will just look at the
job add see the home work coding tests and think "fuck you pay me" and ignore
the opportunity.

Who needs a full 1 day of interviews plus time for phone interviews for an
entry level job.

~~~
tptacek
I probably don't need to be competing to hire the "fuck you I'm too good to do
take-home problems" developers. Candidly: a significant fraction of those
developers don't do take-home tests because they're extremely good at
interviewing and... how shall we put this delicately... not so excellent at
the skills required to solve straightforward programming problems. It's
rational for them not to want to interview at take-home-test companies and
rational for me not to want to hire them at all.

That's why we started doing take-homes: because we kept having the experience
of interviewing people with impressive resumes and sterling references who did
not deliver at the caliber of the rest of our team. But even though we started
doing homework challenges as a way of screening out bozos who interviewed
well, we quickly learned something much more important, which is that for
every talented developer you'll talk to with a strong resume, there are many
--- probably at least 10's --- of talented developers who are underemployed
because they don't have strong resumes.

You also assume a job ad that basically says "We're the world's leading cat
sharing company. Wanted: senior developer. Requirements: 15 years Python
experience, must do homework problems". I don't blame you for assuming that
because for some reason not a single recruiting team in SFBA seems to be able
to write a decent job req, but that's not what our job reqs said, and we had
zero problems.

Finally: what's a company that hires software developers at any rank with less
than a day's worth of interviews? I think you'll have trouble naming one. Even
if you can, I don't think you'll have rebutted my point about the amount of
energy it takes to complete a typical interview-based process.

~~~
walshemj
From what I was told when applying for an avowed job at HMGCC for a Job
requiring DV (TS) clearance you seem to require as much if not more FTF
interview time.

------
pfarnsworth
It depends on how much effort we want to put into training entry level
programmers. Different companies have different requirements. If we intend to
train the programmer from the ground up, then we can probably look for
programmers who are smart, resourceful and self-sufficient. We can train the
rest, like good coding, etc.

If our expectations are higher, then what I look for is code sensibility. Do
they cut and paste the code without thinking "hey, this is kind of dumb to
do"? Do they see repeated code and think "this can be collapsed into something
more succinct"? This is the hardest thing to train for, and the most time
consuming from a mentor point of view, so if we have a high bar for entry
level programmers, then code sensibility is one of the factors on top of the 3
above that I look for.

I've rejected interns that worked for me from a permanent position because
after 4 months, they still had terrible code sensibility. I had to pour
through their code with a fine tooth comb because it was riddled with tiny
bugs, and it needed constant rewrites because they just didn't improve over
the course of the internship. I've had other interns that just "got it" and
wrote great code from the start, or only needed a few reviews and then "got
it". Those are the ones that will be productive quickly and won't hamper the
rest of the team with lengthy reviews.

------
cabaalis
We hire mostly junior developers. I give a simple whiteboard interview with 3
questions, and encourage them to solve it with mostly pseudo-code but have a
few "extra points" things that I'm looking for. Absolutely avoided any kind of
"trick" questions.

I like the homework idea. I don't know about "coming back" but maybe send it
to them a couple days prior to the tech interview.

~~~
jamestimmins
The one downside I've found with homework problems is that oftentimes the set
of requirements (or "suggested features") doesn't line up with the amount of
time you're expected to spend on it.

I've personally been in the situation of receiving a list of 10-12 required
features, with a stack that the recruiters know I am unfamiliar with, and then
been told to only spend four hours on it.

While this was an extreme case, I find that the unreasonable time expectation
is not uncommon. That is quite frustrating as an applicant.

~~~
NetOpWibby
I love the "this shouldn't take more than an hour or two" assurances.

------
kelvin0
I have failed many time at interviewing at technical interviews. I have never
failed any 'real' world job project/endeavor. I've built an entire org's
codebase and this company is now selling products worldwide successfully.
Design,coded,built,tested and deployed code in 4 languages , each element
being critical to the company.

I am an experienced dev, but couldn't tell you my birthday in a technical
interview. All the rest of the interview is fine, I only 'freeze' up at
technical questions.

Sucks.

------
Kequc
I cannot count how many interviews I've done where I've shown up with 10+
years of industry experience and been given an interview catered toward
juniors. If anything there is too much emphasis on interviewing entry level
software engineers. There are others out there with a proven track record.

I'm just underselling myself, shooting too low. At this point do I need to be
applying for management?

~~~
mikeokner
It's tricky for interviewers because a lot of candidates claim to be "senior"
yet can't even solve fizzbuzz on a whiteboard. Look for jobs with "Lead" or
"Architect" in the title, but beyond that, expect that you'll have to go
through some sort of stupid process as a bullshit screen.

~~~
monocasa
Can confirm, have interviewed people for a senior position with a wonderful
resume with 20+ years of experience, who seem to have never seen a computer
before. Flip the bit endianess of a word was one of the problems, and I saw a
guy who literally (supposedly) did embedded software for the space shuttle
just grind for 45 minutes and somehow not make any progress. Like it was
impressive. I would have expected a random walk to make more progress.

~~~
jtchang
Do you mean a word by 8 bits? Or 16 or whatever the architecture defines as a
word?

So 1000 gets flipped to 0001 ?

~~~
monocasa
It was a uint32_t in this case (we do a bunch of ARM stuff mainly so without
any other context, that's what 'word' means to me). And yeah, exactly, 1000 to
0001. We had an environment setup with a bunch of test cases and a stub
function to fill in. When you come in, push the button, everything compiles,
but most of the tests fail because the function is just

    
    
        uint32_t flip_bit_endianess(uint32_t word) {
          return word;
        }
    

I think there's actually a couple passing tests initially that flip to the
same thing.

And like I said, this is for an experienced embedded software position.
Ironically enough I care less about showing that you can code in junior
positions. Those are more "does it seem like you can learn?"

~~~
0xffff2
Is there an efficient solution to this? I'm relatively used to switching byte
orders, but I've never encountered a situation where I needed to flip bit
orders before. I could certainly do it with a bunch of shifts and ors, but
that seems awfully inelegant for an interview question. (A quick search
indicates that there isn't any instruction that makes this easier than bit
twiddling, but I'm not sure if I'm missing something.)

Where would this be used in the real world?

~~~
monocasa
> Is there an efficient solution to this?

To be real, I'm not even looking for a particularly efficient solution; just
"can you make the tests all turn green with ors and shifts" is what I'm
looking for.

There is a O(log(N)) N is the number of bits solution where you swap each pair
of bits, and then swap pair of pairs, and then each pair of pair of pairs, and
so on. I want to say that we clocked it to 15 cycles for a 32 bit word on a
semimodern superscalar core that can do each half of each pair in parallel
until you or them back together.

All of that being said, I'm really just looking for a for each bit, shift and
or it into the right place, O(N) solution. It's mind boggling how many people
with wonderful resumes can't do that in an interview time block. Hence why I
don't have a lot of patience for the whole "well I'm senior and have a great
resume, so you shouldn't have to make me do a stupid coding thing that's
beneath me" mindset.

> Where would this be used in the real world?

Yeah the problem itself is a little contrived, but it's a good use of the sort
of logic and putting the pieces in the right place at the right time sort of
skills you use in embedded programming, and more so HDL, which we require of
our software engineers too. And I have seen it in the real world a few times
on a serial line that has the wrong bit endianess.

~~~
minor3rd
I haven't used bitwise operators since college (approaching 8 years ago) but
wanted to try this as a fun exercise during the NFL playoff game. Here's what
I came up with -- mostly untested.

[edit] Note this is obviously for 16-bit integers.

int flip_endian_helper(int num, int i) {

    
    
        if (i == 16) {
         return 0;
        }
    
        int shift = 15 - 2*i;
        int workingBit = num & (1 << i);
        int partial = shift > 0 ? workingBit << shift : workingBit >> (shift * (-1));
        
        return partial | flip_endian(num, i + 1);

}

int flip_endian(int num) {

    
    
      return flip_endian_helper(num, 0);
    
    }

------
kelnos
One thing I'd be interested in, but feel like I never see, is a post-post-
mortem somewhere down the line, say a year or two later. This gets a little
tricky because you don't want to air your company's dirty laundry or say
negative things that could be easily traced to point to a specific person,
but...

In theory, I like the approach they've taken to hiring associates, but at the
end of the day, what really matters is: are the people they end up hiring
successful at their jobs? It might take a year or more to really answer that
question.

I don't feel like we're very good at this. As 'kinkrtyavimoodh points out in
another comment, we see a lot of "how to interview engineer" type posts. But
we never find out if, in the end, they were effective. This article claims
that everything went really well because they had (or believed they had) such
high-quality candidates that it was difficult to choose at the end, but they
don't actually _know_ that. They won't find this out after evaluating actual
on-the-job performance. It might have been hard to pick from their candidate
pool because their process failed to tease out flaws that are only going to
become apparent later. We just don't know, and likely will never know.

------
B-Con
> Notably, “potential” is nowhere on the scale. Many posts talk about hiring
> entry level engineers on their “potential,” but the term is seldom defined.
> Most people use “potential” to describe a sense that, even though a
> candidate currently cannot be an autonomous engineer, someday in the future
> they could be. We did not attempt to predict the future for any of our
> candidates but rather evaluated them on what they presented through the
> interviews.

Well...

> Some of the qualities we looked for in our associates in particular were:

> Resilience: Learning on the job is hard and we assumed that the associates
> would make mistakes and struggle through difficult concepts. We needed
> people who could endure these struggles and bounce back ready for the next
> challenge.

> Willingness to learn and the initiative to do so: Clover would assist the
> Associates in their growth, and provide teachers and mentors to help along
> the way, but any incoming Associate would need to be responsible for their
> own growth.

> Humility: This is an important trait for all Clover engineers, but we paid
> special attention to it in our Associates. They would have to learn from
> those around them, be respectful of others, and be able to take difficult
> feedback with grace.

> ...

> Deciding the technical skills to evaluate was a long process. We expected to
> teach our Associates most of the technical skills they would need to do
> their job, but we couldn’t accept candidates that were completely blank
> slates.

It really sounds like this is measuring the candidate's "potential".

I would argue they defined "potential" as "resilience, willingness to learn,
and humility", which they feel is an atypical definition.

(Personally, I think that's a decent set of criteria to look for in entry-
level engineers.)

~~~
zip1234
Resilience, willingness to learn, and humility can be the potential, but I
would also say that they are the candidates attitude. Yes, I would say that
attitude makes a big difference.

------
bg4
We give homework and it has been a successful approach. Even if it's a trivial
assignment one can gain a lot of insight on the candidate based on the overall
professionalism of their submission.

------
convolvatron
entry level is bar far the easiest to interview for.

with experienced people they always think they are at or near the top of the
scale, but dont understand that you have a different scale entirely.

with a junior person, ask them to describe a project they worked on. if they
'i dunno, school stuff', then you're done.

if you can engage with them meaningfully about their school work, or if
they've done personal projects, have developed any kind of insight, show a
spark of excitement or creativity then you're done - hired.

just keep an eye on them for a while to make sure they're engaged and
absorbing culture. invest a good amount of time in their success (mentorship,
pair debugging, whiteboard talks). if they dont start contributing more than
they are costing in time after a few months, find them another role or fire
them.

~~~
mynameishere
I love your attitude, but when I got out of school, _nobody_ gave a flying eff
about personal projects. This was before Google Analytics, so I had to track
IPs myself, but I think something like .1 percent of companies looked at my
website--and that might have been a false positive.

Even the company that eventually hired me didn't look at my website.

~~~
convolvatron
if you've spent years programming and dont have anything to say about it,
there isn't anything for me to go on. you obviously dont have any real
interest in it, and you're below the bar where i'm willing to invest the time
to see if you're teachable. what would you suggest instead? gpa? top school?
test? those dont seem particularly relevant.

edit: oh, maybe i wasn't clear. school projects are fine grist, you just have
to be able to say something interesting about them

~~~
mynameishere
I'm not arguing with you. Just saying the attitude is (or was) rare in my
experience.

------
wyclif
I saw the title to this piece on the HN front page, and thought, "Wow, I'd
like to see what a random tech company does with entry level engineers." Too
bad this isn't a piece of writing that really discusses that in detail. There
is no real actionable content here; it reads more like an HR post or marketing
for Clover. A shame.

------
grad_ml
I would say, try to test the fundamentals. Do they understands how memory is
accessed, have sense what kind of algorithm is used (though it would be too
much to code in 45 minutes), ambitious, have general idea to solve a problem,
can they see issues if things are not performant... Having a conversation,
comfortable conversation is the key, imo.

~~~
walshemj
For a general developer role that seems excessive not sure what you mean by
algorithem here?

I would assume you'd be asking about how the address and data busses work
which varies by cpu and architecture (von Neumann vs Harvard) presumably you'd
be happy if some one explained how PIC memory access works ?

~~~
grad_ml
With algorithms, I mean if the problem demands some optimization, be it
network flow, string compression or some kind of resource allocation
(something like register allocation in compilers), one should have the vague
idea which direction to pursue or approach. This requires somehow rigorous
course-work in school. Assuming everyone takes standard CS machine
architecture courses, having the breadth in multiple domain is, essential, imo
for a well rounded engineer. Now knowing(or having no idea) how memory access,
hierarchy etc works, they won't know when things just get stuck.

------
AlexCoventry
It'll be interesting to see how it works out. Without some kind of real-time
test of facility with abstract symbolic reasoning, I would expect them to get
burned. It's way too easy for people to fool themselves and others about that
kind of capability.

If they asked the subjects to modify the homework slightly during the
debriefing, that might be enough.

------
rdiddly
You with your sensible guidelines, your calibrated metrics, your well-defined
endpoints. Where's all the VOODOO!??! Where's the WITCHCRAFT!??!?!

------
Overtonwindow
I have only one modest proposal: No. More. Whiteboard interviews!!

