
Ask HN: Are my technical interview questions too hard? - yeldarb
We just finished hiring our first 2 JS developers in a while and I was surprised how many seemingly perfect candidates <i>completely</i> failed my technical interview.<p>Our funnel was 150 applications -&gt; 75 potentially viable applicants -&gt; 20 interviews -&gt; 8 finalists.<p>I thought all 8 finalists would breeze through my technical interview but, in reality, 6 of them completely failed it. I&#x27;m trying to determine if I&#x27;m just horrible at sniffing out bad programmers or if my questions were too hard, bad, or both.<p>The format was a 1 hour in-person interview using a pre-setup JSFiddle to perform the following tasks. (I made it clear that asking me questions, using Google, using libraries like jQuery, or whatever else you&#x27;d do during a real-world scenario was perfectly fine). In the comments below I added details about why I asked certain questions and what I was looking for.<p>1) Create a 25x25 red circle using HTML&#x2F;CSS<p>2) Place a matching green circle next to it.<p>3) Now make 10 balls in the HTML instead of 2.<p>4) Using CSS, make their colors alternate between red and green.<p>5) Empty the HTML pane and re-create the balls using JavaScript.<p>6) Now make it a 10x10 grid of balls.<p>7) Switch the direction of the stripes (from vertical to horizontal usually).<p>8) After a delay of 1 second, animate all of the balls&#x27; color to black using jQuery&#x27;s animate function.<p>9) Now, instead of them all animating at once, animate them one by one.<p>10) Read the documentation for async.js&#x27; eachLimit function and use it to animate the balls to black 3 at a time and then once they&#x27;re all finished, animate them all to blue.<p>11) Write your own eachLimit function that matches the signature of async.js&#x27; and use it to do the same task.<p>Most candidates were struggling by task 9. Only the 2 we hired were able to complete all 11 tasks.
======
db48x
Any time you want to know if you succeeded at some task, then you have to
compare your performance against your goals. In this case, you hired two
people at a cost of $X. If your goal was to hire two people then you
succeeded. If your goal was to hire four people, then maybe you failed. Since
you can just repeat the exercise N times until you hire two more people, then
it just depends on whether $N*X fits in your budget.

As for the questions themselves, I'd say that they are too finely graduated on
the low end, and not finely graduated enough on the high end. This means that
you ended up in a situation where you can't tell if one of your candidates is
better than the other(s), while you can tell a lot about the relative skill
levels of the people you didn't want to hire. You want more discrimination at
the high end so that you know which of your candidates you would like to hire,
and which you must hire.

The other thing that you could do is just be willing to hire junior engineers
who need training, alongside the senior engineers who can train them. Anyone
who got question 8 correct, for instance, but not the rest, could have been a
good choice for this role. They would likely have learned a lot over the next
year or two, and then you'd have a lot of capable engineers.

~~~
shoo
Another perspective would be to try to model the costs of the hiring process,
and how the overall cost per hire might change if you could adjust the
survival rates for each stage of the funnel (or other parameters, such as what
the budget is for the role, or flexible working conditions, or how long you're
willing to train people, or whatever)

e.g. i imagine the largest cost is going to be onboarding the people that
you've decided to hire, not the stages in the preceeding
application/scheduleinterview/interview/techinterview/offer steps, so however
you fiddle with the funnel it's probably not going to make the process 2x as
cost efficient, but there is some risk that the funnel is perhaps selecting
for the wrong people .

~~~
yeldarb
Agree, the post-hire costs seem easier to quantify.

One of the things I struggle with is what you mentioned above -- what's the
opportunity cost of losing people who have a higher ceiling / learning
velocity but less immediately applicable experience.

I don't know how to measure that before hiring someone though. Is there a good
way to assess that?

------
sunstone
A talented software guy who doesn't know javascript could well make a better
long term hire than a less talented guy who does. On the other hand any decent
degree holder ought to be able to do this mostly trivial test given enough
time but if this is a sit down test then all you've discovered is who is most
up to date with their javascript and not who would be the better hire long
term.

It ought to give you a bit of self reflection if you realize that your
solution to the problem you faced has been pretty sub par.

------
yeldarb
Reasoning behind the questions:

1-7 - basic HTML/CSS/JS proficiency (and ability to effectively Google for
help if they didn't know about CSS' nth-child or whatnot)

8 - get them familiar with the jQuery animate function (which I wanted to use
later as a visual proxy for any async function) -- I was more than happy to
link them to the jQuery docs or explain the parameters here if they weren't
familiar with it.

9 - testing understanding of async functions and how to use callbacks. Ideally
they used recursion but I would have accepted a calculated timing-based
solution using setTimeouts as well.

10 - wanted to see how they'd go about learning something new given the
documentation -- async.js' docs don't have great examples included so I was
hoping to see them search for them on Stack Overflow if they didn't quite
understand the callback formats.

11 - algorithm design and recursion. But mostly I thought people would get
stuck here and I wanted to see how they dealt with that and what they did to
get themselves un-stuck.

~~~
zerr
Take home would make more sense for these... Researching/Googling/SO-ing under
1h limit and when someone watches you, breathing behind the neck, should be
uneasy.

~~~
dyeje
This. The timeline is a bit tight for a live interview. They have about 5 mins
per task. I think you're probably filtering too hard on people with previous
experience using jQuery and don't have interview jitters.

My suggestion would be offload some of those basic questions to an automated
at home code test. It'll filter out candidates who can't FizzBuzz, so you're
left with a more reasonable 1 hour technical interview with less pointless
candidates.

------
shoo
this might be a good way to filter for people who are fluent in the stack &
dev environment you are measuring with, and have an intermediate or better
level of base programming ability. this isn't a way to measure for other
skills you might want people to have in a dev team (e.g. structuring code
sensibly, controlling technical debt, experience with production deploys or
whatever), and you're potentially going to miss highly capable devs who simply
aren't fluent in what you're measuring, yet might be able to ramp up to be
highly useful in a month or two.

i'm not a frontend dev , but having an 11 step technical question in only 1
hour seems a bit excessive. i'm more used to having a single technical
question in one hour - with a few facets and some asymptotics, with an
optional harder extension question for the less common person who can get a
good solution to the first problem rapidly.

it kind of depends on what you want for the role --- do you have a bunch of
boxes in the org chart where you want people who are intermediate-ish and can
hit the ground running cranking out new features? or do you want some people
with really good potential and are prepared to invest a bit in training them?

instead of focusing on the technical interview stage of the pipeline, another
option would be to try adjusting the earlier stages. What evidence do you have
that the filtering from 150 applicants down to 8 finalists is not throwing
away a bunch of potentially good hires? e.g. What would happen if you tried to
run all 150 applicants through the technical interview first?

~~~
yeldarb
We ordered our steps in the funnel that way because we wanted to prioritize
people who wanted to work for _us_ (as opposed to getting _any_ job or just
the top paying job) and who had good communication skills and seemed like they
would make a good fit with the team.

The technical skills are important, for sure, but we'd rather have someone
less experienced who was motivated to learn because this was their dream job
than someone with tons of experience that didn't really care about having
_this_ job.

That's a good point about the timing; I did allow people to go over the time
limit if they seemed like they were still making progress and needed it. Prior
to the interviews I thought steps 1-8 (which were setting up for 9-11) would
take about 15 minutes and we'd have the remaining 45 for the meat of the
interview.

~~~
lordCarbonFiber
One rule of thumb I've found when constructing interviews is take how long it
would take you (or any other member on your team) to do the problem cold and
then multiply that by 2 to 3x.

This gives two main advantages. The first is you have plenty of time for a
conversation. Even assuming ideal conditions it takes between 2 and 3 times
longer to walk someone through a solution as you implement it than just to
incept an idea. Second is the interview platform is all ready stressful and
deeply unnatural so building in some buffer for that necessary context switch
helps you extract more signal.

That all being said, if you beat a 2% rate on your total funnel you're doing
pretty well. Most people just aren't going to be a good fit; that's just the
reality of this industry with huge choice for the candidate (you almost
certainly will not be anyone's first pick; the numbers are against you), near
limitless specializations, and no real qualification body.

------
gonyea
This endless interview question is really stupid. If you’re in an interview
saying “OK, NOW!” then you’re a huge waste of time.

Ask specific, probing questions to assess knowledge in your ideal areas, in
some combination with an open ended problem solving question.

~~~
yeldarb
What worries me is that I had done a bit of that previously in the first
interview. Some of the people who seemed perfectly able to talk about things
in theory wouldn't have passed FizzBuzz.

For this position we need people who can actually write code so testing coding
ability seemed relevant. Hiring someone who can talk about code but not write
a for loop or if statement wouldn't have been a good hire.

~~~
gonyea
The open-ended problem solving question is a coding question.

------
mabynogy
You wasted hours of people's life for no result. You've probably misjudged the
good candidates and excluded them for small details.

You should look at a people deeper. Listen to them more carefully. Try to
understand what they are interested in. You don't need their resume nor tests
for that. If you feel one is interested by programming, hire him immediatly
and try to work with him. If you were wrong, assume your responsability.

If you can't do that, send me your candidates. I'll tell you who are the
goods.

~~~
duxup
I'm not sure what that first sentence means, but that seems like a bit much of
a response for an interview...

A lot of people are "interested in programming" that doesn't make them a good
hire, or that they'll ever do the job. I think the interviewer wants folks who
can do the job, now.

~~~
mabynogy
As the comment above explains, recruiters can do that because we let them do
that. I advise people to ask for money to pass a test. The test is the
problem's recruiter, not the candidate.

A lot of people say they are interested in programming but very few are.
Programming is difficult. More and more people do that for the social
position. They should do something they are good at. Most of them who
persevere in computers become managers.

If a recruiter isn't able to evaluate someone by talking, he shouldn't recruit
people.

~~~
yeldarb
I'm not a recruiter. I'm a startup founder. And I'm working on hiring people
I'm going to work with every day.

As much as I'd like to hire every person interested in coding, unfortunately
that would be detrimental to our core mission of building a product. And in
the medium-run it'd be bad for everyone anyway because we'd just go out of
business.

I definitely learned I can't evaluate people's technical ability only by
talking... which worries me. Because prior to this I expected that I would
have been able to predict who would be able to code and who wouldn't. Some of
my "favorite" candidates before the technical interview wouldn't have been
able to complete FizzBuzz.

~~~
mabynogy
You should only hire those who are interested by what they do. The others are
bad and won't be productive.

If you can't evaluate if you can trust someone (for something), how can your
business be successful? Relationships are everything in trade.

Embrace the risk and pick someone. The one you trust the more.

------
dylanhassinger
I think I would pass your interview but it seems like a lot of busy work. IMO
asking devs to talk through these challenges, instead of actually do them,
would be better bang for the buck and more respectful of the candidates' time.

------
jjk166
I'd say drop the first 7 questions. You have no reason to doubt anyone who can
do the last 4 can do the first 7, and you have no reason to care whether
anyone who can't do the last 4 can do the first 7.

~~~
seattle_spring
You basically just said "eliminate core CSS / HTML questions and only ask
about jQuery."

------
thorin
I agree that you'd probably want to reduce the number of tasks. 5-6 would be
fine. As someone who does a bit of JS as well as maybe 10-20 other skills I'd
be happy doing most of these, but I would need to be able to look up the
syntax for a bit of this stuff. I think having a small number of tasks and
then talking around is better than having a prescriptive pass/fail. Almost
anyone could work this kind of stuff out with the internet and Stackoverflow
but would that make them a good fit for your role?

------
Chyzwar
These are not good questions for JS developer interview. Most of these
questions can be asked in a phone interview/online test. You want to have
people that know how to build stuff. If you are dragging someone for an in-
person interview then spend a bit more time on pair programming interview.

Good people do not need to apply anywhere to get well-paid jobs. You want to
poach people, find people from open-source contributions or your network
recommendations.

------
debacle
1/2/3 - Shows a knowledge of basic, modern CSS.

4 - Somewhat advanced IMHO, and how often do you need this?

5/6/7 - Tedious.

8 - Okay, showing some knowledge of JavaScript. 9 - Showing the ability to
write a control structure.

10/11 - Showing the ability to interact with the DOM and modify someone else's
code.

Why would you give this code test after the interview? Why not give it to the
potentially viable applicants first?

What kind of salary are you looking at? Anyone over 2-3 years of experience
should be able to answer most/all of these questions.

~~~
yeldarb
We gave the non-technical interviews first because we wanted to prioritize the
people who wanted _this_ job over the ones who seemed to be applying for
_every_ job.

If we're going to spend a ton of time onboarding and training someone we want
them to be happy with what they're working on so they're more likely to stick
with us longer term.

It was surprising how many naked resumes we received with no cover letter and
no indication that the candidate even knew what company they were applying to.

------
duxup
Just as someone learning / working on a career pivot into the web dev field.
You said they get to google... that's the key for me and makes the questions
quite reasonable in my mind.

I probabbly couldn't do many of those (well I could some) off the top of my
head, but if I can google, yeah I can do that. I know enough to get most of
those done.

Heck I'll try them on my own sometime.

------
arandr0x
I don't do frontend but they all sound fairly easy to me, although 9 is odd
(I'm assuming you want people to compute timeouts? To prove they're decent at
arithmetic? In which order should the balls animate, row-first or column-
first?).

10 is filtering out people who have anxiety about doing things in an interview
that they don't yet know how to do -- if your work environment doesn't require
learning something brand new while a manager looks over your shoulder, you may
be better off having it as a homework assignment.

11 is fine but the people who froze on 9 and 10 because they didn't understand
9, don't like math, had their brain freeze up on them at the "I don't know
what async.js is!!!" step, or something else don't get to it.

In addition, while I would be able to do this in an hour, the fact that there
are 11 points may be pre-emptively stressing some people (some people get
anxious at long lists of tasks, time-limited, that don't entirely fit in their
working memory), and things like reading through documentation can take a
bunch more time if someone is more thorough or less efficient at searching
(prefers tutorials to documentation, has trouble googling "async.js" because
of e.g. blog posts on ajax hogging up their search results page), if someone's
eyes are tired (at this point they have been coding at least half an hour),
and so on.

My suggestions would be, present the questions 1 by 1 not 11 at once, verbally
in addition to having the current question in writing. At the looking up docs
steps, have the link to the docs ready, or otherwise give a heads-up to the
interviewees before the interview ("you may want to read up on async.js as
preparation"). Otherwise, give the same exercise as homework.

After questions 1-8, if the candidate is satisfactory, say "great job! I only
have a few more questions", thank them for their patience, generally validate
their good answers. Make them feel like you're asking for their help vs. just
evaluating them.

~~~
yeldarb
Sorry, I probably wasn't as clear on the format as I should have been. I
didn't present them all 11 tasks up front, I walked them through them 1 by 1
verbally.

The questions flow pretty naturally from one to the next since it's mostly
building on the previous step each time.

And yeah, I tried to make them feel comfortable and less like this was a test
and more like a collaborative coding exercise. Wanted to try to make it as low
stress as possible but I know that's hard given the interview environment.

Re 9 - I was expecting people to use callbacks to do it rather than computing
timeouts (but a couple people did try to compute the timeouts and that was a
fine solution as well -- asking people about why they chose what they chose
led to helpful discussions). Something like this is what I would have done:

    
    
        var circles = $('.circle').toArray();
        var current = 0;
        var animateNext = function() {
            if(current >= circles.length) return;
            $(circles[current++]).animate({backgroundColor: 'black'}, animateNext);
        };
        animateNext();

~~~
arandr0x
It's not how I would code it but it sounds straightforward for someone who's
done any jQuery, frankly. How did your candidates fail on this one? Did they
just freeze up and not try?

~~~
yeldarb
They seemed to just not be able to grasp the asynchronous nature.

Mostly confusion between calling a function and passing the result as a
parameter and passing a reference to that function. But also struggling with
the recursive nature of calling a function from inside of itself.

I should also say it wasn't the syntax or JS/jQuery-specific aspects of it
that were the problem. Most couldn't explain in English/pseudocode what their
strategy for solving the problem would be.

~~~
arandr0x
You may be filtering your pipeline according to the wrong criteria, and
rejecting candidates strong in algorithms but weak(er) in JS specific or
framework-specific stuff at a stage that's too early. Or those candidates may
be unicorns -- usually the early-career people who like thinking about
recursion don't jump into applying to frontend roles, which they see (wrongly
IMO) as a lower tech, design-oriented kind of role.

If it's an entry-level role and you don't have good signals for the steps
prior to the interview, maybe consider a pre-interview assignment that
includes some asynchronous concepts, and administer it much earlier in the
pipeline. That's a slightly lesser investment but should be as good a signal.

If it's a senior role, then your questions are way too easy, but also you
should go on Linkedin and ask individual people with good experience to apply,
that's what everyone else is doing.

Edited to add: Well, I saw in your other answers that you are filtering
candidates based on "really wanting" the job very early in the process... I
work in a region that is not the hottest job market wise and even here no one
could afford to do this. Software devs are just too in demand... you're going
to get pretty mercenary candidates unless you go with personal referrals. If
the non-technical aspects of the candidates are more important, which with
your ordering of the hiring pipeline it sounds like they are, then you need to
consider hiring fewer people and growing less fast, pr hiring for fit first
and training the recursion skills later.

------
cbluth
> Most candidates were struggling by task 9.

Make the test 5 tasks long.

~~~
et-al
Good advice. The steps were kind of tedious to read. Maybe just trim it to
steps 6-11, which still tests HTML/CSS knowledge.

\- make a 10x10 grid of circles, center it in the page horizontally and
vertically

\- alternate the colors of the circle

\- have them switch direction

\- animate, etc.

~~~
yeldarb
Sorry, I should have explained. I didn't give them a list of tasks at the
start; the format was more conversational and I verbally walked them through
the list.

The reason I did it that way was to give several easy "warm up" questions at
the beginning to get them comfortable.

I like your idea of combining several of the first steps together though -- I
may try that and see if that's more or less stressful for the candidate.

------
bjourne
Glad to hear you actually hired those that performed best on the test! To
often companies serve out tests that are so hard that none of their _current_
developers can solve them and they still don't hire those that can. Yes the
test is really hard. I would have trouble solving it especially if someone was
staring at me while doing it. Those you are hired are no doubt stellar
developers!

------
evadne
They are not hard for the role

------
flukus
It sounds like you're hiring for front-end developers/designers to sling html,
css and a bit of JavaScript rather than "JS developers" which these days could
mean someone that works exclusively on the back end. If it's those back end
guys making it through the earlier filters then these results would make
sense.

It might not be the questions but the rest of the process.

~~~
yeldarb
We need people who can do both of those things. Unfortunately there's not much
Node-expertise around here so we didn't really come across any back-end Node
people.

I figure I can teach someone the Node-specific stuff if they have a solid
foundation. Similarly, I think I can teach almost anyone HTML/CSS so I wasn't
too concerned about those parts either; they were mostly just necessary as
setup for the later questions.

The things some people tend not to ever be able to fully grasp are recursion
and async programming which is why I wanted to make sure to include those.

------
master-litty
I would succeed in this interview but question the role. Critical thinking and
abstract thought seem discouraged with all these "do it or fail" questions.

If this is really a JS role, a Node developer could potentially be an
excellent fit with a little CSS and jQuery refresher but would fail the
interview miserably.

~~~
yeldarb
Sorry, I shouldn't have made it sound like "do it or fail" \-- I was limited
to 2000 characters here so couldn't do much more explanation in the post
itself.

Discussing how to approach the more difficult questions was a big part of the
interview and the thought process behind how they approached it was more
important to me than coming up with the "right" answer.

We didn't have any exclusively-Node developers in our finalists (I would have
loved that though; Node is my jam) but for those that had less JS-specific
experience we gave them a couple of weeks to brush up on HTML/CSS and pointed
them towards refreshing their memory on relevant JS topics (even going as far
as buying a few of them the JavaScript the Good Parts book to read before the
technical interview).

------
risto1
You're making the common mistake of not thinking through what it's like for
the interviewee in that situation. People are notoriously bad at designing
technical interviews because they think of it in the lens of what it looks
like for them and not what it looks like for the interviewee

\- You said it's a 1 hour interview. It's really more like a 30 minute
interview, because the interviewee will likely spend at least half of that
time (probably more) thinking about how you're judging their behavior rather
than focusing on the tasks. A good interviewer can prevent this sort of thing
from spiraling out of control, but it's never eliminated completely.

\- #2-#4 are redundant. If the interviewee can create a red circle, they can
create more than one of them with a different color. Interviewees will
underestimate their time management, so they'll take longer on these trivial
tasks. They might even overanalyze what you're trying to do here, thinking
that there's more to what you're looking for and spend longer on something
trivial

\- Questions like #3, #5, #9 are the kinds of questions interviewees will
overanalyze. They might wonder what you mean, or whether there's more to what
you're looking for. They can ask questions, sure, but you expect X and you
won't tell them that you expect X, and they'll spend stupid amounts of time
wondering what your X is. It's not an exaggeration either. Sometimes
interviewers implicitly expect certain things without making them explicit.
And you might only find out later (maybe even never), when they tell you that
they wanted something that you definitely could've provided but the
interviewer never specified. Or worse, you might spend additional time doing X
and then they actually wanted Y. Interviewees will try to think about and
tease out these things, which takes away substantial time from solving the
actual problem

\- Get rid of delay of 1 second for #8, this just slows down interviewees and
doesn't give you as an interviewer any special knowledge about the interviewee

\- #9 and #10 are basically the same question, get rid of one of them

\- Use common sense. You probably expected that most of those 20 people should
breeze through that interview. If they get something wrong or get confused
about something, don't necessarily filter them out -- Think about the
weaknesses in your own evaluation before doing that. Interviews are guidelines
not hard rules, don't make the process kafkaesque

\- I'm glad you're asking this sort of thing, because people should absolutely
care a lot about false negatives. The 4 that made it might be the best ones,
they might be mediocre candidates that were just good at your particular brand
of interviewing. You want the best people on the team, not the people that are
best at passing some weird interview that has little to do with their job

~~~
yeldarb
This is good feedback, thank you.

------
locklock
to be quite honest I would say this represents a lot of aspects of what's
wrong with tech hiring. it's basically an abstract quiz, you're asking people
to do random semi-related challenges while a clock ticks down and someone
watches over their shoulder. this is basically zero percent representative of
what actual coding is like and likely filtered out a few people who would have
done well at whatever the actual job you're offering is.

it's not that the questions are too hard, it's that the whole challenge is
bogus. if you have to ask these questions, make it a take-home challenge and
discuss the implementation with people when they come to the in-person
interview. being able to explain the hows and whys of already-written code is
much more valuable than being able to write code under pressure, and much
closer to what someone would actually be doing at work.

even better would be to figure out what exactly you're looking for in
candidates and design the interviews to reflect that. right now it seems like
you're looking for people who are good at coding under pressure... is that a
normal part of the job? it very well might be, but i doubt it. think more
about what you really want from people on a day-to-day basis.

------
curtas
You can merge some of the questions.

------
golfstrom
hire people, not test results*

*test results in a foreign, artificial, high-stress environment, using an IDE no one uses for normal work.

