
The Problem with Using HackerRank as a Programmer Screening Tool - drdrey
http://dreynaud.fail/the-problem-with-hackerrank-as-a-programmer-screening-tool/
======
stmw
This does a nice job of explaining why it's not a good solution. For the
company, it does not measure any kind of communication or collaboration
ability (which is so important for programming). For the candidate, it does
not give her/him any more information about people at the company or provide a
positive, interesting experience.

As an engineer, you work with other humans first and machines second.
Platforms like HackerRank seem to miss that, while better ones like
interviewing.io put it front and center.

~~~
user5994461
A code screening doesn't need to measure communication. It only needs to
measure basic coding ability. You can either write fizzbuzz in a reasonable
amount of time or you can't.

There will be plenty of opportunities to evaluate communication during the
other stages of the interview.

~~~
stuaxo
Fuck it, don't bother with any job that uses these things.

It's lazy on behalf on the employers.

~~~
analognoise
I'd agree with you, but you have to interview probably 400/500 people, all
with passable resumes, just to find basic competence (let alone the skills you
might really need).

What if you're talking about interviewing thousands of people, a number of
which are passable on paper?

It's a search problem, not laziness.

------
user5994461
I don't recall that HackerRank allowed to read and write to the filesystem.
I've never seen any stock test doing that.

The default test settings generate code for parsing the stdin input and create
the body of the program.

That company really went out of their way to make a terrible test. That's not
representative of HackerRank.

I used to write tests for one company. The tool is great and totally flexible.
A test should be taken as a representation of the company writing it, not as
an example of the platform.

~~~
drdrey
OP here, the test was definitely in HackerRank's library, not something
custom. I went through the exercise as we were evaluating HackerRank
internally, not as a candidate (important context I should probably have
mentioned in the post)

~~~
user5994461
That changes everything. It's not a real test at all.

There is a broad range of exercises, a lot of which are for coding
competition, way too long and too difficult for a short screening test.

I'd advise you to keep the majority of the exercises dead simple. An exercise
to print numbers from 1 to 100 will tell you a lot about the candidate. An 1h
dynamic programming exercise will stay unanswered most of the time, that
doesn't tell you anything useful.

------
nikisweeting
A hacker-rank screening test is exactly that, an initial screening test. It
tests basic aptitude and ability to implement a simple program to spec. It's
not a substitute for face-to-face pairing interviews and projects.

> 1\. they expect input and output in a necessarily contrived form

> 2\. there is no communication or creativity involved

1\. aka a spec 2\. not the job of a screening test

This whole post assumes the premise that a screening test needs to fulfill all
the requirements of a full interview process.

~~~
stmw
As a hiring manager, that's just not a useful "initial screening test" for me.

------
zebraflask
Nailed it.

Let's not forget companies that send algo-heavy tests for front end roles (my
personal favorite). Can't they come up with a front end version? Is it better
that they don't?

I'd find it, and similar sites, less objectionable if they were used as
interview "extra credit," rather than as gatekeeper screening tools.

~~~
contrarian123
I had a math teacher in middle school that put it pretty well: if you have two
candidates that are equally qualified, but one knows calculus and the other
doesn’t, you may as well go with the one that knows calculus.

From a hiring perspective, it cuts down the size of your candidate pool.
Interviewing is expensive, so it’s better to choose from 5 people instead of
30.

From a personal perspective, I don’t want to work with you if you don’t know
how to use a hashmap. There’s a difference between filtering duplicates
efficiently and validating a binary tree, so it does it some injustice to lump
it all together as irrelevant algorithms testing.

~~~
Jach
I'm very careful inferring information I didn't actually test for. If the
output of my process ranks two people the same, I can't do anything but flip a
coin for one and recommend hiring both. I can't go back to their resumes and
see "ahh yes, this person knows calculus, might as well go with them." Because
the other person might know calculus too but didn't think it relevant to put
on the resume.

If the process is giving equal outputs to multiple candidates, then you might
need to fix it to be more granular. There's plenty of room for subjective
criteria, e.g. it shouldn't be hard to have a few ranks on "seems nice to work
with" metric and so it shouldn't be hard to have a process where equal outputs
are unlikely.

On the other hand there's the concept of a journeyman in the craft world,
anyone with that level of skill is basically interchangeable for the usual
jobs. So you might by design have a test that just outputs "at least
journeyman level" or not, and you can just hire the first one you find.

~~~
johnsonjo
Some people are different and will differentiate candidates by sometimes
seemingly unrelated things. I remember hearing a story in my Engineering
Communications class that someone was selected for a job over an equally
qualified candidate for putting down that they liked to play chess as a hobby.
Apparently the person hiring liked chess and thought it stimulated the mind,
and I believe they also thought it gave a more human feeling to the candidate.

------
user5994461
Some help for the OP next time: [https://thehftguy.com/2016/07/13/cracking-
the-hackerrank-tes...](https://thehftguy.com/2016/07/13/cracking-the-
hackerrank-test-google-to-the-rescue/)

~~~
drdrey
> Special kudos to the guys who are able to solve a problem -with a perfectly
> optimized solution- in less time than it would take to actually read it =D

oh my

------
collyw
The problem is that it is completely unrelated to the job in most cases. That
may put off some decent candidates.

~~~
stmw
Right, as I said above, doesn't give you any sense of what other engineers are
the company are like. Nothing wrong with algorithmic coding interviews, but it
should be coupled with human interaction.

~~~
expertentipp
Wrong - it does. Such process in company indicates that the engineers employed
there didn't mind to go through such process and don't see a problem in
imposing such process on others i.e. people with volatile visa/legal situation
(e.g H1B in US, American in EU, Russian/Ukrainian in EU), and probably awfully
underpaid (I mean the salary range 40-50k USD in a developed EU country).
Basically company is cherry picking engineers willing to engage in race to the
bottom.

------
expertentipp
Got this HackerRank link from the company I applied to. Five questions from
completely different domains (database query, graph algorithm, construct a
regular expression, etc), 120 mins. Submitted blank solutions. You need cheap
compliant robots, not humans - stop looking for them.

~~~
jon_adler
I would contend that HackerRank has done a good job of screening and filtering
on this occasion. Given clear instruction, the only skill demonstrated was
insubordination.

~~~
user5994461
I would add that the topics given are paramount to development. Refusing to do
anything demonstrated that he doesn't want to do any work -or can't- for the
position he applied to.

~~~
expertentipp
The topics yes, but on-demand, sprint-like solving quizzes in these topics,
and context switching between all of them in short timespan, are these the
requirements for the position? Then the compensation must be respectively
outstanding? In most cases - no, it isn't.

Seriously, submit your current workforce to this smartass test under condition
that they need to pass it to come over to the office tomorrow.

~~~
user5994461
Don't be so quick to judge. Answering any question might have been enough to
pass the screen, but you will never know that because you didn't try.

My current workforce had to pass tests much deeper than that. Using database
queries, graphs and regex is part of the day job.

~~~
expertentipp
> Answering any question might have been enough to pass the screen

Not in this area/location, not in case of this kind of company. I know of
cases where the score 67% was too low (2 tasks on 100%, 1 task 0% but missing
couple lines of code for 100% solution). It's basically partially automated
filter with non-technical people sending rejection emails based on score in
the list without digging into solution. I gave up on it not from arrogance,
but based on experience.

------
cdicelico
I feel like you took the specifics around how _your_ evaluation was done and
_your_ choices regarding how you solved the problem and conflated those with
the entire HackerRank platform, which seems at best disingenuous and at worst
intellectually irresponsible.

Regarding the idea that "they expect input and output in a necessarily
contrived form", they are giving you a problem to solve. You could say any
expectation around input and output is contrived. As a professional, you will
not always be able to control this, and need to be able to work against the
constraints that are imposed on you.

Regarding the idea that "there is no communication or creativity involved",
this is just flat out wrong. You might have had a bad experience due to the
way the exercises were set up by the interviewers, but that's not because of
HackerRank. What I did is limit the allowed languages and progress people from
initial code puzzle screenings to pair programming sessions, where the
communication and collaboration evaluation comes in. In every case (puzzles
included), there is plenty of room for creativity. In fact, in my own hiring,
I have seen a very wide range of different solutions—some very interesting and
creative!

I found your choice of Bash to be pretty odd. I'm not sure I'd allow people to
write their solutions in Bash, because I realize that a lot of it comes down
to fiddling with things like the sed expression you described, which I don't
care about. If, as an employer, I want to evaluate your problem solving
skills, I'd prefer languages that abstract as much as possible away and force
you to focus on the problem as a whole. If, as an applicant, I wanted to
highlight my ability to solve problems with code, I would never in a million
years choose Bash.

It seems like the way the company you applied to set up the HackerRank
platform, mismanaged their communication with you, and selected and configured
the questions combined with your choice of Bash, as well as your own
personality idiosyncrasies (like getting stressed about the countdown instead
of focusing on the problem & remaining calm) came together to create a perfect
storm for you, but in my own experience, I've really enjoyed doing their
puzzles. I've also had great success using their platform for my own hires and
have found that the platform does a great job of highlighting exceptionally
skillful engineers and reducing bias in the hiring process. Our team avoided
several very bad hires who looked great on paper with the help of the
HackerRank platform. So, I think that while it's easy to blame HackerRank,
that's not where the real problem is IMO.

The thing with HackerRank is that a LOT comes down to the hiring manager. If
s/he doesn't do a good job managing the candidate pipeline, communicating,
setting up the puzzles, doing the pair programming sessions, running
interviews, etc., then people are going to have experiences like yours.
There's no magical platform that will solve for a bad hiring process.

~~~
drdrey
Thanks for the detailed feedback! Here is what I mean by "necessarily
contrived IO": in a more realistic setting, we might have to answer a question
like "what host returns the most 500s?" coming in an email or on a support
channel. Or one I actually had to answer a couple weeks ago: "what are our
most frequently firing alerts?" That is all of the problem statement right
there, not a detailed description of the formatting of the input and the
output. Just a user with a question, so you have some work to do to understand
what is being asked and how you can find the most time-effective way to answer
it. I might have follow up questions like "over what time period? For a
particular service or all services of team X? Just in prod or in all
environments?", etc. The user also didn't ask for a specific format, I just
need to get an intelligible answer with reasonable effort. So based on the
context, maybe bash makes sense to answer a one-off question rapidly. In the
context of a customer facing app, maybe introducing some abstraction makes
sense (again, context). This is whst I would consider a more realistic "on the
job" setting, you have to understand users and make trade offs and understand
constraints and it's all a little bit fuzzy. In contrast, an online judge
necessarily imposes more rigid constraints because of the automated
verification -- I don't really see a way around it.

~~~
anilgulecha
(I work at Hackerrank)

> an online judge necessarily imposes more rigid constraints because of the
> automated verification

The platform is evolving to not be as rigid anymore as well. For example, we
have a product (in a closed beta) which assess candidates on complete projects
(eg: build a nodejs app that defines this api and behavior), and testing and
scoring is based on unit test cases.

This way, you are testing for more real-world usecases, and also allowing the
flexibility beyond straight IO.

~~~
finnthehuman
That's still just as rigid, it's just increasing scope of the deliverable so a
smaller portion overall has to be dedicated to complying with your rigidity.

