
Algorithmic Interviewing – Optimizing to Hire the Wrong Developer - rcruzeiro
https://medium.com/p/9a3781d57c4b
======
timr
_" Being a great business web developer and being a great algorithm developer
are not mutually exclusive, but they are also not interdependent. The type of
developer that would perform well at this kind of challenge is potentially a
recent graduate, fresh from an algorithms class or one that is more focused on
micro-optimizations than the big picture."_

Oh, good grief. Being able to answer basic algorithmic questions in an
interview isn't a high bar, folks. It's not as if a basic tree-traversal
question is going to trip up everyone but Don Knuth.

The biggest danger you have as an interviewer is allowing a Stealth
Incompetent on your team. These folks all have Github profiles, points on
Stackoverflow, fancy blogs, webpages, etc. They might even have nice-looking,
public-facing projects. You can even argue that these people Get Things Done
(for some value of "things"). But they're still bad hires.

Stealth Incompetent programmers are great at using other people's code. They
can install gems, download jQuery libraries, and stick them all together. They
can make websites with Rails or Django or some other framework, and know
enough to be able to piece together functional "products" from parts. All
fine. But unlike actual engineers, Stealth Incompetents fall on their face the
first time you ask them to do anything that requires taste or independent
thought. They're the people who will introduce a horrible dependency on a bad
gem, or architect a solution that takes quadratic time, when they could have
trivially done better with a tiny bit of thought.

Algorithmic questions are designed to _catch_ incompetent people before they
can infiltrate your team. The "relevancy" of the questions to the day-to-day
work is immaterial.

~~~
zainny
I would argue that, say, a recent graduate who can absolutely nail algorithmic
questions would be the _most_ likely to screw up on the big ticket items that
matter (infrastructure, taste, independent thought) precisely because they
have zero real world experience.

~~~
timr
_" a recent graduate who can absolutely nail algorithmic questions would be
the most likely to screw up on the big ticket items that matter"_

Why do people keep implying that only new grads understand algorithms as if
it's some kind of fact? In my experience, new grads don't tend to know
algorithms any better than they know anything else.

Good engineers keep learning about algorithms because the knowledge is
timeless.

~~~
zainny
Ridiculous. Good engineers solve engineering problems. The huge number of
successful engineers out there in the world solving problems without
remembering their CS course notes tells you everything you need to know about
how important much of the algorithmic subject matter is.

~~~
timr
Uh huh. I can fix my car without knowing metallurgy and fluid dynamics, but
you wouldn't want me building you an engine.

Just because people are getting by doing the technical equivalent of changing
sparkplugs doesn't mean that I want to hire them to do something more
difficult.

------
zainny
This exact topic came up yesterday in the HN post "Bullshit Interviews":
[https://news.ycombinator.com/item?id=6304911](https://news.ycombinator.com/item?id=6304911)
([https://coderwall.com/p/yn9g2a](https://coderwall.com/p/yn9g2a))

I repeat now what I said then: I think there's a very simple rule anyone can
and should follow to not fuck up interviews: Don't ask people to do things in
the interview you wouldn't expect them to do in the job. Examples:

(1) Do people in this job regularly write code on the whiteboard? Lack access
to a search engine? No. Then don't do it in the interview.

(2) Do people in the job design spice racks for blind people? No. Then don't
ask them to do this in the interview.

(3) Do people in the job regularly find themselves coding something on the
level of fizz buzz? Yes? Go ahead and ask!

(4) Do people in the job regularly find themselves writing code to find loops
in a graph? Yes? Go ahead and ask (though, let's be honest - this is
_exceedingly_ rare in comparison to how often it is deployed in interviews)

Verify people have the skills they need to do the job you're hiring for. It
really is as simple as that.

------
kenster07
I find it curious how so many computer engineers tend to have borderline
religious beliefs when it comes to the topic of CS fundamentals.

This is an ineffective way to hire front-end devs, which is the scenario
described in the article. In Javascript, there is limited room to implement
fundamental algorithms, because many of them have already been implemented in
native code, and they will pretty much always outperform a customized
Javascript implementation for large n.

When doing front-end dev for a living, you will not be able to keep your
fundamental algorithms chops by just doing your job. You have to go out of
your way to keep them fresh. As a result, interview questions about
fundamental algos for a position like front-end development will be biased in
favor of those who are recently took CS algorithms classes, who do not
necessarily have any more intrinsic potential as a front-end web dev, and who
likely have less work experience.

As a consequence, it would seem that any employer that uses fundamental
algorithms as a major factor in hiring front-end devs does not have a
realistic conception of what the job actually entails.

And the same argument applies for any programming job where fundamental algos
are not coded regularly.

------
gamegoblin
I have mixed feelings about algorithmic interview questions. There is the camp
that says that you shouldn't memorize something you can look up, but I am of
the opinion that having something memorized fundamentally changes the flow of
thought.

If you just _know_ the lookup times of various datastructures, your thought
process is much less jumbled than someone who must look them up while problem
solving.

I'm not suggesting that it's necessary for someone be able to whiteboard an
implementation of a red-black tree, but knowing what it is, and its advantages
and disadvantages compared to an AVL tree or something is very useful.

As an aside, in a graduate CS class I took (about search engines), the prof
asked "What is the lookup time of a balanced binary tree" to various students,
two of whom gave wrong answers before she got to me. One said "n lg n" and the
other said "n". It was evident that they were basically guessing.

My point is that if you actually understand what a binary tree is, the lookup
time of it should be obvious. Same for most other basic datastructures, linked
list, hash table, etc. I think asked basic, high-level questions about these
are totally fair game.

~~~
ndr
[pedantic] Technically, look up on a balanced binary tree is still O(n) as it
says nothing about the ordering. If it was a balanced binary _search_ tree
then it would be O(log(n)) which I guess is the answer you gave. [/pedantic]

A part from that I totally agree with you on the importance of knowing by
heart the implications of some technical choices.

~~~
nostrademons
(I upvoted you because you're pedantically correct, but being even more
pedantic: if it's not a balanced binary _search_ tree, is "lookup" even a
valid operation on it?)

~~~
marcosdumay
Well, lookup is a valid operation on any dataset.

~~~
nostrademons
Not really: "lookup" is not part of the API (as taught in a typical data-
structures class) for stacks, queues, and priority queues, and technically
arrays/vectors let you lookup by integer but the O(N) lookup by object is only
an extension that most languages provide as a convenience.

If we really want to be pedantic, we'd have to define what the binary tree is
used for, because there's no convention for what "lookup" means in this case.
A heap, for example, can be implemented as a balanced binary tree, but the
only methods it provides are push, pop, and top.

------
jimbobimbo
I'm in a school of thought that algo interviews are useful, but not for the
knowledge of some particular algorithm.

I was usually asking people to write atoi() implementation without using high-
level constructs on the whiteboard, piece of paper or in the browser window.
Things I was looking for:

1\. If I'm interviewing for C# position and a person chooses VB.NET, that's
probably a red flag.

2\. If they choose C, hey! show me your pointers arithmetic, strlen() at the
beginning of the loop is a huge red flag.

3\. There're always giveaways about how person is familiar with the language
(and mind you, I do not expect any tricky stuff to be used).

But the important part is not the code - the important part is to watch how
the person attacks this token problem, how much time do they spend on it,
where they're when I come back in the room. Then we'll talk about what the did
in order to solve the problem and why did they do things they did. I won't be
running this problem to check if the output is correct or to see if it
compiles at all - we'll just talk it over. Will we talk about technology stack
in use at the company, assess person's architectural skills? We sure will. But
at the end of the day I'll expect this person to write some code at their job
and I need to be sure that they know what they are doing.

Does Github account help? It does to a degree. I'll sure take a peek to see at
the coding style or how the projects/code are structured, but honestly, I just
don't have time to fully assess someone's capabilities by reading their
existing code thoroughly. Plus, it doesn't give me any clues on the thought
process when they were writing that code.

------
moapi
Every now and then, these posts come up on HN people mention how bad the
interview process at certain companies is and how it should be changed.

I also work for a company that does these types of interviews. While I
wholeheartedly agree that asking these algorithms/data structures or other CS
fundamental questions is not perfect, precisely because it may induce quite a
bit of false negatives, the engineers who does end up successfully completing
the process at my company are very competent ones. So from an anecdotal
perspective, the process works fairly well in the fact that it does not
produce too many false positives.

~~~
seiji
There's a broken culture of companies thinking they should only hire the
smartest people in the world. Most companies don't need brilliant people, they
need competent people who will work great together.

[Disclaimer of bitterness: I'm not one of the smartest people in the world. I
get rejected after one to three interviews all the time. The author of the
post is right: nobody ever looks at your online code and your resume only gets
read once by HR to set up the initial pre-interview.]

Testing for concrete intro-to-CS level drivel may give you a company full of
people who spend 30 hours a week doing topcoder or project euler, but... you
don't need that (if you do need that, good for you, but most companies don't).

It also tricks the interviewee a bit -- "Oh, they're asking algorithms
questions! Maybe this'll be a fun CS research job." [six months into the job]
"Uh... all I've done is write SQL and fully library-driven backend website
code. Boring."

------
ArekDymalski
I strongly believe that for many companies such problems with hiring process
start much earlier - in the phase of defining the requirements (competencies).
When they don't know _precisely_ what they need (and both "ninja" and
"experienced front end dev" are far from being precise), they simply can't
succeed in measuring it.

------
xsmasher
If you're looking for people with the proverbial "smart and gets things done"
properties, you need to give them something to do and see how smart their
solution is.

I give a task, like "write some code to store data in X; here's the interface
I'd like to use" or "write some code to find X given data set Y". The problem
should have "layers" and follow up questions.

Some devs recognize the task and says "this looks like an application for
$FOO" and apply some $FOO. Others do not know the name of the optimal
technique, but are able to derive it on the spot from first principles.

Junior devs should at least be able to come up with a brute force solution
that solves the problem, even if they can't take it to the next level. If the
problem requires some deep "ah-ha" moment then it's not a great test; the
whole point is to separate applicants into strata.

------
lrobb
"My experience gave me the feeling that this was more of an interview assembly
line than someone really interested in my relevant skills"

That's exactly what it is...

Effective interviewing is hard, and like most things hard you won't be very
good at it without some practice and study.

Asking an algorithm question is relatively easy.

------
n1ghtmare_
I was recently interviewed by Google, didn't get the gig, didn't do very well
If I'm honest, however even though I'm a bit bitter, I'll admit that knowing
algorithms and data structures is important. I was rusty before the interview
so I hit the books and I was amazed how many things I've forgotten and I do
believe knowing those thing makes you a (much) better developer. Whoever says
otherwise, do yourself a favor and learn at least the fundamentals.

------
lhnz
Aside from the main point of this article, could anybody recommend a good
algorithms/data structures book?

I want something that gives examples of _when_ you should use particular
techniques/theory and the _benefits and trade-offs_ of doing so? That is, as
opposed to learning the rote implementations and time complexities, I want to
understand the practical application of knowledge...

I recently looked at some coding tests for Spotify [0] and found an
interesting article talking about the solution for one of these. It was
interesting to me since he just pointed out that something was a maximum
cardinality problem on a bipartite graph and then applies the Hopcroft-Karp
algorithm [1]. Now I don't care at all about being able to code any of those
things from scratch, but I'd love it if there was a book which I could read to
try and learn the names of different techniques and when/why they are
applicable. I want a better map and I want it to be written in english.

Any ideas?

[0]
[https://www.spotify.com/uk/jobs/tech/catvsdog/](https://www.spotify.com/uk/jobs/tech/catvsdog/)

[1] [http://alonso-vidales.blogspot.co.uk/2013/03/new-spotify-
puz...](http://alonso-vidales.blogspot.co.uk/2013/03/new-spotify-puzzles-
reversed-binary.html)

~~~
m_ke
Try Skiena's "Algorithm Design Manual" [0]. Another nice book would be
"Programming Pearls" [1].

[0] [http://www.amazon.com/Algorithm-Design-Manual-Steven-
Skiena/...](http://www.amazon.com/Algorithm-Design-Manual-Steven-
Skiena/dp/1848000693) [1] [http://www.amazon.com/Programming-Pearls-2nd-
Edition-Bentley...](http://www.amazon.com/Programming-Pearls-2nd-Edition-
Bentley/dp/0201657880)

------
JulianMorrison
I think my answer to the "write a sort" thing would be "are you kidding? I'd
use the one in the language's standard library, because I'm not an idiot. Or
if I had no choice, and was using a language and environment so primitive it
had no libraries at all, I'd write bubble sort because it's easy. Or if it
actually mattered I'd go read up on it and learn how."

~~~
nostrademons
Oftentimes that's the _first_ answer that the interviewer is looking for
(without the attitude). But then they want to see you code a MergeSort or
QuickSort up from scratch, and explain the difference between them.

Why? Because what happens when your data set exceeds available RAM? Then you
can't use the stock libraries, since they're all designed for in-memory
datasets. You have to rely on external sort, which is a cousin of merge sort
but has fairly different properties because of the vast difference in
performance between RAM and disk. Or what if you have multiple computers
available to you? Do you implement a parallel external sort over the network
as a library (or service), or do you rely on domain knowledge of the keys to
do some sort bucketing to each computer, like a radix sort? What are the
trade-offs between each one?

Software engineering is somewhat peculiar in that 95% of the time is spent
gluing pre-made components together, and yet the majority of the money is made
off the 5% that requires actual creativity and knowledge. Why? Because gluing
components together is a commodity. You can do it, but so can every other
college grad who knows how to Google StackOverflow answers on the web. The
folks who make $300K+ a year at Google/Facebook or get acquihired for a mil or
two a piece or found a startup that strikes it big are the ones who know how
all the "standard" stuff works on the inside so they can take it apart and
piece it together in new and innovative ways.

------
DavidWoof
Did we all read the same article? He complains about "algorithmic interviews",
the comments here are about AVL trees and sorting algorithms, but what he was
actually asked to write was "a string parsing implementation".

Well, OK, I suppose it's possible they tried to get him to write yacc, but it
sounds much more like they asked him to do some trivial loops and tree
traversal, at the very most some basic recursive descent parsing.

This ain't rocket science, it's not complex O(n) analysis, it's only slightly
above the fizzbuzz level. If you're writing code at all then I think it's
reasonable expect you to understand recursion and loops, even if much of the
job is just going to be hooking up jquery plugins.

------
coolrhymes
I don't agree 100% with the author on not asking them algorithmic questions.
Its a process on how they go about solving it rather than just the outcome.
Maybe for the web developer position, you might not need such type of
questions, but for a full stack developer the answer is unequivocally yes.

I made a mistake by hiring someone who knew django [who originally wrote
plugins in WP] and the coding was just plain horrible. One such example.

`for foo in queryset: foo.prop = 'i m bar' foo.save() ` Simple things like
this, where a good engineer would know because of O(n) nature of the code, the
so called programmer didn't.

And then there are guys who fork every possible repo, have blogs about how-
tos, etc but really can't code for nuts.

~~~
philangist
Coding noob here. Can someone explain the problem with the

`for foo in queryset`

implementation? Is it that for loops are inherently inefficient? What would be
a better alternative?

~~~
victorf
I don't see a problem with an O(n) algorithm to set some property on n
objects.

~~~
victorhooi
I think the issue is the call to save() (I assume this is meant to be outside
the comma).

Basically, it'll hit the database after each query.

~~~
philangist
I originally thought it might be that too, but it seems to me like you'll have
to hit the database n times for n queries no matter what. Unless Django has
some builtin batch save functionality.

~~~
victorhooi
Hmm, there's bulk_create(), introduced in Django 1.4, to create multiple new
objects in one DB call.

However, I'm not aware of anything that would do the same to update
properties.

------
thedufer
The argument about whether to ask algorithmic questions reminds of the "can an
airplane on a treadmill take off?" question. There are two sides who arr sure
that they're right and the others are wrong, but they're just answering subtly
different questions.

In this case, the people against always seem to come back to questioning why
you would ask something that's just memorization, while the pro-algorithms
side is pushing for questions that candidates wont know the answer to, so they
can see how they think.

I am in the second camp, if you cant tell.

------
capkutay
I personally don't look for someone to answer algorithmic questions correctly.
If they can just show the ability to solve a problem with a bit of complexity,
that's all that counts. It also just reveals a lot about how that person
thinks and solves problems which is absolutely an important factor of a
potential job-hire. However, I know some interviewers just keep score of your
answers..I don't necessarily agree with that.

------
codex
Do you hire a mechanic who knows how the car works in intimate detail over a
mechanic which knows how to fix cars? Depends. But usually if they know how
the car works chances are they are also great at fixing them.

In tech it is the same, with the additional wrinkle that you want to hire
aptitude over experience.

------
liruqi
But algorithm coding is just about personal programming ability. What if the
company is using a non-opensource lib? I don't know how difficult the string
parsing problem is. But interviewer may just unsatisfied about your slow
implementation.

------
je42
I think algorithmic interviews are fine if the interviewee has mentioned the
algorithms on the C.V.. In general, I only talk about topics the interviewee
has mentioned on the C.V. or are super important for the org/project/position.

------
rvivek
Interviewing process is constant optimization just like every other workflow
process in a company.

Disclaimer: I run interviewstreet.com which helps companies screen & hire
programmers using coding challenges. We have had a substantially HUGE number
of developers screened (might not be able to reveal the exact number) through
the process and using the data points, here's what we found (btw, this is also
constantly optimized)

a. Asking a 5-year experienced engineer to solve a graph theory challenge or a
complex tree problem is a pretty useless indicator. The goal of such an
interview challenge should be to test the problem solving skill of a candidate
which is essential across any programmer role.

Can this person actually take an array of objects, perform an operation to get
the result such that it works for any size of the array without throwing an
exception? The data structures used in the question should be simple enough to
start working on the challenge. You will be surprised how many errors, corner
cases come up which are often missed. And as you gradually increase
constraints, you can check their thought process of how their algorithm
changes.

b. Make the problems interesting - put in actual effort to make the problems
interesting, sometimes relevant to the problem you are actually solving. I
often send this link to our
customers(www.itasoftware.com/careers/puzzle_archive.html) and also help them
design problems like this. That's by far the most interesting publicly
available challenges. The other one is Quora, quora.com/challenges which is
slightly harder though but very interesting (and has proved very effective!)

c. Calibration: Surely github/bitbucket/SO profiles are important and can
serve as a data point. However, it's probably going to be very hard to
calibrate. A web server coded in Python vs a new MVC framework written in PHP
- who is a better candidate? Who has actually thought through the design of
the problem better? It's tough to evaluate. The programming challenge
interviews act as a data point (one of the interview rounds) to check these
skills in a contained problem/environment helping the company calibrate the
performance against the rest of them.

The problems should involve a combination of the ability to write good code,
focus on problem solving skills and not on remembering an algorithm from the
CLRS text book, intelligent thinking (not to be confused with weird
math/geometry problems unless your company is working on that domain) and
_importantly_ the ability to catch hints and solve it in a better way. This is
a data point, an essential one but there are more to technical interviews.

If used in the right way, it can prove to be hugely effective for your process
not only in streamlining but also as a way of generating interest from
potentially interested programmers.

~~~
zainny
I am actually pretty decently ranked on Interviewstreet because I do have a
personal interest in algorithmic stuff (I was top in Australia for a little
while :)). I used to do your codesprints just for fun.

My feedback to you would be that Interviewstreet is heavily heavily geared
towards academic algorithmic questions which the average developer would not
have a chance in hell of solving. For example, in order to solve some of your
old Hacker Rank questions within the time and memory constraints you pretty
much have to know the studied and documented solution you would only ever be
exposed to in doing a masters degree in CS. Heck, your problem descriptions
were often thin veneers around the exact academic statement of the problem.

Despite the fact that I'm someone who actually does excel at these types of
problems, I think they are absolutely the wrong way to hire people and
whenever I've seen companies actually use Interviewstreet it makes me cringe -
and yes I know companies set their own questions, but you lead by example and
so most questions are abysmal.

