
The Programming Interview Experiment - matm
https://sourcegraph.com/blog/programming-interview
======
donretag
At this point in my career (professional developer since the late '90s), I am
extremely picky about who I work for. The interview is a two way street, where
the prospective employee also gets to ask questions to the potential employer.

In the described workflow, the candidate must spend several hours before
talking to anyone in depth at a company. This scenario would simply not work
for me. I will not spend hours of my time for a company that I potentially
might not want to work for.

~~~
jasonpeacock
Would your rather that they have you come on-site and perform the coding
challenge for half the day, then continue their normal interview process for
the other half the day?

Or worse, go through a normal 5-6hr interview loop doing random questions on
the whiteboard?

I think their approach is much more flexible and approachable. The overall
time commitment is the same either way.

~~~
donretag
You can achieve a mutual understanding during the initial technical (not HR)
phone screen. This approach eliminates the candidate's ability to ask
questions before a multi hour commitment. The traditional phone screen is
important. Most questions cannot be answered by HR.

I have a much better understanding a company after the question portion of a
half hour phone screen than I can get from searching the web for information
about a company.

------
fishtoaster
To start, I think your interviewing practices are significantly better than
the stock whiteboard-based version (and your criticisms thereof are spot-on).

That said, some concerns:

Leading with the source graph challenge seems a bit much. My experience has
been that it’s necessary to have at least _some_ filter for the unwashed
masses before requiring that much work of someone. At my last company, we had
a 30-minute non-technical talk with a founder (later replaced by our well-
trained recruiter/office-manager). This helped filter out the people who
weren’t what we were looking for at a really high level (e.g., an ops engineer
applying for a dev role, a really junior person applying when all we have room
for right now is a senior one, a dev who only knows pascal, etc) before making
them invest the time required for an in-depth take-home test.

Also, what’s with the 24-hour limit? It seems too long to prevent cheating,
but short enough to be annoying. For similar challenges, I’ve given people as
long as they want (up to a few weeks), but asked for a blow-by-blow writeup of
how it went and how long it took them. We may have gotten some liars, but we
never had someone arrive on-site who claimed it took them 2 hours when, based
on their skill, it probably took them much longer. People seemed reasonably
honest about it, and the convenience seemed to be appreciated.

Asking to see source code for a significant project is tricky. I know good
developers who wouldn’t have any code to show for that because everything
significant they do they got paid for, and can’t in god conscience show to
someone else. For example, I worked pretty long hours at my first job and
built some interesting stuff, but I didn’t have much time for side-projects
because of it. A friend of mine works normal hours now, but does contracting
on the side. Plenty of good code, but nothing he could show to someone else.
That said, asking to see some source is very high-signal, so it may be worth
the tradeoff of only selecting people with significant free side projects.

That said, it’s still a way better process than teasers and whiteboard
algorithms. If I were looking for a job, I’d definitely apply. :)

~~~
jasonpeacock
But what if the ops engineer can pass the coding challenge? He maybe trying to
change roles.

Or the dev that only knows pascal - if they can pass the challenge using
pascal then I'd think they could easily learn whatever coding language your
team uses. Learning a language is easier than learning coding.

That's what I like about coding, and about these blind challenges - it's
purely skill based. If they want to apply and can pass the test, they they're
worth talking to. If the wrong people are applying (and passing) then your job
description and challenge are wrong and need to be calibrated.

~~~
fishtoaster
It is possible that someone with little dev experience could complete our
challenge. We were mostly trying to avoid wasting time: theirs to complete the
challenge and ours to review it, if we thought it was extremely unlikely
they’d complete it.

However, when I referenced the hypothetical ops guy, I was referring more to a
misalignment of goals: someone looking for a job where they’d be doing more
devops stuff vs our need for a dedicated developer.

Regarding the hypothetical pascal developer, it really depends on your need.
I’ve been in the position to hire bright people who can learn our tool set on
the job, and that’s great. I’ve also been in the position where we can’t
afford a month or two while a new dev learns our language an framework before
they become productive. When in the latter position, it makes sense to weed
people you’re not interested in out early.

I would love it if a well-written job description would prevent unqualified
people from applying, but my experience has been that it just doesn’t. That’s
why some people use fizz buzz ([http://www.codinghorror.com/blog/2007/02/why-
cant-programmer...](http://www.codinghorror.com/blog/2007/02/why-cant-
programmers-program.html)). That’s why we used a short phone screen.

~~~
vonmoltke
> However, when I referenced the hypothetical ops guy, I was referring more to
> a misalignment of goals: someone looking for a job where they’d be doing
> more devops stuff vs our need for a dedicated developer.

If someone were looking for a devops-oriented job, why would they be applying
for your non-devops-oriented position?

~~~
bcbrown
Do you think it's a bad idea to have a high-level conversation between the
company and the prospective employee before assigning a take-home test?

~~~
vonmoltke
That depends on the attitude you go into the conversation with and whether
your take-home test can be automatically scored well enough for a coarse
filter to reject the people who obviously cannot do it. It also depends on how
many applicants you have. Spending half an hour per candidate gets onerous
when you have hundreds, so cutting to the chase in a semi-automated way may be
a better means of whittling the pile down than having a recruiter, founder, or
hiring manager filter it based on conscious or unconscious biases.

I was highlighting that line because I think it indicates a subtle bias
towards people based on background and assumes people are not making reasoned
decisions about what they are applying to. I understand that there are tons of
unqualified resume blasters out there, but I figure nearly all of them will
either ignore the take-home or submit such schlock that it will be easy to
automatically reject them.

------
jasonpeacock
"Bring the source code—we’ll ask to take a look at it."

Seriously?? What actual company do you know that would allow its IP to walk
out the door? And possibly to a competitor?

Additionally, projects are usually the work of a team, and are heavily
iterated upon. "your contribution" is usually hundreds of little changes
building upon other changes, not "this is the application I wrote".

~~~
beliu
This is a good point, and one I forgot to address.

We would never ask to see proprietary source code. Hopefully, a candidate
would have a side project or open source work they're willing to share. If
not, we'd definitely work around this. For team projects, we'd ask them to
describe their contribution. So far, this hasn't been a big problem, but we're
still iterating and making the process better!

~~~
usea
The article is definitely not clear about this. In fact it says "bonus points
if it's open source" implying that you expect to see proprietary work. It was
a bit of a red flag for me while reading.

~~~
bcbrown
I interpreted that as "bonus points if it's contributions to a big official
Open Source project, as opposed to your personal pet project with no exposure
or users".

------
RunForFun
In 12 years and more than 10 clients I have yet to run into any developer so
incompetent technically as not to be able to perform his/her job effectively.
However, what I do run into are individuals who cannot conduct themselves
appropriately, those who cannot work in teams, and those who overlook
delivering the value for their client/employer that they were hired to
deliver.

All of these individuals can code, no question. However, they are impediments
to accomplishing the business goals that my teams are tasked with solving.

I find it interesting that everyone I know who has been terminated or laid off
has been done so for reasons not related to technical ability, however
technical ability is the majority of what's interviewed for.

I think it is much more important to hire people who can work with others and
deliver the value desired, but yet this is not what most employers interview
for. In my opinion, if I can get a credible recommendation that a candidate is
technically competent, then I don't think spending an extra minute on
technical questions is important. I would rather be sure they they are
competent in all other aspects of the role.

------
bduerst
As someone who completed grad school and went through many interviews not too
longer ago - I can tell you that this type of process is a much, much more
enjoyable than the typical phone + whiteboarding interviews.

I think that some companies become so intensely focused on hiring the best
candidate that they sometimes lose touch with the fact that the candidates are
human too. Interviewees are rational and trying to find a job that has a best
fit. By taking the time to give them a project to work on, it allows them to
get a taste while demonstrating that you have respect for them.

My favorite interview was where a company gave me a sample data set, and
basically said "Do anything with this in python". I ultimately didn't get that
job (not enough experience), but I remember that company and the guy who I
went over my creation with, and now respect them more than the dozen other
companies who made me feel like I was on a conveyor belt.

------
fsk
What about candidates who don't have several hours to spend on a programming
assignment? Everyone seems to want one nowadays, and I don't have time to do
one for everyone who asks.

I've done a bunch of programming tests, without progressing to the interview
stage after doing them, and now I usually refuse.

~~~
georgemcbay
Agreed.

While I do think that there is plenty wrong with traditional programmer
interviews and that new approaches are needed, it isn't really a scalable
solution on the candidate's side to spend hours to days on challenges for
individual companies if everyone is asking for one. It becomes sort of like
everyone wanting their own app store, or their own always-running polling
service running on a desktop or mobile or whatever. If there are just a couple
of people doing that, no big deal, but once more than a couple of entities
start doing it, the system falls apart. If you're Google, Apple, Github, or
some other prestige employer you can probably get away with this system even
despite the scalability issue; otherwise good luck with it.

As a practical compromise, companies should be willing to waive the specific
challenge if the candidate already has their own existing shareable source
code they can point to, thus allowing a candidate to reuse the same example as
a portfolio piece with multiple companies.

However, few companies are likely to do this since the same basic ego problem
that breaks traditional programmer interviews comes into play ("Well we do
things very different here, we're very special, so we must make the candidate
jump through hoops X, Y and Z").

~~~
fsk
What makes it even more insulting is that, I do the assignment, and still no
interview.

If you give a programming assignment, then you should at least have the
decency to give an on-site interview to everyone who gives a solution that
compiles and runs and gives the correct answer. That's why I usually refuse to
do them now.

Not only am I spending a couple hours on someone I never met, the conversion
rate is so low that I'm convinced it's a waste of time.

Also it's insulting. You're asking me to waste a couple hours of time for the
privilege of maybe getting a chance to talk to you? When "maybe" turns out to
be "almost never", I've really soured on the pre-interview programming test.

------
bri3d
Why ask to see source for a project above and beyond the work-sample test
you've already administered?

I think work samples are a wonderful way to screen candidates (really, the
best way) but to presume that all qualified candidates will have an additional
sample of code (to the scale of a "significant project," even) available
upfront for you without requiring an additional commitment seems disingenuous.

You end up applying an implicit filter for "developers who feel confident and
comfortable contributing to OSS and/or possess copious free time for a
personal project," which probably isn't really what you were trying to select
for and tends to unfairly filter out certain types of candidate.

------
mathattack
Doing real code is very important. I'm curious if the OP has had any instances
of catching cheating. I've seen instances of this with work products in other
areas.

------
nanook
This is a great post and I agree with everything you say and I am glad you
guys are doing this.

I just wish you had more information about your company online
(Angellist/Crunchbase etc). I think few people would be willing to put in 3-4
hours on a coding challenge unless they had a good idea about your funding,
size etc.

I might have got a coding challenge from you a few weeks back but skipped it
because I didn't have enough information. Ah well.. might do it this weekend.

Hope this helps!

------
danbruc
Not in a single interview ever have I been asked about the runtime of an
algorithm. On the other hand it is mentioned in almost every article about
interviews for programming jobs. How many companies do actually ask such
questions? If your future job is mainly building standard business
applications, web sites or whatever, no one cares about theoretical runtimes.
If your future job is somewhere in research, game, database or operating
system development or some other fancy stuff, no one will care to ask for
trivial things like asymptotic runtimes. Personally I think most of the time
people - interviewers and article-about-interviews writers - just mention
Landau notation because they think it makes them look smart without realizing
how trivial it actually is.

~~~
jasonpeacock
All those companies should be care about algorithmic runtime, it's part of
performance and scalability. If you're not thinking about that then you're not
writing production software.

I ask about runtime at every interview (part of "coding, data structures, and
algorithms"). It is most definitely not a trivial concept - how do I define
the correct solution to a problem when there are multiple approaches?

The requirements for a correct solution (even simple coding problems) include
style, correctness, performance, and scalability.

~~~
danbruc
I meant that Landau notation is a trivial concept, not finding a good
algorithm to solve a given problem or determining the runtime of a given
algorithm.

------
fatman
What would happen if an interviewee 'flipped the script' \- gave the
interviewer a coding challenge? Maybe under the guise of assessing the skill
level of his potential co-workers? Is there any way an interviewer takes the
bait?

~~~
fatman
Or, what if the interviewer explicitly challenged the interviewee to come up
with an interview coding challenge? And perhaps even worked the challenge,
with the interviewee hinting, helping, and evaluating? Maybe it would be a
good way to demonstrate complete command of a subject, or the ability to think
on one's feet, or even to evaluate the candidate as a teacher/leader of
younger developers?

------
mamcx
This part is the key, to me:

>The second thing is the strong focus on topics taken from >classroom
algorithms and the lack of concern about >practical programming ability. A
candidate might somehow >remember that the way to implement an optimal string
>suffix matching function is to use a suffix tree, but >where is the question
that gauges the ability to create, >test, and deploy a complete application in
a reasonable >amount of time? Which skill is more important to you?

~~~
gtz60
I thought that was a an unfair comparison. One is a small piece of trivial
knowledge, the other is a huge amorphous amount of voodoo and experience. But
I agree that practical knowledge trumps theory.

------
Bahamut
I had an interview once where I did not have to write a line of code - it felt
very refreshing to have my capabilities respected, although I just somehow
lucked out it turned out.

~~~
fishtoaster
Interesting- I have the opposite reaction. I would (and have) refused to work
for a company that doesn't ask me to code in an interview. It may be nice for
me, but it means they probably didn't ask my potential coworkers to code
either, and I don't want to work on a team picked like that.

~~~
Bahamut
Well, my GitHub is public (and was probably convincing enough), and everyone
but me at the company was asked to code during the interviews - there was
evidence on the TV from someone who coded for an interview that morning even.
I was found from a local AngularJS meetup, and someone at the company was
trying to gently prod me towards interviewing with them over the course of
months...which wasn't so much of an interview process as seeing whether I was
interested, since they clearly wanted me for my expertise.

------
OhHeyItsE
When I get asked things like: “Write a function that finds the longest
positive-sum subsequence in an int array and runs in O(n) time.”

I really wonder if those places ever find people that can do this. I never get
offers from the places that ask this. I always get offers from the ones that
have a similar process to the one in the article. I don't think I've ever even
met someone who could answer these questions on-the-spot, with a marker on a
whiteboard.

~~~
fsk
“Write a function that finds the longest positive-sum subsequence in an int
array and runs in O(n) time.”

That's a horrible interview question. If you know the trick (saw the question
before), it's easy. If you don't know the trick, it isn't reasonable to expect
someone to figure it out in 5-15 minutes.

That's a brainteaser question disguised as an algorithms question.

~~~
d3gamer
But there isn't really a "trick" here, though. The point of these questions is
to figure out if the candidate has enough CS fundamentals to construct a
reasonable solution to a problem that they've never seen before.

~~~
fsk
There is a trick. Loop over the array, remembering the maximum and minimum. If
you've seen it before, the question is MUCH easier, making it a bad question.

------
akanet
Anyone interested in doing more complicated questions during the phone screen
should check out [https://coderpad.io](https://coderpad.io) \- you get a live
REPL or compiler (depending on whichever language you're using), and can run
code instantly from your browser.

The best part is the environment is shared between you and your candidate, so
you can both see the code being written and execute, live.

------
qianyilong
I would add that the respect for candidate time is critical. I applied to one
place and they requested a similiar coding project but this was just before
the christmas holiday and they gave me a deadline of christmas day which just
struck me as a little arrogant to think that I am really going to be doing
their challenge on christmas.

------
owenjones
Whenever a post like this comes up I'm always curious why so much of developer
interviews are focused an algorithmic performance. Is it because it can be
mathematically proven while so much of what otherwise makes "good code" is
subjective?

------
lauradhamilton
Asking candidates to write actual code for your real app could be legally
problematic.

Legally, the candidate will have the copyright to any code he writes unless
there is a "work done for hire" agreement in place.

------
iggym001
This interview process assumes candidates don't have anything else to do. I
think they should pay the candidate for time spent looking at their code and
making suggestions for improvements

------
bitcrusher
It's a total nit and pet-peeve for me, but one does not "Architect" code. You
architects domains.

You "design" code.

