
How to get hired (or, 'The silly story of interviewing in the valley') - sgrove
http://trapm.com/how-to-get-hired-or-the-silly-but-adorable-st
======
timr
_"Then, acting as though this was the first time I'd seen this problem, I
would ask if it was ok if I thought aloud as I worked my way through the
problem on the board. I'd mumble to myself about moving-this-piece-over-here
and-now-we're-going-to-get-this, and lo-and-behold, I accidentally solved it
in constant memory space, in C - a language I didn't even claim to be
particularly good at! Only someone with amazing problem solving skills would
be able to do that their first time thinking through this problem so
earnestly!"_

All this tells me is that tech interviewing is horribly broken. What we have
here is a story of a guy with a little bit of experience who has (if his story
is true) successfully learned how to game the interview systems at dozens of
startups. If you interview people, this article should terrify you. You can
argue that it isn't "gaming" if the guy turns out to be a good employee, but
that's wrong. If he can trivially game the system, so can anyone else with the
ability to go to dozens of interviews and remember questions.

There's a fix for this problem: make the interview about more than the answer
to the stupid question. If the candidate answers your question a little too
quickly, ask something harder. Chase down a detail. Pick an experience on the
resume (and not an obvious one), and dig as deeply as you can. Good candidates
know the details. Bad candidates don't have details you can chase.

If you're interviewing correctly, no candidate should be able to come in and
snow you with some regurgitated C code from previous interview experience,
because you'll be able to out-flank them at every attempt to cough up a line.
Unfortunately, there's a corollary: you need to know the subject better than
the person that you're interviewing. If you don't, you have no hope of
screening for good people.

~~~
Evgeny
It's not really gaming the system. He solved the problems, himself. He asked
questions, learned and then came up with more and more effective solutions.
Aren't these exactly the traits an employer should be looking for? Didn't he
put a lot of effort and time?

Here's my own example. I've been recording some of the problems I have to deal
with at work, and some of the things I come across in my spare time in the
form of a blog, for the last 3 years (with some gaps here and there). I
recently started looking for a better place to work. I included the link to my
blog in my resume. I got an offer recently, it will be a substantial increase
in salary. I only interviewed with ~5 companies to get an offer, that's my
easiest offer and shortest job search ever.

Now, would you say that I 'gamed the system' because I had a blog, and most
don't? I guess you can say so, because when I started my blog, I had that in
mind - it may be useful next time I look for a job. But I did not mindlessly
type stuff in - I made sure I understood it as much as I can, and quite often
writing the solution down helped my understanding. Same with this guy - he did
not just memorize how to do it.

 _I had one YC company ask me to create a linked-list structure in ruby, and
then reverse it. I did it recursively, mentioned that it would blow the call
stack, and then re-implemented it iteratively. I also remember there was some
problem where I used symbols rather than strings and that would cause some bad
memory leaks over time in a long-running process._ \- From this passage, he
might not be the "top 1 percent" everyone tries to hire, but at least top 5%,
that I'm sure of. Or maybe he gamed my impression too.

~~~
kamaal
The definition of gaming the system is different here. These days unless you
are implementing a library yourself. You are probably never ever going to even
reverse a linked list ever in your life. Or deal sit down with a math text
book when you code. Those days are probably gone in the 80's.

That means asking these questions is akin to you taking an approach to judge
an candidate on things which have nothing to do his job. That's why so many
programmers don't about all this. The reason is for 99% of the work they are
likely to do _they don't need to know it_.

Under such circumstances when you ask these sort of questions the only way
candidate can deal with them is not by actually learning how to solve them.
But learn them just for the interview sake.

So you end up hiring some one on a criteria doesn't matter. And the candidate
manages to beat you by system of work which defeated your original means of
judging him.

I think this is what gaming the system means.

~~~
gordonguthrie
As I read the article and the comments I thought "how would I reverse a linked
list?"

Turns out it is pretty straight-forward in Erlang: lists:reverse(List).

Reading their contributions on github is a much better way to get a sense of
someones capabilities.

~~~
eru
You could just ask: How is lists:reverse(List) implemented in Erlang, or if
you don't know it, how could it be implemented? What are the trade-off and
limitations for different implementations?

Since most abstractions are leaky, you can make a good argument, that learning
how libraries tick, even if you are never going to rewrite them, is a useful
skill.

~~~
gordonguthrie
But I want to find developers who will help me deliver the business
objectives. Rewriting list reversal code doesn't do that.

~~~
eru
Understanding the limitations of libraries might help you use those libraries
better. Using libraries can be a part of delivering business objectives.

I don't think that list reversal is a good interview question. At least not as
a positive discriminator. Though it might help you weed out people: I'd
probably not want to hire people who can't figure out how to reverse a list.
If they'll stumble upon such a simple problem, they might also stumble over
the more complicated problems we have to solve to deliver business objectives.

I wrote `can't figure out' deliberately. Not knowing how to reverse a linked
list, but being able to come up with an algorithm after a few minutes thought
is perfectly adequate.

~~~
gordonguthrie
I agree with you - it is a weeder-outer, but not an includer-inner - there are
lots of people who can write code to reverse linked-lists that I don't want to
hire.

In particular I don't want to hire the people who would rather write their own
implementation than use a library (think Not-Invented-Here and Yak Shaving).

~~~
eru
Indeed. By the way, in purely functional implementations of a list reversal
there are non-trivial trade-offs to make. (See Janestreet's standard libraries
for OCaml.)

------
tptacek
Programmer interviews are famously terrible. Take a second to think about
FizzBuzz, why anyone ever thought we'd need something like it, and how it
actually continues to serve a purpose.

Everyone thinks they're better at interviewing than they are. They often have
a survivorship bias. In clueful companies, it's not that dead weight doesn't
get hired. It just gets washed out. Interviewers tend to remember the people
they end up working alongside.

There are dev questions that are harder to study for. They tend to be open
ended. A few of my favorites:

* Describe some code that tends to follow you from project to project (I mostly interview C devs, so I tend to ask this as, "what's in libyou.a?"). What's in your bag of utility functions?

* What's the worst library you've ever worked with and why did you hate it?

* Estimate the amount of time it will take you to complete system X. Drill down, both by forcing the candidate to specify components (do an informal design exercise) and then _cost each of those components_. Estimation is an extremely important skill regardless.

* What's a piece of functionality that tends to crop up on lots of projects that you'd never implement yourself if you had the option of using a library? (Usually, I'm trying to get C programmers to tell me that they would not in fact hand-hack doubly linked lists out of structs with nested struct pointers).

* Describe the last system you contributed significantly to; how did those contributions break down structurally (did you write libraries for X, Y, Z; did you write plugins for X, Y, Z; did you extend the engine in X, Y, Z ways). Now describe the trickiest bug you ran into on that project.

For C devs, I always used to ask, "You're testing a component you just wrote
and finding that it crashes in malloc. Diagnose the problem." This is less
relevant now, but there are probably surgical questions you can ask e.g. a
Python dev that verify that they actually have the experience of working
professionally in the language.

~~~
zmj
I'm not sure I understood you correctly about FizzBuzz, but if you're saying
that it isn't necessary then I'd have to disagree. I wish the world was such
that FizzBuzz wasn't an effective filter, but it is.

I really like the rest of your questions.

~~~
patio11
Thomas is saying that the hiring process for programmers is so
_comprehensively_ fubared that asking the FizzBuzz question to someone who has
worked as a senior developer for 8 years at a well-regarded firm _provides
valuable signal_ and that if you fail to ask it you might _hire people who are
completely incapable of programming_ even with your fancy-dancy recruiting
process, which shows that the pipeline and standard interview process are both
borked like crazy.

~~~
chc
Essentially: It is not common for doctors to be asked to locate the
interviewer's mouth, ears and thumbs. If an interviewer asked that kind of
thing, you'd really have to suspect that he couldn't tell a good doctor from a
bad one.

But programming interviewers do need to ask this kind of thing. So you really
have to suspect…

~~~
potatolicious
The problem isn't that we're asking doctors to locate their mouth, ears, and
thumbs.

The problem is that the resume and candidate pipeline is so crappy, that the
interviewer _cannot be sure_ if the person sitting across the table is a
doctor, or a plumber, or a carpenter, or an accountant, or maybe a bum off the
street.

My interviewing life would be a lot easier if there was a firm guarantee that
every candidate knows at least basic programming - and by basic I mean _basic_
\- i.e., can put together a for loop that compiles, in a language of their
choice.

The second part of this problem is that the industry has so many jobs where
the _absolutely clueless_ can survive that "X years experience" in-industry is
not a trustworthy metric for competence. "5 years at County General" for a MD,
with a clean record, is a pretty decent guarantee that your candidate is in
the ballpark. "5 years at Accenture" for a programmer guarantees nothing, not
even the ability to write FizzBuzz.

Elsewhere in the thread someone mentioned being incensed that, even with his
years of experience, he was being asked elementary questions. That's why -
years of working experience is _not_ a valid signal for competence.

~~~
Cass
"5 years at County General" for a MD, with a clean record, is a pretty decent
guarantee that your candidate is in the ballpark."

Scary but true: There are plenty of experienced doctors who couldn't pass a
medical Fizzbuzz. My supervisor at my last teaching hospital, who has been
treating patients for at least five years, failed the Fizzbuzz question I
asked him. ("What's that other class of antibiotics you can't give patients
with penicillin allergies?", if you're wondering.)

(There's also a few stories of people faking medical licenses and working as
doctors for years before anyone noticed [1], but those are probably too rare
to worry about.)

1: [http://www.sueddeutsche.de/panorama/hamburg-falsche-
kinderae...](http://www.sueddeutsche.de/panorama/hamburg-falsche-
kinderaerztin-verurteilt-1.499146), couldn't find an English language link,
sorry.

------
troygoode
People talking about how he "hacked the interview process" are missing the
point: as a hiring manager I'd gladly hire this guy even knowing that so much
of what he did was not entire genuine. He took a problem (I need a job),
implemented a solution (applied and interviewed at a TON of places in that
short of a timeframe), and successfully iterated on that solution until he had
completely _mastered_ it! In a very short time period he went from being
unable to solve a variety of programming problems (or doing so poorly) to
internalizing the how & why of the optimal solutions - this is what good
software development is about!

I care a lot more about initiative and problem-solving skills than I do
reversing linked-lists - this guy has the former two in spades.

~~~
overgryphon
Sure, if you're looking for someone who is hard working and willing to learn,
then this type of candidate is perfect for you.

But if you're screening for candidates with development experience who you can
rely on for technical knowledge, the idea that someone can game your interview
process in less than two weeks without previous knowledge is horrifying. But
I'm guessing that people looking to hire experienced candidates aren't basing
hiring decisions on questions about reversing linked-lists.

~~~
kamaal
I think the point he was trying to make was. If you are productive enough and
serious with a little skills you are can be hired.

The reason being you work hard your way towards success. This is far better
than the person who knows merely facts.

Knowing doesn't always doing. But in most cases if you are doing something you
are likely to know what it is.

------
observer1
"....this process, starting from ~50+ raw leads whittled down into ~40 initial
phone interviews/coding challenges, which went to ~25 next stage interviews,
eventually down to ~6 really good offers, all in about ~1.5 weeks."

Does this timeframe seem a little dubious to anyone else? I count 65 technical
interviews plus interviews with hiring managers, etc., followed by receiving
offers. And all of this happening in about 8 days. This story would be a lot
more believable if it had a realistic timeframe.

~~~
sgrove
Nope, it's true. I woke up early in the morning (easy to do sleeping on the
kitchen floor in mountain view without a heater), went through craigslist
every morning, had an email template I'd individualize for each posting, and
send it out. I'd work through the code challenges as they came in, set phone
interviews into the late night, in-person interviews in the city during lunch,
etc.

It happened, I took it pretty seriously because of the impending cliff of
being homeless.

------
steve8918
I call complete and utter bullshit.

Each "technical interview" I've been to is about 3-6 hrs. I've yet to see a
Silicon Valley technical interview where it's not at least 3 hrs, meeting with
3 people. So right there, fitting 25 "next stage" interviews in 1.5 weeks is
bullshit.

Where is he going to find the time for 40 initial phone interviews? That
requires talking with the recruiter, and then the recruiter scheduling time
with a developer. Anyone who has realistically interviewed in Silicon Valley
knows that recruiters are very slow, and things tend to get muddled up very
easily.

In 1.5 weeks, assuming they don't interview on weekends, that's 6 1-hr phone
screens, plus 3 on-site interviews per day. The logistics simply don't work.
It's a complete lie. If he had said 1 month, then it still would have been
impossible with 1.5 phone screens and 1 on-sites per day, but it would have
been slightly more believable than 1.5 weeks.

~~~
sgrove
Rough. But you're free not believe, I don't have a problem with that.

~~~
steve8918
Sorry buddy, but please explain the complete discrepancy in what you reported.

How do you fit 3 on-sites and 6 interviews per day? Even if your onsites were
1 hr long and they offered you a job after a single interview, you still have
to drive back and forth to each location. How did you have time for the 6
phone screens per day?

~~~
sgrove
I didn't have a car actually. I just scheduled all the interviews as close to
possible every day, and walked around SOMA/FiDi from morning until evening
going through different stages of different interviews - initial chat (morning
coffee/lunch/dinner/drinks), whiteboarding session, meet with the team, build
a mini-project, discuss final offer.

Each successive stage is more time intensive, but there were also fewer of
them. There's a lot of noise, especially in the beginning, so a lot drop off
very quickly. From there, it was just a matter of tightly packing the schedule
and doing it from day to night.

Like I said, the "you'll be homeless again very soon" factor was a big one for
me.

~~~
steve8918
Even more interesting then, because you just lost 2 hrs every day commuting
back and forth on Caltrain.

"Each successive stage is more time intensive, but there were also fewer of
them."

You just said you had 25 onsites. Unless you knew beforehand that you were
going to cut out of them early, there's no way you could have scheduled 3 per
day. And at 6 phone screens per day, the last 2-3 days worth of phone screens
occurring near the end of the 1.5 week period would not have been able to
produce an onsite within 1 or 2 days, therefore, that means most of the 40
phone interviews must have been front loaded in those 1.5 weeks, with the
onsites being back-end loaded. Which means that your density of onsites would
have been higher than 3 per day.

It's a great story, but you should use more realistic numbers next time.

~~~
sgrove
All good points. This is just what I remember, and I don't dispute it could be
wrong.

------
rauljara
I was flabbergasted that someone was asked implement a linked list in Ruby. My
initial reaction was 'That is an incredibly stupid thing to do.' My slightly
more in depth reaction was "Isn't that what mutable arrays are for?" And my
final reaction was, "Maybe I'm missing something." At which point I tried to
figure out the benefits of a linked list over a ruby array. I wasn't able to
come up with a good reason to use a linked list in Ruby, but maybe someone
could enlighten me.

~~~
brown9-2
I think you are looking at this from the wrong angle. Just because it made a
good interview question (for the interviewer) doesn't mean it's something
you'd actually want to use in everyday practice.

~~~
kamaal
Why would you ask questions to a candidate which have nothing to do with his
job?

Will you ask a marathon runner to prove his worth by asking him how quickly he
can sprint?

------
Aqua_Geek
Are these kinds of interview questions really that prevalent? From the hiring
side of the table, I can understand wanting to quickly know whether somebody
knows some of the "basics" like data structures and whatnot. But I can't help
feeling that you're needlessly limiting your applicant pool to only those who
got a CS degree (and paid attention during the process).

Why is so much emphasis placed on the theory of how linked lists work (for
example) instead of the practical application of where they'd be used?

Edit: Big O notation is great and all, but shouldn't the emphasis be more on
shipping than optimizing? You can optimize your MVP to death, but if you never
finish it what good does that do?

~~~
ejames
Yes, interview questions like this are common.

The problem with "practical application" questions is that the answer almost
always comes down to "it depends". The correct optimization or fix is specific
to the actual thing you are doing; that's what makes it practical.

In order for an interview question to be useful, both the interviewer and the
interviewee must be able to answer it. The applicant doesn't know the
interviewer's product well enough to make accurate technical diagnoses. The
person doing the interviewing doesn't know about the applicants' products
either. Bogging the interview down in specifics to fill in the background is
usually not helpful - plus, it gives lots of opportunities for the applicant
to bullshit the interview by listing a bunch of technical details that might
not actually have been accurate or important.

Theory questions are favored precisely because they are not dependent on nuts-
and-bolts practical details. "Reverse a linked list, writing your code on this
whiteboard" is general enough that most applicants can answer.

I understand that people who lack CS degrees might not know the answer.
Interviewing means accepting that you don't get a 100% accurate result. I
would only switch away from the "basic CS question" problems if I was
convinced that some other question has a higher accuracy rating, and I'm not
sure of that.

~~~
Aqua_Geek
> The problem with "practical application" questions is that the answer almost
> always comes down to "it depends."

Agreed. Talking about application of theory is way too vague and leaves way
too much room for bs. But I think my original point was unclear. I'm not
suggesting that interview questions be "When would you use a linked list?" but
rather "Let's hack on something using a linked list (something beyond just
simply reversing it)." Like the article mentions, once you've built a linked
list before and have optimized (and reversed) it to death, there's little to
be gained from doing it again for somebody else.

------
brown9-2
_I started asking if I could assume TCO - usually knowing they would say no,
but it would set them back a bit because that's an unusual question._

What does TCO here mean? Tail-call optimization?

~~~
anonymoushn
Yes. For a moment I wondered why he wanted to use TopCoder Open in the middle
of an interview.

~~~
r00fus
or Total Cost of Ownership... I guess I'm hopelessly "enterprisey".

~~~
AdamTReineke
That's two of us.

------
russell
I use a similar approach. I spent 20 years as a consultant, so I did a lot of
interviews. I would send out lots of emails. I didnt do many tailored emails,
but I did a carefully tailored cover letter designed to be amusing to the HR
ladies. You have to get by them first.

I figured the first few interviews were throw-aways, just to get my
interviewing skills back up to speed and to find what this years interview
questions were. Yes, they go in fads. One year the programming task of choice
was to reverse the words in a string. See it a couple of times and you get too
be pretty proficient.

I keep a stock of questions to push back. I havent issued programming
challenges, but I have asked, "What is your pain point? What technical issue
is causing you problems?" Usually we get into a long discussion of the
challenges theyare facing.

~~~
shard
What did you do in your cover letter to amuse the HR ladies? Please share.

~~~
russell
Do you remember the Johnny Cash song "I've Been Everywhere... Chicago, Reno,
Fargo"? Sometimes I think my resume reads like that.

After I added those lines, I got a 25-30% increase in responses.

------
breadbox
So ... what exactly is the turing-machine dog question? A google search left
me unenlightened.

~~~
andrewflnr
I figured it was just humorous nonsense.

------
krupan
A lot of discussion here about memorizing the programming questions, but I
think the most interesting part of this is when he turned the tables at the Do
You Have Any Questions For Me stage and really interviewed them.

I haven't asked anyone a programming questions like he did, but I did ask some
tough questions about the team, work environment, management style, company
goals (looking for a quick exit? in for the long haul? etc.) at my last job
interview (after some good friends gave me the advice) and it both gave me a
lot of insight into the company and team that I was considering working for,
and I think it impressed them that I was thinking about those things.

In short, know what you want from a team, manager, and company, and take time
to interview the people interviewing you to make sure they have what you are
looking for.

------
nadam
I don't understand why is there such a strong emphasis on these kind of
interview questions. I think showing a piece of nontrivial code which the
candidate have created in the past tells a lot more about the candidate.

For example this tells more about me than the interview questions I usually
got: <http://codeclamp.com/ccui.js> , because it tells about how I think long-
term, how I solve relatively hard and relatively complex problems, not how I
survive a quick interview (solving relatively easy algorithmical problems but
under very high time pressure.). (By the way, if you have an interesting well
paying job for me, I might be interested...)

------
langsamer
I think a much better indicator of how valuable a person will be in your
company is not if he can solve puzzles on the fly but how munch time he puts
into coding. If a person is coding in his free time and keeping up with
technology and experimenting even though his current job doesn't require it,
that person is more likely to have a breadth of knowledge that will pay
dividends over time. Something like that cannot be gauged by him/her solving a
"reverse-linked-list" problem.

Furthermore, a person involved in the community is more likely have in-roads
with other strong developers, which again might pay dividends in the future in
terms of hiring.

~~~
gcp
In my experience interviewing candidates, the two are very strongly
correlated. A person coding in a lot, including in his free time is more
likely to have implemented a wide variety of things, and will be much more
hands-on to solve these kind of riddles/exercises, than a person with only
book knowledge who coded the same kind of stuff for years on an end.

------
codenerdz
I had a similar experience interviewing after being off the job market for a
quiet a few years: You will be rusty initially, but then you will pick up
steam.

Just like in your "real life", solving interview problems is not about
reinventing the wheel with 100% custom solution, its about pattern recognition
where the pattern is a problem you know a solution for. The more problems you
solve(and possibly fail) the more data your internal pattern recognition
algorithm(your brain) has to help you with the solution.

I dont see it as gaming the system, I see it as being able to prove that you
are capable of problem solving.

~~~
sgrove
Yeah, it's a positive signal that someone was able to pick up from past failed
interviews. It's a weak filter, but it's fine to have. From there, it's best
to work with the person on short projects, and then hire them as a contractor
to find out if they're a culture match and can solve the problems you're
facing at an acceptable rate and in a maintainable way before hiring them on
as an employee.

------
seank
It's worth mentioning this as a counterpoint to 'startups don't reply':
<http://news.ycombinator.com/item?id=3351699>

------
eru
I can't post to the site, could somebody who can (or can contact the author in
some other way), leave a note to the author, notifying him of a typo in "For
people hiring, remember that anyway can pass yours tests with enough
experience interviewing, even someone as bad at coding as myself." (It should
be `anyone' not `anyway'.) Thanks!

~~~
sgrove
Got it, thank you eru!

------
pnathan
This article really indicates to me the fatuousness of this sorts of
problem/solution interview approach.

I agree with timr in that a more more intelligent approach is to throw a
problem at the candidate, then drill down into their thinking about the
problem.

It's a bit silly if you're hiring based on regurgitation of algorithms in my
opinion.

~~~
tikhonj
I certainly agree. However, I haven't actually seen any pure problem/solution
interviews--and this is just for internships where you wouldn't expect the
companies to be as thorough. I had questions ranging from "find and fix the
problems in this code" to "how would you design a library to do..." to
"propose a solution to this problem and prove it works". This might be because
I mostly talked to smaller companies.

------
tldrtldr
Isn't there a book dedicated to coding interviews?

[http://www.amazon.com/Cracking-Coding-Interview-
Programming-...](http://www.amazon.com/Cracking-Coding-Interview-Programming-
Questions/dp/098478280X)

------
systemizer
I had a similar experience. This past Fall, I was interviewing with multiple
companies, and I received questions that were very similar.... some being the
same exact thing. I was nice, however, and told them out front that I had seen
the problem before and asked for another.

I don't regret doing that; some may have passed the interview by keeping quiet
even though they had seen the problem, but those people won't go far in the
world because they cheat. They will eventually get called out for it and lose
their reputation.

