
Don't Leak Interview Questions - igor47
http://igor.moomers.org/interview-questions/
======
yapcguy
Cry me a river AirBnb.

Why not just give the candidate a programming problem, a real computer (maybe
no internet, but all tools and documentation installed), and an hour or two to
solve the problem?

You can monitor their computer screen remotely, record it even for later
review, and also sit in the room to watch them at work and offer guidance if
necessary.

If the candidate passes these real world challenges, then you can do the soft
social stuff and see if they will fit into the team.

~~~
lmg643
totally agree. actual problem solving is more informative than questioning.

but even for questions regarding prior experience - most of the "standard" job
interview questions are widely known. the question being widely known doesn't
make them less effective - it is up to the skill of the interviewer to ask
relevant follow ups based on what the candidate says.

often, I don't even ask standard" questions. i just look at the most audacious
claim on the resume and grill on that topic to figure out what they actually
did.

Then i jump around from claim to claim, out of sequence, so they can't fall
into the rote story of "how I got here."

I think this is a reasonably effective, harder-to-hack way to interview,
unless you're dealing with a pathological liar. (so far, I haven't personally
hit one.)

~~~
yapcguy
Interesting.

Do you do this with all hires, or just junior candidates e.g. recent
graduates, those having taken bootcamp classes, or migrated over from another
discipline e.g. "I used to be in Marketing but now I'm a growth hacker" ?

What if you have a candidate with a proven track record, perhaps lots of open
source code, perhaps well known in the technology community? Isn't there a
risk that grilling them on a topic might just antagonize them to the point of
not wanting to join the company?

~~~
lmg643
good question. if you have first hand knowledge of a candidate's
qualifications, it is going to change the interview for sure. but that doesn't
mean you don't need to figure out if there is a fit with your team/company.

i don't think focusing on specific details is antagonizing if it is done
right.

a great candidate will embrace the opportunity to talk about their experience.
it will probably be an awesome conversation for both of us. i will learn
something, and they will see their experience in a new light trying to explain
to someone else.

a weak candidate will stumble on details of their experience, or will be
forced to admit someone else did steps x, y, or z. or that the claim is
inflated.

------
reginaldjcooper
If you don't know how to pivot an interview question when the interviewee is
already familiar with it, you aren't qualified to perform interviews.

If the goal is to find how they approach problem solving, you can adapt by
asking about increasingly difficult aspects of the problem. If that isn't the
goal, your hiring practices are wrong.

------
yesimahuman
Considering how stacked the interview is against the candidate, I really have
no empathy for the interviewers in situations like this.

Throwing puzzlers at a candidate indicates you are hiring for normal
engineering talent (i.e. this person doesn't have a previous reputation that
makes them special) and so vetting the person is going to be highly imperfect
whether or not they can solve your puzzle.

------
raverbashing
I disagree. Good questions (and good interviews) don't require previous
knowledge (and if it exists, this can be worked around, or will have little
effect)

You know, it's not about a, b, c or d. It's the whole answer.

And I still don't know why interviewers insist on whiteboard coding. Get a
computer, google stuff together, try some code, because _that 's how it works
in the end!_

~~~
macspoofing
>And I still don't know why interviewers insist on whiteboard coding.

Worse. They require syntax correct whiteboard coding. I got to imagine that
pseudo-code should suffice in almost every case.

~~~
yapcguy
You could say you're using a special pre-processor to parse any incorrect
syntax before compilation. Might get a laugh and some brownie points!

------
michaelt
While I appreciate the need for calibrated questions, it seems naive to expect
a system to work if it depends on questions not to leak - especially if you're
working with recruiters, or rejecting a lot of candidates.

What would be interesting is a system that works in spite of questions
leaking. Personally I like the idea of a calibrated question pool so large
that, if someone learned the answer to every question, they would have become
qualified for the job. Of course you'd have to do a lot of recruiting to
calibrate such a large question pool.

------
bonemachine
Consider:

If your interview process can be substantially weakened by having some of your
questions leaked... then maybe it wasn't such an effective process in the
first place.

------
robterrell
I agree: I won't leak your interview questions if you'll agree to make them
fair. Deal?

To me, fair means: not asking me to write code on a whiteboard! Or, if you
insist upon it, not complaining about not-compiler-perfect syntax, assisting
me with the correct function names or argument order of library functions I'd
normally autocomplete or look up, if I've forgotten, and focusing on the
thought process and problem-solving rather than keypresses.

But really, most fair would be: don't ask me to code your stupid whiteboard
interview question. Look at the shitloads of code I have on github, or ask me
to walk through some interesting problems I had to solve recently, or ask me
for my first-blush take on an interesting problem you're trying to solve. Make
it a conversation instead of an interrogation.

Because, you know, a startup filled with people who are excellent at solving
interview questions on a whiteboard might not be the best team for solving
real-world problems. I work with a guy who can solve a rubik's cube in under
90 seconds -- that's about as relevant as some interview questions I've seen.

~~~
base698
Who does this? Seriously? I've been on probably a dozen interviews where I
would rank my performance from dismal to awesome, gotten offers. I was never,
under any circumstance, expected to write syntax perfect code on a whiteboard.
I have written plenty of stuff on whiteboards, in a few cases until my hand
was cramped, but never did anyone bitch about the kind of issues you mention
here.

A good white board interview question asks about a concrete problem like, "How
does code complete work?"

------
georgemcbay
I have a lot of the same qualms shared by many of the responses so far in
regard to how unrepresentative to real world work the typical whiteboard
coding interview is.

But even if I didn't, what's the point of asking such a large group of people
not to leak questions, especially when those most put off by your interview
style (for whatever reason) have the most motive to leak your questions?

At best you'll be ignored, laughed at a bit and people doing what you're
asking them not to do will continue doing it. At worst (and IMO the more
likely outcome) you'll end up with a variant of the Streisand effect where
even more people leak just because you asked everyone not to.

In closing, I guess I will mention how odd it seems to me that "startups"
(granted, Airbnb being a startup anymore is debatable) are all "pivot this,
disrupt this, change the system", and then complain when an interview style
that was already old and busted in 1970s isn't working out for them in this
new Internet world.

------
newobj
You know at this point, a candidate who had prepped on known questions would
be a huge plus in my book. Unbelievable how many candidates I interview that
don't even know a single thing about the product, and who are unable to code
the completely predictable problems you know will be thrown at you in a dev
interview. Did they just wake up from a drunken stupor in the lobby with a
note pinned to their chest I wonder? It boggles the mind.

~~~
saraid216
Phone screen with FizzBuzz. Admit that it's not complicated, but that you get
so many unqualified people that it _does_ serve to filter, but make them do it
anyways. It's 5 minutes at most and saves them the effort of traveling to your
office AND saves you the effort of interviewing seriously.

~~~
spinlock
I crashed and burned the first time I encountered fizzbuz. My solution was
fine but I didn't believe it could be so easy. I spent the rest of the
interview paranoid that I'd embarrassed myself and came across like a paranoid
idiot.

~~~
newobj
Did you get hired?

------
iamshs
This happens in consulting and engineering a lot. Case interviews and problems
are known among fellow students who are interviewing later, and questions get
put up on glassdoor too. You will have to iterate around the problem. This is
unavoidable aspect. Shell Recruitment Day problems are known in detail online,
their online assessment test is copied and solved in detail. Shell hires lot
of engineers, and good ones. Consulting firms hold one of most intensive and
competitive interviews. Questions will leak, issuing these moral statements
will not help, this is your problem to solve.

------
makmanalp
I think the better option is to ask questions that don't have one correct
answer? Those are the hardest ones anyway, and the most relevant to the real
world.

You explore the options beforehand, and try to figure out benefits /
drawbacks. Then when you ask the question, you see if people see those
tradeoffs too. Even better, you pose the question slightly differently each
time, such that a different option is more favored every time. This filters
out people who think they have the "right" answer.

Who knows - one day a candidate could even surprise you with an answer you
didn't think of!

------
davidrupp
Folks -- the value in this article is not in the admonition about leaking
questions (the HN title does not help). The value is that the author is spot-
on about how we (I include myself) should approach the task of conducting the
technical interview. Read the article again; leave out the last section if it
bothers you that much.

------
grey-area
_When you publicly post a question that you were asked after interviewing, you
are undoing weeks of work for your fellow engineers. In it 's place, you a
creating weeks of new work as we develop, test, and calibrate on new
questions._

There's something very broken about a process which takes weeks to prepare a
question, and something even more broken when programmers conduct interviews
up to 6 times a week (how do they have time for any real work?). That's more
than 1 interview a day...

Questions shouldn't be secret puzzles which take ages to fabricate, but ways
for the candidate to demonstrate what they know and what they can do.
Otherwise you're leaving wide open the possibility that you'll hire someone
good at gaming the system, not someone good at the job.

------
adamnemecek
Playing the world's saddest song on the world's smallest violin.

------
chuable
I have no sympathy here - interviewing is hard for both parties and isn't
going to be solved by imposing rules like this; you won't get better hires by
asking them not to prepare.

Also, (possibly off topic...) am I the only one that thought of the Kobayashi
Maru after reading this?

------
ahmedahamid
Quite honestly, and I have been on both sides here, my sympathy goes rather to
the job seeker who is trying to find a better job in one of very few
industries where you have to solve puzzles to get a job!

Not to mention tech. giants that ask inhuman questions just because everyone
wishes to work for them.

Sorry, until things are different, I'll leak every interview question I get my
hands on. Tech. companies will always be able to figure out ways to examine
candidates.

------
onedev
No. You can pretty much go fuck yourself.

I hate interviewing as it current exists and I'm going to make sure each and
every one of my friends know what I got asked so that they can better prepare
themselves and have a better experience.

~~~
pjscott
In doing that, you throw off the calibration and put people who _aren 't_ your
friends at a relative disadvantage. Is this what you want?

~~~
onedev
By torrenting and pirating music or movies you are stealing; do you really
want to be a thief?

------
liquidcool
OTOH, someone who seeks out interview coding problems and solves them might be
a good fit for your team. If they're just memorizing answers someone else came
up with I agree that's misleading, but I would also imagine they would not be
able to talk about it intelligently/insightfully in the interview.

------
tehwebguy
The negative comments in this thread seem representative of bad personal
experiences and not the article.

He's saying that he wants a chance to work through a problem with the
interviewee, not stump them. There is value to seeing what a candidate _knows_
and _has previously done_ , and I'm sure that by this point in the interview
he has determined those.

But _these_ questions are about how the candidate works, and it sounds to me
like he has put a lot of effort into making sure this is not like a crappy
puzzler, but a conversation that he will help them through.

------
ArekDymalski
This post is based on two very common and very wrong assumptions: 1. To assess
the candidate's competences you need some kind of magical, unique question
that has only one right answer that reveals the competencies. 2. The interview
is some kind of war/race between the recruiter and the candidate in which both
sides have to outsmart each other. With such mindset companies loose the
chance to recruit actual talent they need.

------
jmgtan
I do understand what AirBnb is trying to convey, but I sympathise more for the
job seeker since I had really bad experiences with these types of question.

Once I was interviewing for a senior software engineer position for a big
financial company. Before the start of the in-person interview process they
had each candidate solve a Java programming question regarding
encryption/decryption (according to their HR only 30% passes the initial
filter and my solution was one of the best with regards to design, time to
finish (I only spent 1hr to finish the exercise), runtime efficiency, and test
coverage). They gave us each a laptop and 3 hours to solve the question. I got
really good feedback after the exercise and was asked to go through the
remaining gauntlet of more Java interview questions (2 more interview sessions
w/ different people (tech lead and vp) to be exact).

At the last interview with one of the executive director, he dropped me a math
puzzle question that is not related to the job at all, suffice to say I bombed
the last interview. This was really frustrating and is a complete waste of
time on my end since their office is a bit far from where I currently work.

------
Fauntleroy
If your interview process relies on simple riddle questions, you might want to
reassess the way you interview. The most constructive interviews I've been in
involve pair programming and architectural discussion. Focus less on questions
and more on seeing if this candidate can do the things you want them to.

------
cscurmudgeon
In the list of immoral things, leaking interview questions comes much below
spamming and creating fake profiles.

~~~
pjscott
True enough, but I'm not sure I see the relevance to the point of the article?

~~~
prostoalex
He's talking about [http://davegooden.com/2011/05/how-airbnb-became-a-billion-
do...](http://davegooden.com/2011/05/how-airbnb-became-a-billion-dollar-
company/)

------
rilindo
Related by possibility off-topic: How about signing a NDA?

I am not advocating it (in fact, I probably would have significant
reservations about any company that will pull that on me), but if you are that
concerned about the questions leaking out, do something to legally oblige the
candidate not to disclose the process.

Otherwise, put up with it. Or try a different way to vet out candidates.

~~~
nawitus
I had to sign a NDA at a tech interview once, and they asked me to code
FizzBuzz.

~~~
bitbytebot
That should be a t-shirt slogan.

------
ek
I have signed NDAs in which I have agreed not to disclose interview questions
I was asked, and I think NDAs are a fair solution to this problem. The only
potential problem for an organization like Airbnb is that it can require a lot
of infrastructure and legal horsepower, which perhaps only larger companies
have the luxury of.

~~~
greenyoda
How could such an NDA be enforced? You generally wouldn't have any way of
proving who leaked the question (unless you ask every candidate a unique
question), and even if you did spend the time and expense to litigate against
the leaker, it would be hard to convince a jury that disclosing the question
would have incurred a non-trivial loss for the company.

If an interviewer asked me to sign an NDA, I'd think the company was run by a
bunch of crazy control-freaks and walk out.

~~~
prostoalex
This is somewhat standard practice. At Google/Facebook campuses you have to
agree to an NDA in order to print out your guest badge.

------
StavrosK
Off-topic: The person's thumbnail photo is a few megapixels in size and almost
50 kb :(

------
davidgerard
Always leak the interview questions. Always. Every time.

------
icecreampain
I'm European and have rarely had to be interviewed, but the few interviews
I've been on have never asked me any programming specific questions, rather
focusing on what my interests are and how I am as a person.

It sounds to me like many companies are trying to hire quiz solvers and people
who can figure out how many people can fit into a car. Do the people being
interviewed not have any downloadable code samples or coding projects that the
company could have eyed through prior to the interview?

