
Whiteboard Interviews - pw
https://www.tbray.org/ongoing/When/201x/2017/03/04/Whiteboard-Interviews
======
tptacek
Interviews don't demonstrate ability or aptitude so much as they demonstrate
ability to keep a cool head under incredible pressure. It doesn't matter how
soft you make the whiteboard question: you are getting lower quality signal in
an in-person office interview than you will in any other venue. Interviews are
incredibly hostile experiences and nothing you can do, including comfy chairs,
bright, cheerful surroundings, or free drinks will change that.

We need to engage with the fact that these interview processes are just
stupid. We know a number of better ways to extract signal from candidates, but
we keep interviewing face-to-face for technical ability because that's how
it's always been done.

A number of companies are trying to embrace work-sample challenges to boost
signal from candidates. But I don't think they get it. They assign take-home
work before or after an interview, and still make decisions on candidates
based on an "interview loop". If you have an interview loop composed solely of
people, I don't believe in your process. We had a sort of interview loop at
Matasano, but the open secret was it largely didn't matter, because if your
(numeric) scores on our work sample tests were solid, _the burden of proof was
on the interviewers to override the work sample tests_.

We worked with a number of different hiring groups at Starfighter last year,
some of them quite smart, and at none of them did I see a process that
deliberately and meaningfully insulated itself from "I'm having a bad day and
haven't eaten lunch yet and now I have to interview someone" bias, or from
"I'm putting you, the candidate, in the most uncomfortable imaginable
professional circumstances and expecting not only perfect recall of the
intricate details of your craft but also that you recite those details in a
way that demonstrates _confidence_ and _ability to get things done_ ".

All the hiring I see is still done by by subjective X-factors; worse, each
interviewer in a "loop" has a different X-factor. Where's the table-flip emoji
when I need it?

~~~
angersock
So, to hit it from the other side: maybe none of that matters.

For an industry practice that everbody is saying is so badly broken, people
sure seem to be making it work--or sidestepping the braindamage.

For CTFs like Starfighter, or for any take-home work product exam that is
objective ("Pass the tests, meet the performance requirements, etc."), the
emphasis is placed on the deliverable and not on the candidate, right?

Which in turn suggests a question--is this perhaps a signal that we don't need
to be hiring employees at all if we can just spec out the work correctly and
the associated acceptance criteria?

~~~
calcifer
> For CTFs like Starfighter

CTF?

~~~
simplehuman
[https://en.wikipedia.org/wiki/Capture_the_flag](https://en.wikipedia.org/wiki/Capture_the_flag)

------
news_to_me
This article is a nice reminder of the truism, "don't do things in a shitty
way and they won't be shitty." That is, whiteboard interviews aren't
inherently terrible, it's just that a lot of interviewers ask terrible
whiteboard-based questions, like asking to code in unrealistic circumstances.

But I think that also kind of highlights what DHH and others are trying to get
at - a lot of programming interviews _are_ terrible because they abuse the
whiteboard, and that needs to stop.

~~~
wry_discontent
The problem is that you have to have some way to evaluate someone's skill
level relatively quickly. Take home assignments will never work. Part time
work, or contracting work until you're satisfied is going to be even worse
from the perspective of a job searcher. Imagine you get a new job that pays
half of what your last job did and then you get dismissed at the end of 3
months and have to restart your search.

I don't understand all the fuss. Whiteboard interviews are unpleasant, sure,
but I've done a lot of them and they're not that bad.

~~~
SilasX
Most professions have solved it: some really tough, relatively objective,
centrally-administered exam that's a strong sign of knowing some minimal
competency. Then, point to the authenticated results instead of going through
that at each employer.

~~~
dudul
Isn't it what a college degree is supposed to represent?

I'm not saying they _are_ what you are talking about, but that they are meant
to be such objective exam.

~~~
SilasX
Not really. They're correlated but colleges do not coordinate on the exact
same tests for the final exams, nor even what specific classes you're expected
to have. (In some ways, that's a good thing -- colleges should have variety on
what specifically they teach.) So it's not objective, and it may not meet
security standards -- one university may do the honor system and not bother to
actively stop cheating, while others may not trust that as a mechanism.

Usually, professions have some test that you need to pass in addition to
getting the degree. You have to pass the Bar exam after getting a law degree,
the EIT exam after getting an engineering degree (then FE after 4 years under
a PE), you have board exams after completing med or pharmacy school , there
are actuary/accountancy exams after those degrees, etc.

They serve a more orthogonal role to the degree, in proving mastery of a more
specific body of knowledge.

------
throwaway_374
>>"Back when I was at Google, I was most­ly in­ter­view­ing peo­ple for
“Developer Advocate” po­si­tion­s, and a lot of peo­ple some­how got in­to the
pro­cess with­out be­ing able to code at al­l. So, ear­ly on, I’d ask. “You’ve
got a list of ob­ject­s, write some code to se­lect one of them at ran­dom.
Any lan­guage, don’t wor­ry about syn­tax, as­sume the built-in ran­dom
func­tion is good enough.”

That was ac­tu­al­ly a nice ques­tion: If you want­ed to dive a lit­tle
deep­er, you could ask the can­di­date to sketch in unit test­s. And if you’re
talk­ing to some­body super-technical, ask “Your code is in pro­duc­tion and
some­times it’s throw­ing illegal-index ex­cep­tions un­der heavy load. What’s
go­ing on and how do you fix it?” Just be­cause that’s a cool prob­lem, very
real-life, and most peo­ple smile when they get it."

Genuinely curious as to what the reasons for illegal-index exceptions under
heavy load would be? I genuinely can't think of any reason. Perhaps the
problem is under specified. Unit-testing would need to be seeded to be
reproducible.

~~~
fnbr
I could see it being some sort of shared memory problem, where the system is
misallocating memory because it's running out of free space.

~~~
joshuamorton
Or similarly, the application is multithreaded and the list length at `idx =
randint(list.length)` time is greater than the next line `obj = list[idx]`,
when you pick one of the last few indices and objects are removed.

------
charles-salvia
I've been doing interviews pretty frequently lately. Maybe it's just the
limited scope of my experience, but I've never asked anyone to code bubble-
sort, nor do I know anyone who has ever asked that, nor have I ever been asked
that at a job interview. The closest thing I've seen to this, probably, is
generic tree-traversal questions, like find the lowest common ancestor of two
nodes, or whatever. Apparently the feeling these days is that these sort of
questions suck because they're not "real world" enough, and you'll never need
to know about such stupid things like "nodes" and "algorithms" and "computer
science" on the job.

So now I guess the trend is to ask more trendy, design-oriented questions
like, "how would you design Facebook?", etc. But the problem with all this
backlash to algorithm-heavy whiteboard interviews is that it ignores the
reality that there are tons and tons and _tons_ of crappy candidates out
there. I'd prefer a candidate who understands pointer arithmetic and can
implement a recursive descent parser over someone who vaguely describes how
Twitter might work at a high level. It's not that I am claiming that most on-
the-job programming is implementing algorithms and data structures (it's
obviously not), but being able to do these things indicates a certain
quantifiable level of aptitude.

The trick is to present the questions in a way that isn't vulnerable to rote
memorization. Don't ask someone to simply code a tree or graph traversal
algorithm. Instead ask them about the most efficient way to route packets or
whatever. And more importantly, tailor the interview questions to the actual
experience they claim to have on their resume. If they claim to be a machine
learning expert, ask them to sketch out how a multilayer-perceptron actually
works. If they claim to know about the Linux kernel, ask them to design a
page-cache, etc.

I agree the "you have 5 minutes to implement quick-sort now!!" is not helpful.
I also hate the stupid logic-puzzle crap. But let's not completely dumb-down
the whole interview process as a remedy to all this, just because "real-world"
programming is more about import java.util.* rather than coding up custom
radix tries.

~~~
Apocryphon
> the reality that there are tons and tons and tons of crappy candidates out
> there.

Maybe we need to start asking why that's the state of the industry right now.
How did it come to this where quantity is so high and quality is so low? And
how can these candidates better themselves?

~~~
watwut
Because it became cool and well paid and thus many guys decided to give it a
shot, fake it till they make it. Some even succeed. Also, because many peoples
confidence is much higher then skill. They get away with it because confidence
tend to be confused with ability.

------
agentgt
I absolutely hate whiteboard interviews.

I have been lucky enough to have had only one whiteboard interview in my
career and I will say it was awful. Luckily I didn't really need the job as I
had freelance on the side. IIRC the company was TripAdvisor... I will not name
the interviewer's name but he does have a blog... sadly I think he is actually
proud of his interviewing and management techniques).

I have absolutely terrible handwriting and even worse ability in penmanship
layout (basically what I call consistent and appropriate spacing).

Before the interview the interviewer didn't even shake my hand and barely made
eye contact with me. He looked pissed before we even started (its ironic again
because the guy praises his people skills in his blog). In the interview I was
asked to do a binary tree sort. I did it recursively. He wanted it to be not
recursive and with an array.

At this point I was pretty nervous and my whiteboard looked like a 5 year olds
scribbling. I just wanted the interview to end so I sort of just gave up.

A couple months later I ironically started a successful recruiting software
company. Even more ironic the interviewer later tried to start his own startup
but failed and had to go back to TripAdvisor (this was about 3 or 4 years
ago).

I tried to connect on LinkedIn with him to give him some feedback... alas he
did not connect.

The reason I wanted to leave him feedback (besides the sick inkling to rub in
my success) is I have learned so much from my recruiting software company. I
see people desperately applying for jobs everyday (we are career portals as a
service. We try to make the apply process as painless as possible).

People seem to forget how uncomfortable it is to go looking for a job
(probably why he went back to his old company).

------
mcbits
Being able to communicate via whiteboard is a useful skill. I'm not good at
it, but I appreciate those who are. That seems to be what this author is
advocating: jotting down ideas, designs, diagrams, temporary notes. If the
team depends on everyone being comfortable at a whiteboard, then it's worth
testing at an interview.

I don't think that is what most people are pushing against, but rather actual
whiteboard coding. Taking the whiteboard out of the picture, the task is to
write code in an unrealistically broken editor. You get a partially
functioning backspace but no insert, indent/outdent, search/replace, etc. And
the editor restricts your typing speed to 10% of your normal capacity. On top
of that, strangers are staring at you.

If whiteboard _coding_ really leads to better hires than other forms of
evaluation, I'd like to see the data.

------
OhHeyItsE
I don't think anyone has any issue with having to physically use a marker on a
whiteboard during an interview. What else would you use for a system design
question?

What is insane is expecting a candidate to be able to navigate a million state
modification and off-by-one gotchas using only a marker and their short-term
memory.

~~~
kevinr
> I don't think anyone has any issue with having to physically use a marker on
> a whiteboard during an interview.

I have friends whose RSI is sufficiently bad that it could be an issue.

------
amalfra
Seriously people think whiteboard interviews are reason for diversity? I think
the much bigger reason for diversity issue is that companies are reluctant to
even interview if you don't have a brand name associated with your profile(a
tier 1 college degree or previous exp at top tech firms). There are lots of
job postings these days which state tier 1 college degree as requirement. I am
from India and here the situation is if you are not from a tier 1 college you
are a moron and designated to do low quality cheap work. It may not be the
case in other areas like the valley, but here it seems the filter for
participating in the interview has more to do with diversity than how
interview is conducted.

------
tbirrell
I'm glad he does it right. Now if the people interviewing me could do it
right, that would be great.

------
MorePowerToYou
I think the optimal solution to the hiring problem requires trying out the
actual relationship. It's similar to dating. Predicting the long-term success
of a relationship based on a first date is sub-optimal.

A service that provides this "trial relationship" could work. When I'm looking
for a new job, I could do one hour per day of real work for a few companies.
After five days, both employer and prospective employee will have a better
understanding of each other than a single day of interviews allows.

There's practical concerns. Companies certainly wouldn't give prospective
employees full access to internal codebases or data. Companies would have to
break off small tasks which could devolve into throw-away take-home
assignments. Employees at the company would have to endure the annoyance of
dealing with prospective hires. It might be worth it for everyone.

~~~
w0rd-driven
This is sadly rife for abuse. Why hire anyone? You can just string enough
candidates along to possibly not work a full solution but it could be a first
line or give you a fair number of prototypes. Granted I don't know that this
could ever be viable but companies would sure try because free work > paying
any amount. I think most of us understand the phrase 'you get what you pay
for' and yet I've personally seen many individuals and companies believe
they're getting a steal.

I feel like there are a few good ideas. I personally love the idea of work
related assignments but only if the result is actually measured. If I don't
get the job I know I've wasted my time and I can handle that. What I can't
handle is giving me a task and completely ignoring the results. What will that
signal to me about day to day operations? That you'll likely ignore the very
real value I bring to your organization on a consistent basis. That's a big
enough red flag for me to stop everything and walk right out the door. If I'm
supposed to extend the courtesy of not wasting the interviewers time I expect
at least some attempt at that same courtesy.

------
m3kw9
The issue is the dependence of code completion, google, and IDEs that writes
boiler plate code for you, and companies just use stupid services like
hackerrank to test algorithm implementations, which has to run and compile
properly u see that environment.

------
nunez
Two weeks ago, a coworker and I were thinking through ways of implementing an
extension to a library we've been writing for a client. We were drawing all
sorts of things to come to a consensus of what to do. Through doing all of
that, I finally experienced a real-life reason to whiteboard code, since after
writing down psuedocode on the whiteboard explaining the implementation that
we were agreeing upon, everyone on our team "got it" immediately.

I think whiteboard interviews are incredibly useful. We use them in our
interviews, and they've been successful in helping us learn more about our
candidates' abilities and for them to show us what they know.

------
orless
In my previous company we used ask candidates to code a very simple task:
reverse a string in Java, on the whiteboard or on paper. It is a trivial task
but many seasoned people failed it to the extent that we thought "you don't
really know Java, do you?".

To this day I'm very suspicious when I don't get a coding task on an
interview. That's generally a pretty bad sign. Best option is when I get a
notebook with "my" IDE, but I also coded on whiteboards, paper, Google Docs.

Frankly, I don't know what other options are there. There should be code in
the developer interview. Take-home task? GitHub account? Probation day?

~~~
projektir
> It is a trivial task but many seasoned people failed it to the extent that
> we thought "you don't really know Java, do you?".

This is the kind of bias that I think an interviewer should never have, and
part of what makes interviews so stressful: that your ability is called into
question off of something very simple. You don't know whether the candidate
knows Java or not. You only know that the result of the interview is negative,
which could say just as much about the interview as it does about the
candidate. Assuming "you don't really know Java, do you?" with so little
evidence strikes me as disrespectful at best. Why is this considered OK?

ashark makes a lot of very good points as to why someone may know Java but
stumble with this question.

~~~
nissimk
Exactly. So many of the proponents of the whiteboard algorithm question
industry are thoroughly convinced that they are eliminating people who will
fail on the job. Of course it's impossible to verify this. The only thing you
can verify is the performance of candidates that do pass your test and you
subsequently hire. I'm sure there are a lot of bad hires even at the companies
where they use these interview techniques. The believers are clouded by
survivorship bias and confirmation bias. They get no information from those
that they reject and they refuse to acknowledge the bad people that they
accept.

Didn't the guy who started WhatsApp fail his facebook interview? :-)

[http://www.businessinsider.com/facebook-rejected-whatsapp-
co...](http://www.businessinsider.com/facebook-rejected-whatsapp-co-founder-
brian-acton-for-a-job-back-in-2009-2014-2)

Here is some debunking of other commonly held misconceptions about this
interview strategy:

1) The questions are not tricks -- actually every coding algorithm question
that has a "naive" solution and an optimized solution requires a trick. That's
why you can study for these interviews. The studying involves memorizing the
catalog of tricks and identifying which one to apply based on the question.
Even the simplest and most commonly used tricks (using a hash table, multiple
pointers into the same array / multiple accumulators, bit shifting) are still
tricks.

2) The interviewers have a large catalog of these questions so you're not
going to pass by memorizing stuff. As I mentioned in #1 above, this is false.
The list of tricks that need to be employed to solve these puzzles is finite
and definitely less than 100. Most common interviews will probably be solved
by one of the top 10 tricks.

This page has 80 solutions: [http://www.geeksforgeeks.org/top-10-algorithms-
in-interview-...](http://www.geeksforgeeks.org/top-10-algorithms-in-interview-
questions/)

3) This method may have too many false negatives (when they reject a candidate
who would have been a good member of the team) but it eliminates false
positives (when they hire a candidate who cannot code). With enough dedication
and time, anybody can train to pass these. It's not really a skill that is
super useful for getting things done in a programming job.

Some people are very good at riddles but have no common sense. They do not
make good members of a software development team but they are very good at
these interviews.

~~~
orless
Tricks? Algorithms? Puzzles? Riddles?

Are we still talking about string reversal task?

------
joeax
Maybe I'm out there as far as this topic is concerned, but as a software
engineer I enjoy whiteboard interviews. If you can't get up in front of a
couple potential colleagues and talk/draw code, how are you supposed to
whiteboard complex designs in meetings after you are hired, possibly to
higher-ups?

But I agree there should be set parameters in an interview: ask every
candidate the same open-ended design questions, stress that they just use
pseudocode, keep the problem questions small (5-10 mins max), don't be a
overlording interviewing jerk, etc.

------
djhworld
> _By the way I got one of those in my in­ter­view day at Google, and
> an­oth­er at Ama­zon, and I blew them both._

Interesting to hear this, can't have done that badly as he got jobs at both of
those companies!

------
the_arun
Whiteboard really helps one to express his/her thought process to the other
person and I'd hope it will continue to exist in the interviews.

~~~
m3kw9
Then the interviewer should explicitly say he wants to know your thought
process too. 99% of them just say code this and let me know when you are done

