I remember when I was prepping for my Microsoft interview. There was so much lore online as to the types of questions they'd ask and how to prepare. I even found 'How Would You Move Mount Fuji?' in the library and started reading through it. It was an absolute waste of time as I, fortunately, learned weeks before the interview. As an example, one of the questions I was asked was 'how would you write the C function strcmp() without using any libraries?'
My gripe these days is that places I've interviewed with now that I have a few years of experience are still concerned about big-O notation and a lot of the more academic questions. I feel like I shouldn't have to go back to my college notes to prep for an interview for a job that I'd really be using my current experience for. I think it comes down to other developers not spending time really thinking about the quality of their interview questions. They just do a search for 'programming interview questions' and call it a day.
My most recent interview was with Amazon. I was asked the dumb big-O runtime stuff as well as how I'd count all of the stars in the universe. It pissed me off, but I did like the question I was asked about how I'd design an elevator system and then asked how my solution would scale. That's not too far off from what I'd be doing, albeit in an abstract form.
"What is the big O time of heap sort?" vs "Would you mind writing a function that returns true if any two numbers in the list sum to zero? For example, it would return false for [2,3,5] and true for [-3,5,3]"
Are completely different questions. The first is useless: It tests if someone remembers how a heap sort works, and whether or not they use the term "worst case" in the answer.
The second one is much better. Some could sort the list, (nlogn time, constant memory) then walk through it. They could use a hash table (idealy n time, assuming no collisions, but more memory). You also get to see if their solution handles the 0 case. If they answer it very quickly, you can move on to the three sum problem. If they recognize the question from university or something you can do a different one.
If someone doesn't know the big-O time of iterating through a list of integers once for every integer, then I seriously doubt their dedication to the field of software. This is not some pie in the sky doubt, it is of legitimate concern whether someone knows how to properly approach a problem of this nature with the right data structures, code, and communication.
Good interviewers put the interviewee at complete ease before they bring out the technical questions. From an economics point of view, it is better to hire the person that is in all other ways identical, yet interviews poorly, since they are less likely to want to leave.
Very good illustration, I must say. I strongly disagreed with the GP's rant about knowing O-notation, but I realized he might have faced the first type of questions a lot (e.g tell me the O-complexity of X algorithm).
My experience with Amazon was actually the opposite, but I might have been lucky. All the interviewers were very friendly and started from simple questions and went on to more complex ones. Never a "do you know concept a from language Y". In fact, one of them worked with me through the solution to a pretty tricky problem, and when we were done, he actually took a picture of my solution on the whiteboard! So yeah, it was fun :)
That mirrors my experience with Amazon too. Although it was 3.5hrs of full-on whiteboard coding, I really enjoyed the process and felt like I learned a lot. 3 of the 4 interviewers were excellent.
I generally agree, most logic questions are useless, but Big-O notation I use every day. Programmers who don't understand the performance difference between Array#include? and Set#include? in ruby are dangerous!
I'm curious to know where you use the notation in daily programming. In code comments? Discussions with other developers? Other documentation?
I use the spirt of Big-O almost daily, but do the actual calculations rarely, and basically never use the notation. With some experience, you quickly get an intuitive feel for execution time, which is good enough in most cases.
If I'm reading up on a new technology it's useful to understand what they mean when they say 'operations are O(1)'.
Additionally, when writing an API for use by other developers its useful to say, this operation runs in constant time, that one in logarithmic time.
Redis, for example, mentions the computational complexity of all operations in its docs. Additionally, everyone should know that most decent sorting algorithms are O(n log n), and understand the implications of that.
The last time I talked about Big-O notation was three days ago, when talking about whether or not inserting/deleting from a list should be done online or queued up:
Zarel: and for insertions, aren't they O(nm) for n insertions into a list of size m either way?
Cathy: inserting n elements into a list of size m when done with queueing requires at most m iterations
Cathy: when not done using a queue, it requires at most n*m iterations
Cathy: a factor of n is the difference
Cathy: of course, n might be pretty small, depending on how many insertions happen in the debounce wnidow
Cathy: n might be 10-15
Zarel: how do you insert n elements into a list of size m in m iterations?
Cathy: iterate over the list of m elements
Cathy: well, i guess you'd need to sort the list of n elements first
Zarel: oh, I see, I forgot about sorting the list
Zarel: so it'd be O(n+m)
Zarel: which is admittedly much faster for our purposes
Zarel: well, considering n is likely to be approximately 5, "much" might be an exaggeration
Cathy: in any case, i agree that inserting/deletion approach is the right way to do this
Zarel: incidentally if we wanted we could apply certain optimizations to improve an insertion from O(n)
Zarel: for instance, binary search
Cathy: it's not immediately obvious whether that is faster than the sorted queued
Zarel: actually, that's not a bad idea regardless
Cathy: but it's certainly faster than the naive single insertion
Zarel: I mean, O(n log m) is significantly faster than O(n+m)
Zarel: when n is ~5 and m is ~1500
Cathy: true, and it actually makes it clear that using a queue is unnecessary
I don't know of a better way to discuss optimizing an inner loop than to use big O notation.
Actually using something like a math notation to solve a programming problem is asking for too much. What are those general problems which can't be solved by standard libraries, frameworks and IDE's these days?
And even if you are facing problems that mandate you to roll out your stuff how much of that requires these math tricks.
Given all this, if you truly want to hire a math geek to get some magic done. I guess you are looking at a very small pool of people to hire from and it should hardly be a problem to hire one.
Big-O notation is hardly "math geek" stuff. I'd not call this comment ridiculous, because I might have misunderstood it, but catch me in my cups and ask me for my true opinion and maybe I'd say that.
Big-O notation is pretty basic, and very useful. Combined with a memorized table of powers of two (you don't need all of them - I know up to 2^20, and this has proven sufficient - just enough so you can guess log N for a given value) you have a good chance at being able to make quick calculations about time and space requirements for things. Which, since we don't have infinitely fast computers with infinite amounts of memory, often comes in handy.
I'm not talking about Big-O. I'm talking about the general theme of these interviews.
Besides that I still don't understand this. I don't even know the last time anybody around me ever had a problem or a show stopper problem anything to do with Big-O or an algorithm.
Its always how the programmer can take pressure, how he can contribute to making the project happen, getting things done, how he can collaborate with fellow team mates, how he responds under times of crisis, how good he is with tools, what are his everyday practices, does he write unit tests, does he do code reviews, does he document the API's he writes, how does he package his code, how update he is with software world(tools, framework and techniques) etc etc.
Lets be frank, Its always these things that are relevant today in practical everyday software projects today. Unfortunately people hardly ask questions on these and then complain of bad software practices among people these days.
> I don't even know the last time anybody around me ever had a problem or a show stopper problem anything to do with Big-O or an algorithm.
Either your team does horrifically easy work, or no one has told you about their abstract work. I am sorry, but even writing CRUD apps I have to think about the informatics of the system to make it work reasonably well.
All one really needs to know about Big-O is that nested data lists can be exponentially harder to process. If you understand that then you understand the need for it and when the time arises for you to create these type of structures you'll calculate the time needed to process them accordingly using said method. In turn helping you predict future performance and avoiding bottlenecks.
In your example, I see Big-O as a side effect of using the right data structure for the job. If you need a data structure to hold items that should not be duplicated or ordered then a set (or hash which is likely what the ruby set is based on) is what you want to use because it is not only faster but conveys meaning to the maintenance programmer. If you need order maintained then move up to an ordered set. If you need to keep distinct duplicates then some sort of list is going to be needed (Array, Linked, etc...).
I guess what I'm trying to say is that Big-O between a Set and Array do not matter if you can't use a set to begin with. If the problem fits a set then using a set is right regardless of the performance implications.
I do agree with your overall premise of laziness though. Most programmers default to a list likely because it is the first thing ever taught, and for some reason rarely think beyond it. For me, part of the fun of solving coding problems is finding the best fit data structure :)
It is on purpose. A <bigcompany> recruiter confessed. He said they are specifically targeting recent grads. So plenty of questions are of that type.
I guess they want to indoctrinate and take advantage of them while they are "fresh".
> did like the question I was asked about how I'd design an elevator system and then asked how my solution would scale. That's not too far off from what I'd be doing, albeit in an abstract form.
That would piss me off though too. "So does Amazon write algorithms for its elevators?". Why not just simplify a problem they solve. "So you have key value storage and you want to make it distributed". Why even talk about elevators?
Actually, yes...they do. I work in a building with an elevator control system designed by an internal engineer.
More importantly, the elevator control system uses mathematical expertise that is normally used solving variants of the multi-commodity flow problem, used extensively in the Supply Chain.
Sib has it about right. It is an NP complete problem, and I'm pretty sure the solution is using a Heuristic, but it does seem to be pretty suboptimal during lunch time. Then again, maybe it just seems worse because you wait longer as a tradeoff to having a faster ride to your floor.
When load is not particularly high, it works well (seemingly efficient for passengers and in energy consumption). When load gets high (think ~noon), it tends to perform pretty badly in terms of latency.
Time complexity is important, and hiring someone who doesn't think so is dangerous. That said, it's important to ask about an algorithm you've already discussed. Asking "what's the big-O of <generic algorithm>" also tests for whether you remember the specifics of that algorithm, which isn't terribly useful.
There are jobs where algorithms matter. Where someone picking the wrong data structure or algorithm can completely fuck up the problem, or where the company's secret sauce is doing something much faster than their competitors are doing it.
Yea this is where I'm at with it. If I walk into an interview and the first question is what is the difference between an abstract class and an interface my interest in the interview is probably going to decrease exponentially. How about asking me about my design approach to a specific problem, what frameworks have I try when presented with a specific problem and my thoughts on it, or what I'm currently learning.
I get why they do it but at a certain level it's almost a sign of disrespect to be asked to fizzbuzz if you can verify I've been working at reputable companies for the last x yrs. Let's have a more nuanced discussion about technology so that you can confirm that I didn't just cram certification questions over the weekend and I can get a gauge on the type of problems I may have to tackle on the job.
You assume people working at programming jobs for years on end have a clue... that's not always true. I've worked with people who couldn't do fizzbuzz, or describe the difference between an interface and an abstract class, or any other basic programming stuff.
If they throw you an easy pitch, knock it out of the park and they'll ask you something more difficult afterward.
This is precisely right: I ask people questions like the difference between an interface and an abstract class as a gauge, because I've had so many people answer them really badly.
The question wouldn't normally come out of the blue, though, but as a follow-up from part of a design exercise where they would have just used one or the other in their design, and as part of finding out whether they could justify their choice.
The interview process is so badly gamed that these days its kind of a ceremony. In a recent interview, I gave a candidate a practical problem. He was allowed to use the sed man page, and then I gave him a file and search/replace problem. Nothing much! The candidate fails! Instead he asks me if there is going to be a algorithm/data structure interview. I told him, I don't know but as far as I was concerned I was only going to test him on practical everyday problems.
It was almost like I had committed a sin. He was like, every company has those standard one's so why was I doing like a practical test. These days all candidates do is go through most common puzzles/algorithms/data structure stuff. Its actually quite easy to get access to those. Just spend some time on interview forums and you can practically game any interview, in any company any where around the world today.
My approach is straight simple. Throw a manual/documentation at the person. Give him a problem common enough to test a practical everyday situation. Check if he can pass the the test of reading the documentation and figuring out the solution to the problem. Or give him/her their favorite IDE/text editor + documentation + problem and let them solve it.
If they can do it, or even get the approach right they pass the test.
Else if all they are going to do is vomit some mathematical theory from college text books, which isn't even remotely relevant to our everyday work- I'm hardly interested in hiring such candidates.
May I ask where you work? I'd love to interview at a company like yours.
I'm in the process of changing careers to programming in my late 30s. I dabbled in coding on and off for most of my life, but two years ago I started taking it seriously, read K&R, etc, and completely fell in love with programming. And so I start thinking about working full-time as a developer, and schedule an interview for a programming position for a company in the industry where I've worked for the past decade. I've always been a confident interviewer, but I totally froze in this phone interview. They started the interview off with a tough algorithm problem that I probably could have solved on my own in 30-40 minutes, but couldn't possibly solve on the spot in a conference call, with several people listening in and quizzing me. From there, things went downhill quickly, and I struggled to answer questions that I could have easily answered on my own at my desk. Since then, I've been burned on two freelance programming assignments (that I should have known better than to take given the level of desperation involved), which was quite a blow after spending a decade freelancing for dozens of clients without incident in another industry.
I love programming, and so I continue to code whenever I get the chance (and have two rather large projects underway), but these experiences have shaken my confidence. If most programming jobs involve similar interview processes, I'm not sure this would be the best move for me.
However, this comment has given me hope. If there are companies out there like this, then I stand a fighting chance -- I would excel at an interview where I'm given a problem, a time limit, vim, and then told to go solve it with no whiteboard or group of people watching. Let it be a hard problem, but make it realistic -- not some obscure puzzle or algorithm -- and let me use my own tools. I wish there were some job board where companies with this type of interview process posted jobs.
I'll even spend the extra time on it if we can do something similar to "Here is the problem and the docs. Email me back within the hour with some working code and we'll get on a call to talk through your thought process".
It's like trying to run a competitive 100m without shoes at some interviews I've been through lately.
I do something like that. My first test is ask to reverse a string (not using the reverse function!) in any language and/or pseudocode. Is incredible how badly the test goes with the people! I interview for myself or others more than 100 people (of any background, including university, tech schols...) and I think only 10/12 people do it correctly - barely-
To make it more fair, I always let them alone. And give internet and a computer, and remember them they can use internet and search. Not even with that amount of help the thing work well (I truly have find someone smart enough to copy-paste something and return to me in 1 minute or less).
Then i do the puzzle question, but know I retire it some years ago and some more question about real stuff...
Reversing a string is actually an incredibly common interview question. It's a bit like FizzBuzz (IME even more common, but I can't really speak about the world at large).
It might be an incredibly common interview question, but it's not an incredibly common day-to-day task, particularly when there are methods built in to classes that perform this function. An interesting task? Perhaps. But not reflection of the dev's skills, per se.
You're not asking about syntax with this. It's a problem solving question and at the end of the day, the only real metric here is not whether a dev could code this up, but whether he can code it up quickly, under time constraints, while being stared at, under pressure, with rent or a mortgage on the line, and maybe kids to feed.
I am hard pressed to think of even one real life dev task I had in my entire career where I had to solve a problem under those circumstances. For most devs, we take ownership of the problem and solve it at our desk under reasonable time frames without the spotlight. For bigger problems, we might think about it in the car on the way home. We might sleep on it over the weekend.
That doesn't apply to reversing a string, of course. But you have to consider that, depending on the dev's background and skills, he may be spending time in jQuery, maybe Spring and Hibernate (for Java guys), maybe even some CSS. Maybe he's a GWT guy? Maybe he was assigned to a team dedicated to (God forbid) EJBs. In other words, he may be multifaceted and spends more time working with an API than "just" raw language coding. The language is just a means for working with the API.
And the one thing you want to measure is not something he does daily. Now, if he can't use a for loop, feel free to mock him. But reversing a string? That's clearly a puzzle.
EDIT: I can't get this part out of my mind:
"Is incredible how badly the test goes with the people! I interview for myself or others more than 100 people (of any background, including university, tech schols...) and I think only 10/12 people do it correctly - barely-"
So you continue to use that question? Despite the results? So that when you do hire a guy that passes, he won't ever do this on the job because you know you damn well he should use the reverse() method? Because you won't pay him to code it raw and waste time and money when there is a 1.5 second solution to this? That's sadistic. :-)
Well, yes. Is the most simple one I can think that show if a person can declare variables, do a loop and show the result. If exist something better, I wanna know!
Before that question, I do things more complicated. Like do some CRUD, or some kind of task that was realted to the company I was before. But my first results with that lead me to simplify the interview a lot (in time and questions) because I see that if a person can't do this kind of task is pointless try with something better.
And I don't do much presure. I let them alone and give them all the time they want, with internet acces and that. I continue doing my work while they complete the task.
Is important to understand that in my past companies I was tasked to interview fresh graduated coders, and for a lot of backgrounds. And they pay badly. So no much oportunity of get "top players" back them. However, I interview several people with (in theory) experience or from reputable universities and that not change at all the end results.
P.D: Some point to the complications of do this task properly (because unicode). I don't even expect that kind of depth in the answer, looking more in the basic skill, code formatting, and the ability to explain the code writed...
I don't see how reversing a string is a "puzzle" - I'd put its difficulty on par (or maybe slightly harder than) fizzbuzz. In most mainstream languages, it requires 2 pieces of knowledge:
1) That a string is implemented as an array of characters.
2) How to reverse an array.
The implementation is straightforward and should be trivial for just about anyone who's done much development.
I don't know that the question is that great, but it certainly isn't bad as a fizzbuzz-type question to use as a filter.
def a = "Can a fella get a job?"
StringBuilder b = new StringBuilder()
for (int i = a.length(); i > 0; i--) {
b.append(a.charAt(i-1))
}
print b.toString()
}
But it doesn't use arrays because quite frankly, I haven't used an array in years. I despise the noise of the brackets. :-)
Seriously, I use ArrayLists instead of arrays. But even on this FizzBuzz lite exercise, I got tripped up. My "i > 0" was off by one. And I started down the path of a.substring, but that was a bad move. I quickly switched to charAt.
But about the only relevant piece of this: on an IDE, I was able to place a breakpoint, see where I'm at, what my variables look like, and make adjustments.
That's real life. My original cut of this would have disqualified me. Actually, even the cut above would disqualify me since I didn't use arrays.
But I need to confess. What you see above is literally the first time I've ever reversed a string in this manner. It's just not something I do. And it didn't roll off the tongue so to speak. I knew I had to process the string from the end and work backwards, but it still felt like trivia and not a substantive inquisition into my skills or experience.
Trivial? A bit. It took me about 5 min. The off by one was killing me. It hits you in two places: the i > 0 and the charAt(i-1).
That isn't a bad solution. You obviously hope that someone you're interviewing can come up with something like that fairly quickly, but I certainly wouldn't disqualify you based on your method of solving it or an initial off-by-one error.
> What you see above is literally the first time I've ever reversed a string in this manner. It's just not something I do.
There is a certain amount of value in that - seeing how someone reacts to and solves something they've never seen before. Can you come up with a reasonable plan for attacking the problem, implement it, test it to see if it works in a bunch of cases, and make the necessary adjustments?
To be fair, the coding you do in an interview is a bit artificial - the problems may not be typical and you are without the environment you are usually coding in. The good news is that it is a skill that you can work on and become better at, and it is usually worth doing so if you are pursuing a job and you know these kinds of things will come up. I don't think it is wasted effort either - I've been trying to work on my algorithm/data structure skills by solving interview-type problems, and I've noticed that the work I've been doing to learn to solve them better has been beneficial in my daily programming as well.
Your solution is fine, you've passed, flying colours. I might poke a bit around the edges to see if I can get you to say the word "array", but mostly out of interest.
The important bit is that this is not an assessment of your ability as a programmer, it's a negative filter: Are you one of these people who simply can't program, because if you can't, let's not waste any more of each others time.
I've never liked arrays due to the brackets. But in a twist of irony, I now spend all day in Groovy and I use maps, thus brackets all the time. Yet I don't even notice.
So after carefully thinking about it, I now realize its not the brackets. I can't explain it. There's something about arrays I just despise despite having no similar ill will towards ArrayLists. They just feel so 70s.
And at 1), you went wrong, in any language where strings are in unicode. Combining characters, byte order markers, surrogate pairs... And this is why I dislike the 'reversing a string' question. Because if you actually know anything about real world string handling, you know that correctly reversing a string is impossible, because there just isn't any logical way to reverse some constructs. How do you reverse a string which contains bidirectional control characters? What is the reverse of an optional line-breaking hyphen? Even simple things like parentheses might trip you up.
Once you really start thinking about reversing a string, you realise why it's so difficult to do robustly: because there is actually no logical reason to reverse a string.
As an interview question, I would only credit an interviewee for this question if they basically objected to the requirement in the first place, and explained how bad an idea it was - but since I wouldn't expect most interviewees to do that it would be a poor test to introduce in the interview.
That is a really good point and it complicates a lot of string-based questions, which is why they are almost always constrained (either implicitly or explicitly) by assuming that the string is ASCII only, which seems reasonable to me.
There are probably better filter questions to use that don't require as many artificial constraints.
Reversing a string was initially a good interview question because in C you can keep probing multiple levels of knowledge with it. You learn if the interviewee knows the algorithm, then if they understand how to do it in place with a pointer, then if they understand the XOR pointer trick. It's not a very interesting problem in a language like Ruby or Python.
Dood....I checked your HN profile which led me to your blog. You're talking about decompiling Java code and bytecode diffing. That's hot. See, now that is dev stuff. That's hardcore. Love it.
Now, that was a puzzle :) It was pretty fun to diagnose, even if the knowledge isn't incredibly useful and probably a lot less impressive than it might look at fist glance. Figuring out how stuff works (and why it doesn't work like you think it should) is pretty fun - I wish I had more opportunities to do it.
Damn right. Even worse than whiteboarding random algorithms are logic puzzles.
I can sort of see how implementing my own hash table fits into a dev job, but I why manhole covers are round, or light bulbs in a room has absolutely nothing to do with anything relevant in our field.
People who use logic puzzles as proxies for anything in an interview have my everlasting scorn.
My all-time favorite interview was one when I was interviewing for an SDET intern position at Microsoft. That interview (one in a day of four interviews) featured a great progression: logic puzzle, code the fixed input solution, code the n input solution, test your solution.
The puzzle was (apologies to the interviewer, but it IS a common puzzle) the eight ball and balance problem, where one ball is heavier than the other seven and you need to use a balance scale to find the lightest of the balls in the minimum number of weighs. I'd heard of the puzzle but never worked it out. I initially jumped to the 3 weigh solution, but the interviewer challenged me to keep looking, so I kept thinking and I found the 2 weigh solution. He then asked me to write code that could solve the 8 ball problem. Pretty straightforward. Once I finished that, he had me adopt it to be able to handle [edit: any number of] balls. This tested my ability to abstract my code. Finally, he asked how I would go about testing the program and we talked through that.
I cannot wait to steal that progression when I end up doing technical interviews someday.
Anyone who's studied information theory would destroy that puzzle though. In the same way that someone with a physics degree would perform way above average on a "count the stars" question. Or someone whose dissertation was on computational biology would excel at a string-matching algorithm question.
It is an example of the interviewer (or company/process) mistaking a specific problem as representative of all problems. Presumably they're looking for someone with "general problem solving skills", rather than "knowledge of astrophysics".
You can interpret the response more broadly, though. In this case, it was probably pretty apparent that the person hadn't worked through the problem before, so it is indicative of problem-solving and attitude. If he'd recited the answer from memory that usually comes across and then you have to pick something from a different area. If, as an interviewer, you literally can't find a problem that the interviewee hasn't already thought about, (i.e., he or she knows about computational biology, astrophysics, information theory, etc and none of this was obvious from their resume and previous positions) that can also be a good sign.
It's easy to game your answer as an interviewee though. HN actually had a post a while back about someone who "interviewed" for a living while on unemployment; by the time he was done, he had mastered pretending to think deeply about the answer to any interviewing question.
It's possible, but I doubt it's easy. I assume that the "mastery" of thinking deeply about answering interview questions was self-reported, right? I'd be curious to know how many of the interviewers would agree with him about it.
This is something that's pretty easy to test out in practice interviews with friends. My experience is that it's a lot easier to come up with questions that require thought and insight to answer than it is to answer them.
> Anyone who's studied information theory would destroy that puzzle though. In the same way that someone with a physics degree would perform way above average on a "count the stars" question. Or someone whose dissertation was on computational biology would excel at a string-matching algorithm question.
And...all of this is information the interview panel typically has available when evaluating candidates, as well as the result of any exercises done as part of the interview.
> It is an example of the interviewer (or company/process) mistaking a specific problem as representative of all problems. Presumably they're looking for someone with "general problem solving skills", rather than "knowledge of astrophysics".
If the process uses just one and not a set of exercises spread across problem domains, and gives it a fixed weight in the evaluation process and considers it independently of the candidates resume, the rest of the interview, etc., treating each component orthogonally rather than evaluating them together holistically, that might be true.
But that's not a problem with the puzzle as a tool, that's a problem with how the tool is used.
Anyone who's studied information theory would destroy that puzzle though.
I understand this comment in a general sense--the scale's behavior represents two possible states, so you can use it to filter the answer space in O(log_2 N) ala binary search, and you can get away with O[(log_2 N) - 1] if you partition the answer space the right way.
But what's the information theoretic interpretation of the problem?
It is a source coding problem. Kind of like a Huffman code. Depending on how theoretical or applied your information theory coursework was, you might have worked out simple codes like these in class, and that might be an advantage.
It's actually very similar to your tree reasoning above. The information theoretic outlook uses the probabilities of each of the N states to make a more efficient code, but since there are no probabilities in this case, the two approaches pretty much coincide.
Incidentally, there are three possibilities with each weighing -- scale tips right, scale tips left, scale stays balanced. So your logs should be base 3. In coding language, you are transmitting ternary valued symbols.
And, you should dispense with the O() notation, because the lead constant matters very much in this puzzle, and it happens to be 1.
Re: hash table. I've been asking candidates to basically implement HashMap (for java positions) (without simply using Hashtable). It's an ongoing experiment. Although I've been told that this is a common question, I've actually never seen anyone do very well at it, and I have mixed feelings about it.
On the one hand, it gets at some important java concepts: hashCode() and equals() and the relationship between them. Depending on how they go about it, you can also get into issues of runtime complexity, memory/speed tradeoffs, etc... It's a deep question that offers a lot of room for exploration.
On the other hand, as mentioned, people usually do fairly poorly at it, and so my fear is that it's simply too difficult. I mean, I'd even be satisfied with solutions storing key/value pairs in a list -- inefficient, but workable -- but it's fairly rare that folks even get that close.
To clarify, you don't need to implement a hashing algorithm for this question. (Using hashCode() suffices). And you don't even have to implement a hash table. As I mentioned, a working linked list of key/values would probably be an above average answer, if it came with the understanding of the big-O performance characteristics.
If the motive is laziness or the objective is to trick or intimidate a candidate, then I agree that the method deserves scorn. But I disagree with your position the technique is never reasonable. I like to hire people who know more than I do, and in this circumstance puzzles occasionally help learn about a person.
As a candidate I'm not particularly good at solving puzzles in this kind of situation, but I'm not bothered when people present them because the discussion and interaction about the puzzle matters more to me than getting the answer.
As an interviewer it is important to ask questions that a candidate cannot answer to see how they react. Since I'm a slow thinker myself, I must sometimes resort to "unfair" questions. I don't expect an answer, I expect a reaction.
After establishing a friendly rapport, finding common technical background, presenting an "unfair" puzzle, and discussing the puzzle a bit, then a discussion like this is golden:
Candidate: "Do you believe my ability to solve this puzzle indicates how well I could do this job?"
Me: "No. Though I often enjoy puzzles, I'm not particularly quick with them myself."
Candidate: "Then why ask a question like this?"
Me: "Because I want to see how you react when you don't know an answer right away. In your case, I had to resort to tricks."
> As an interviewer it is important to ask questions that a candidate cannot answer to see how they react.
In the context of an interview, that would just make me horribly frustrated--because my default response to not knowing the answer to a question, nor even being able to perceive a method to resolving my ignorance, is to go out and research the problem.
Whether on the Internet, or by asking my coworkers, or finding an expert, or just running actual experiments and seeing a pattern in the large pile of data generated, this is usually "the" way to get a problem solved, and what I spend nearly 80% of each working day doing.
I don't rely on "sudden gusts of inspiration blowing through me" to solve problems, because the time that could take to happen is unbounded, and it can't be made to scale, unlike research, which can be delegated as you please.
But obviously, research is a tool entirely taken away from you in an interview. The only thing you can do to find out more about the problem is to ask the interviewer themselves--at which point you're basically playing a game of 20 questions, where instead of getting all the data you'd realistically expect in response to each query, you get dealt out precise quantities of informational entropy to string you along.
This contrived situation is not one that applies to the average day in the workplace, and being made annoyed by it is no useful measure of how one reacts to real non-obvious problems.
Answering "In the context of an interview, [I get] frustrated because my default response to not knowing the answer to a question ... is to go out and research the problem [and I cannot do that in an interview]" is a nice response to not knowing an answer and would get bonus points for honesty and self-awareness.
The best candidates will recognize and comment on the ways in which frustration can be either constructive or destructive.
"This contrived situation is not one that applies to the average day in the workplace," is a serious over-generalization. Enormous amounts of code are written based on contrived beliefs about the way business does or should operate. The world would be much better off if programmers were better at recognizing and constructively commenting on these artificialities.
I personally think that "I would go out and research the problem" is the best answer. Introspection about response to feeling sounds like a huge waste of time and something best suited for people's psychiatrist.
I think that question about why manhole covers are round is more of a urban legend than anything else. Or it might be a question asked to someone applying for a non-technical role, to see how they reason about the issue (though it is most definitely not the best question to achieve so).
I was asked that question during an interview for a technical role at Microsoft. Granted, it was 13 years ago at this point, but they did really ask stuff like that, and probably still do.
My impression is that they are no longer in vogue - for a few years when I was in college (2004-ish) they were all over the place. Our industry is extremely faddish, even when it comes to interviewing...
You can still see them in some places I believe. I've heard anecdotally that Microsoft is still a fan of them.
When I was trained as an interviewer at Microsoft (circa 2006), they explicitly warned people against asking questions like that (and had been doing so for some time). There may be older employees who stubbornly persist, though.
I should add that my personal experience contrasts with yours - I did a lot of interviews around 2004 and got asked riddle-type questions by one, maybe two, people. My memory is that they mostly died with the original dot-com boom.
Entirely an informational note, but the manhole-covers question actually originated at Microsoft if I recall correctly. There are several references to it around the web, and at least one primary source confirmation by an interviewer that he used the question on multiple occasions [1].
That said, the fact that it was first introduced by a highly successful company doesn't lend it any merit in my book.
Anecdotal, yes, but I have actually been asked the manhole cover question in an interview. Even if it started out as an urban legend, it does appear to be in use in the wild now.
The question never made any sense. Lots of manhole covers are rectangular. As long as the lip of the manhole is round and small enough, it still won't fall through.
Logic puzzle as the name implies are puzzles that can be solved using logic; typically deduction which neither of your given problems are.
It's important to make this distinction because skills like deduction are actually critical to developers, i.e. identifying the source of buggy behaviour.
Logic puzzles test how much the candidate prepared for logic puzzles. It implicitly tests how much the candidate "wants the job", but it also has the effect of turning away the stars who have options. As the best continue to have more and more options available to them where the team members understand that this kind of interviewing is counterproductive, companies that continue to use puzzles will end up with candidates that need them, instead of the candidates that they need.
Logic puzzles are definitely useful ... how else can the interviewer feel superior (after all, he/she already knows the answer and has probably forgotten how they came by that answer).
Just as a logic puzzle might be a bad predictor of how you'll perform on the job, it's probably also a bad predictor of the quality of the company doing the asking.
I really don't understand why manhole covers or stars in the universe puzzles are considered "logic puzzles". Logic questions are not of the form "why?" and "how?". Their structure boils down to "is this statement true or not?"
It is a poor choice of word. I'd suggest the reason for calling them "logic" is that the process by which you explain what you're thinking and why to the interview gives them an insight into your ability to use logic. The puzzles themselves are categorically NOT logic puzzles.
Yes, you need a little logic. But for those puzzles it is (arguably more) important to know some key facts about the world which form a basis for your reasoning. I don't know, maybe reasoning based on wrong facts will satisfy some interviewers.
Say, I can assume that half the stars in the universe are visible from earth and that there is only one galaxy and reason from that. The basis is clearly false, but I will be able to show off my "logic".
Interviewers ask puzzles and give you tricks because they want to see your problem solving process. Obviously, anyone who hires simply off a litmus test of such questions is not doing their job, but that doesn't preclude the use of such brain teasers/problems at all.
Asking a candidate a difficult brainteaser that may be even close to impossible can be extremely revealing. There isn't a canned response, you get to see them work under pressure, you experience live problem solving and you can understand how they move through their work better. All of these factors are just as important as competency when hiring.
A great programmer doesn't "fizzle" in front of a difficult interview question. He/she may or may not get the correct answer, but as long as they have the requisite problem solving skills and communicate with their interviewer well they will come out fine. I'd say that showing how you fail can be quite beneficial when trying to get a job, especially if you happen to fail gracefully :)
It seems like 37signals is just echoing platitudes to gain karma/publicity at the expense of cheapening the discussion.
People can be nervous on job interview and therefore "shut down" and fail simple tests. This does not reveal a persons real problem solving skills.
I don't consider myself especially nervous in job interviews and when taking tests and exams. But I still had a much harder time solving simple programming tasks than I would "back at my desk" with no-one watching. YMMW.
It would suck to want a job so bad you completely freeze up on simple interviewing tests :(
On the other hand, if I'm hiring someone for a high pressure job I don't want them to fail when under pressure.
I understand your point but I don't really buy it. I train lifeguards as a hobby/side-job (I'm in uni right now) and pressure is the number 1 reason they give us for failing their final pratical test. I can't give a kid a permit to work as a lifeguard if the pressure of an exam makes him screw up because the real life pressure of saving somebody's life is even greater.
The exact same applies here for all jobs where you expect the engineer to work in stressful situations.
"On the other hand, if I'm hiring someone for a high pressure job I don't want them to fail when under pressure."
But those are in no way the same kinds of pressure. The pressure of being able to meet a release date, design incredibly safety sensitive algorithms, etc, are completely different from the social pressure of having someone evaluate work you normally do by yourself, in real time.
This kind of interviewing would work well for the kind of stress a salesman gets, not the kind of stress a software engineer gets.
I completely agree and that's why that part of the interview shouldn't be the only one. As someone said earlier, it should be a negative filter of incredibly unsuitable candidates.
In a perfect setting the interviewer wouldn't evaluate the candidate's ability to solve the problem, but his/her ability to approach it, explore it, ask questions and take steps in the appropriate direction.
But in how many coding jobs is pressure a real and common occurrence?
Even working in front office banking roles, there's only 1 or 2 times per year where you really have to keep your nerve and perform under genuine pressure on timescales of seconds or minutes.
The kind of pressure where 'we need to get this feature out of the door by the end of the week' is a completely different thing altogether.
... pretty much in that order.
Brain teasers test none of those things (maybe a some communication)
I've never met a coder that couldn't eventually figure out a way to hack the code to do what the PMs want. But I have met a lot of coders that take way too long and/or produce horrific code that is a buggy maintenance nightmare. Those people often do ok on brain teasers (or at least do well enough that the interviewer doesn't feel like he can give negative feedback).
A more important point is that brain teasers are a waste of time compared to a million other things you could be doing with the candidate to determine their aptitude, like actually writing code to solve a problem.
"Asking a candidate a difficult brainteaser that may be even close to impossible can be extremely revealing."
I prefer problems that are ridiculously simple on the outside; however, allow for lots of optimization. Choose the next best move in chess or go for example (or design a simple sudoku solver). A completely simplistic solver just chooses a valid move. It becomes more interesting as one looks at how are you modeling state, how are you deciding what is best, etc. This way most people will get something that works, you can see their thought processes (where are they optimizing first, why are they doing that), and maybe learn something new.
"Choose the next best move in chess or go for example (or design a simple sudoku solver). "
You make the assumption that your candidate knows anything about those games. If you placed me in front of a chess board, said "these are what the pieces do" and then asked me to tell you the next best move, that move will either be A) blindingly obvious to anyone who has played chess for years, or B) too difficult to someone who has never played the game before.
Reapply the same paragraph to Go and Sudoku. You've created a filter for someone who plays games, not problem solving skills.
No...I don't make the assumption they know anything about those games. I start off talking to them about some of the above games.. One could just as simply walk through other games (checkers, tick-tack-toe, hangman, etc.). Or...if they've never played games, then you could walk through other somewhat classic problems (how should one load an airplane, the back of a car when travelling).
My point was largely that there are a variety of problems out there which have "no right answer" yet are fairly intuitive to get started on...and figuring out modeling of the problem space in combination with decision logic is an excellent example of problem solving which may help one see how a candidate attacks problems, optimizes, etc.
Sudoku's kind of a fun one, because brute force works just fine. More elegant solutions use the same strategies humans use, but recursive brute force gets the job done just fine.
All good, but the success of this method also depends on how well the interviewer interprets/evaluates the interview. Unfortunately, the danger here is that inexperienced/immature interviewers can easily discard good candidates this way. This is similar to the skill required to grade a written test for partially correct answers vs. right/wrong ones. Hence the criticism on wide-spread usage.
I have heard of interviewers asking candidates to solve various levels on mobile brainteaser apps. Something like this game https://itunes.apple.com/us/app/smart-kick-pro/id590049769?m... could work well as it involves problem solving skills and is a great fun little ice breaker.
> Interviewers ask puzzles and give you tricks because they want to see your problem solving process.
It depends on the "problem solving process" you are trying to get insight into.
These procedural "how does the candidate get to the answer?" questions are great for procedural thinkers. Even if they don't know details that feed into reaching an answer, they can show you how they'd get there.
Unfortunately, if your staff ends up entirely composed of procedural thinkers, you may find your company challenged at innovation. For that, you need candidates good at "Eureka" moments.
Your Eureka candidates will often do quite badly in the procedural "what's your thought process?" questions in the one hour interview context. Measured over a 24 hour period, they may do notably better.
"The problem solver initially has a low probability for success because they use inappropriate knowledge as they set unnecessary constraints on the problem. Once the person relaxes his or her constraints, they can bring previously unavailable knowledge into working memory to solve the problem." -- http://en.wikipedia.org/wiki/Eureka_effect#How_people_solve_...
What you're looking for here is both broad and deep domain expertise, coupled with an ability to understand the question well then let the subconscious mind pull together the insights needed for a breakthrough solution. This process cannot be talked through out loud. The answer may take longer (hours, a day or two, probably reached in the proverbial shower), but will often be an answer that gives you a long term competitive edge over a blunt procedural approach.
This said, it's not clear to me how to interview for this ability. I'd start by considering a way to understand the depth and breadth of a candidate's exposure to technology and domain knowledge. This is the opposite of interview processes using dives into the interviewer's area of expertise. I might consider a two part interview a few days apart, where the candidate gets the context in part one, and shares their thinking in part two.
> Asking a candidate a difficult brainteaser that may be even close to impossible can be extremely revealing. There isn't a canned response, you get to see them work under pressure, you experience live problem solving and you can understand how they move through their work better. All of these factors are just as important as competency when hiring.
Except that this situation is an artificial construct. You're covering their mouth and nose and saying, "Now show me how you breathe." You are favoring candidates whose problem solving is informed by the belief they can reach an emergency answer on their own through brute force, rather than those whose problem solving starts with the assumption that someone else has already solved or mostly solved the problem and who is great at finding the "shoulders of giants" to stand on, building an informed answer a priori.
I'm reminded of getting my deep water scuba certification. You're not interviewed for a diving salvage job by repeatedly seeing how you handle having your breathing apparatus turned off. Yes, a diver should be able to "perform under pressure", but what you really want are divers who help ensure nobody on the team ends up under pressure.
I'm a self-taught programmer. These stories about programming puzzles in interviews always make me nervous to look for a job.
My first job wasn't a programming job, but I did a lot of programming because the need arose and I was the only one capable of doing it.
I always felt too inexperienced to look for a real programming job. A friend pushed me to interview for the position I'm in right now (full time web developer). They didn't make me do any whiteboarding or puzzles. What I did do is give them a code sample and we discussed how what I could have done better and why.
I got the job and I'm doing quite well, despite my lack of a CS background.
Since I don't have a formal CS background, and most of my development experience is web development, I have basically no experience dealing with complex algorithms, but I've found I can solve a hard problem with enough effort (Google, co-workers, etc).
For web developers, wouldn't more appropriate questions relate to "please design a database for a website that needs to track the following", or "What would your process be for designing and implementing a REST API that would...". These would allow the developer to show you their thought process of working through a large problem without there being a necessarily right or wrong answer.
Exactly. I never ask those questions. I ask practical problems that we solve every day here. Some are simplified of course.
Also I don't try to be confrontational, I take the "let's solve this together". That often puts them at ease and makes for a better experience and a more productive interview.
Now it is still an interview and the selection process could be brutal but this way I get the most _relevant_ knowledge and information about the applicant in the shortest amount of time.
The "lets solve this together" approach is always great - I've had people simply give up and not ask any further questions - those are generally "pass" situations. I rather see someone be genuinely interested in how to solve the question - the more questions they ask, the more I will be willing to help them, and the better they will actually do in the interview.
The problem with this though is the meta-strategy of interviewing. Unless your interviewer tells you up front whether or not it's okay for you to ask an unlimited number of questions, you're basically guessing at it.
I know several companies that interview with a "hint" system, where every time you ask a question, they just give you a hint to the problem, generally regardless of the question asked, and then mark it down. Too many hints and you're passed over.
Interesting - while I never said it was OK to ask unlimited questions I have generally gauged how someone is doing and then let them know it's OK to ask questions or need hints. Some people have accepted the offer, and some people have simply given up - which just astonishes me.
How many questions they ask, how often and how relevant those questions are all taken into account when grading communication skills.
Some are just way to shy and never ask question. That's a slight negative. I am shy too, but I need to get the job done I'll go find out who to talk to, ask question, figure out specs, see who's done what before and so on. That is an important trait.
I'll give out hints as well, it is not too structured. I don't immediately subtract some point or anything. In fact I might hold it against them if they don't ask questions.
- Not everyone can produce real-world code. Most of what we do for a living belongs to our employer. Side projects, particularly in the US, can be an issue for IP reasons (California is somewhat of an exception here);
- Side projects, even open source projects, can be of questionable "real world" value;
- I've personally encountered many "programmers" who can talk a good game but can't code a for loop;
- tests like the FizzBuzz test [1] are, in my experience, remarkably effective as an early negative filter. This is an important point. If someone blows you away at FizzBuzz, it doesn't mean they're an awesome engineer. But if they can't do it, it almost certainly means they aren't. The idea here is to spend the most time with candidates who might potentially work out and wasting as little time as possible on those that probably won't;
- the problem with these kinds of whiteboard coding problems is that the tendency is for interviewers to think the problem needs to be "hard". It doesn't. In making it too hard (IMHO) you risk destroying the value of your filter;
- pop-quizzes of obscure language features, the kind that might appear in certification exams, are a waste of time. I have no argument with that;
- whiteboarding code by itself is not a great filter. It should be used in conjunction with a multi-faceted interviewing approach that involves testing fundamentals, the ability to construct a relatively simple algorithm, the issues of working on a team and on a production code base and systems design.
- the problem with simply talking about "real world" code, as the author suggests, is you're no longer finding a good engineer, you're finding someone you like, someone who thinks like you. This falls under the umbrella of cultural fit, which is of course important, but don't mistake that for engineering skill.
- I think we can all agree that "logic" puzzles like "how would you move Mount Fuji?" or "if you shrunk to 1cm in size and dropped in a blender, what would you do?" are stupid.
- testing "back of the envelope" estimation however can be useful. I mean things like "how much storage is required to store satellite images for Google Maps?" The idea isn't to get an accurate answer. It's to see what assumptions the candidate states and, based ont hose assumptions, to come up with a reasonable ballpark number.
The problem here is that there are many engineers who can't comprehend the possibility that there is someone being paid to be a programmer who can't code. But I assure you this is the case. It's shocking but true. Simple coding tests largely filter these people out so if you're offended by such simple tests, just do it and move on. I assure you there's a reason why they exist.
One final prediction: There's some guy here on HN who always posts the exact same huge comment on any hiring thread. I'm sure it'll pop up any moment now.
EDIT: let me add a point about trial periods and take home assignments.
Both of these are guaranteed recipes for mediocrity. Truly outstanding candidates need to justify the time investment for either option and very few companies have the kind of gravitas that would justify it.
Anything written without supervision will be of questionable provenance at best.
As for whether or not someone will work out in your organization, bringing them in for a day is (IMHO) of questionable value. Many engineers are introverts. I include myself in this. It's incredibly awkward as is to be in a new company or even a new team in the same company. I question the value of any such assessment over what you learn in 1-4 hours of interviewing.
One thing you're testing for is how well their brain works with an excess of adrenaline floating around in their body. All of the best interviewers I've known take great pains to make the interviewer feel comfortable and chatty - after all, they aren't going to be filled with interview anxiety when they're actually working there. A minority of your applicants are going to perform badly in that situation because the adrenal glands will effectively reduce the output prefrontal cortex below an acceptable level to solve a problem that they'd be able to solve handily in a situation that didn't involve so much performance anxiety with strangers.
I doubt that constitutes a majority of the people you pass on, but it's definitely going to be double digits.
I've never hired using a white board coding test, and out of 30 or 40 folks i've had to let 2 go pretty quickly because it was clear they couldn't do the work, but the rest did as well as somebody who would have passed one of your quizes - I'd expect we'd see similar long term failure rates in both sets.
I'm sure it makes sense at google where the hiring procedures don't leave much room for error, but I'm sure I've been able to hire some great folks that you would have missed. Arguably the best heads down brilliant engineer I've hired (well tied for #1) completely melted down on his first interview. Tried again with drinks and a show at Yoshi's and he did so well I would have hired him to be my boss. He easily pulled 3x his own weight for us and was very loyal and easy to work with.
So many people go on and on about how hard it is to hire yet they go around systematically rejecting all the same candidates for all the same reasons.
"A minority of your applicants are going to perform badly in that situation because the adrenal glands will effectively reduce the output prefrontal cortex below an acceptable level"
I have an off topic question. I don't see what the problem being discussed has to do with how glands reduce activity in certain parts of the brain. Sounds to me like a fancy way of saying 'people don't perform well under stress'.
You could of course ask - "what mechanisms control human behaviour, and how does stress specifically affect body systems?" Then, great, let's hear all about adrenal glands and prefrontal cortexes. When that's clearly not what the question was though, I find it a form of poor communication.
I've played musical instruments and sung in front of hundreds of people, it's stressful but I'm confident I could code under the same circumstances.
I have panic attacks if I feel too enclosed, particularly big crowds in small spaces. I would definitely fail to code FizzBuzz if I was having a panic attack! But if I needed to, I could disguise it pretty well. You might not notice that I was having a panic attack.
Stress isn't linear, so you can't really extrapolate from it that well.
Also, I would question whether absolutely everybody on your team needs to be exposed to lots of stress. Sure, some people have to care about live systems crashing (as one example of a majorly stressful event), but not everybody, surely?
Personally, interviews don't stress me out and I can do whiteboard coding tests no problem. A big component of that is because I went to the right schools and they rigorously trained me on that from a young age. I don't think it's fair that it's easier for me to get a job than someone whose grandmother was less well off (she paid for my schooling).
Sorry that you feel I was obtuse - it's a quirk and my editor side doesn't always step up.
Poor communication aside, I was attempting to make a finer point than you read to it.
Stress is an extremely broad term medically, and used broader still colloquially. Medically defined stress may include anxiety, but frequently will not.
Anxiety, or technically in this case "performance anxiety" is a much more narrowly defined state with significantly more understanding about causes it, what it causes, and effective treatments.
It's know pretty commonly as "stage fright" - that's what you eliciting in some individuals when you put them in front of people they just met to prove that they are smart. I think we'd all agree that most programmer jobs shouldn't be withheld from someone who has trouble with public speaking.
I chose to use neuro chemical terms not to attempt to confuse anyone - but to make the point that this is a biological process triggered by hormones and not anything related to raw intelligence or a weak personality.
As an aside, it's extremely likely that they wouldn't have any trouble at all at the interview if they took a 80mg of propranolol hydrochloride (a beta blocker) an hour before the interview. It's probably used by 50% or more of public speakers and musicians playing big rooms. Unfortunately it's expensive, on patent and that's an off label use.
If you're not looking for someone to do regular public speaking then detecting what you consider "stress" probably has little bearing on their job applicability.
> Unfortunately it's expensive, on patent and that's an off label use.
In the UK you can still get it prescribed for anxiety, and get it free on the NHS, so if you're wondering about betas it's worth checking what the situation is wherever you live.
I would strongly recommend that you do seek out medical advice regarding them, there are several very important caveats and contraindications.
I think it's a little more specific than "people don't perform well under stress." A software development job may well entail stressful situations—for example, "The founder got interviewed on Oprah and the servers are melting!" But that kind of stress is different from a interview stress. There are definitely folks that will step up to the challenge of melting servers, but go to pieces under the social stress of an interview.
Now, maybe getting into the biological mechanism by which they go to pieces is too specific, and trotsky could have communicated more effectively by eliding it. But it's an important point, and I don't think it reduces to just "people don't perform well under stress."
"I don't see what the problem being discussed has to do with how glands reduce activity in certain parts of the brain. Sounds to me like a fancy way of saying 'people don't perform well under stress'."
That's how I read it as well. But this got me thinking, why is it a good idea to test someone under interview stress when what they'll be doing 99% of the time won't be in interview circumstances?
I find some stressors don't have any negative effect on my performance but stress in an interview makes it much worse.
Agree. I have enormous social anxiety that completely wipes out my thought process... But give me a project to do with a due date (ex a school project due in 2 days) and the stress from that actually motivates me to get it done and does not mash my brain up like the stress of being in a social situation like a interview.
Unfortunately, there's no good solution to this problem. I recently had a company ask me to work on a 2-day project for them (in the office), for which I performed sub-optimally.
There are some other variables which might be involved (I was working with a new language, database & framework), but I never really found a "flow". Reflecting upon it, I think the pressure to create a product + learn the tools in a given timeframe threw me off.
What do you when a couple teammates walk up to your desk out of nowhere, abruptly interrupt your coding zone, and ask you why the build broke when you changed the FoobarService to return X instead of Y when Z? They can't release the latest features until it's fixed! What you were doing isn't important anymore, you're holding up everything!
If you want to work for a web services company: what are you going to do when your service goes down, you're oncall, and angry users, coworkers or your manager start asking you what's wrong?
Are you gonna meltdown? If so, I think you should address that first. My company, which does do whiteboard interviews, can wait to hire you until you won't freak out, and we likely will.
By the way... real work actually gets worse than this. Sometimes the co-workers involved aren't gentle about things in the heat of the moment and they're gonna say you did something stupid, that your mistake was obvious. You need to address their concerns anyway. Or during an outage, your nervous manager might start checking in every 2 minutes - you need to calm him down.
You're just lumping two different things together.
Everyone has performance anxiety at times - whether rare, mild, common, or occasionally overwhelming it's something everyone deals with. Something like public speaking to a room of strangers or flirting with a super model really doesn't share many physiological triggers with diagnosing code you wrote while people you're friends with look on anxiously.
Assuming that because something's easy for you (and most people) that anyone who has trouble with it is likely to be broken in lots of ways isn't too safe a bet.
If someone gets overwhelmed by vertigo on a skyscraper does that mean you won't be able to trust them to drive in rush hour traffic? If a guy pisses his pants when he gets a gun pulled on him does that mean he can't be trusted to be an airline pilot?
Those are exaggerated to make the point, but interviewing is an art. You have great latitude in terms of making people comfortable or uncomfortable depending on your goals. With enough effort I'd be willing to bet you could engineer an interview that would send almost anyone over the edge.
So now that it's just a case of degrees, all you have to do is decide whether you're hiring someone that needs to be able to solve programming puzzles in front of a stranger that's actively critiquing them.
Your argument all comes down to dismissing the content of the interview. Since you clearly assume the interview is worthless, you aren't even discussing this in good faith, so I'm just ignoring this post.
It didn't at all. It carefully explained why the two activities produced entirely different chemical reactions in the brain. If you're too invested in your process to even consider that a test of one thing may not translate into the performance on a totally different task then perhaps you're taking this all a bit too personally.
> What do you when a couple teammates walk up to your desk out of nowhere, abruptly interrupt your coding zone, and ask you why the build broke when you changed the FoobarService to return X instead of Y when Z? They can't release the latest features until it's fixed! What you were doing isn't important anymore, you're holding up everything!
This should have been caught before it was an emergency, through standard processes such as integration and unit testing.
> ... you're oncall, and angry users, coworkers or your manager start asking you what's wrong?
If your coworkers and managers are angry during an outage, or asking anything other than "what can I do to help", they're not helping.
> Sometimes the co-workers involved aren't gentle about things in the heat of the moment ...
Those coworkers need to be instructed in proper workplace behavior.
The time to be critical about work is before an emergency occurs. Institute policies to ensure code quality and IT robustness, define roll-back plans to be enacted in case of failure, and supply other policies, procedures, and evaluation criteria necessary to steer your employees to ensure that the chance of success is maximized (and the chance of outage is minimized).
When (not if) the shit hits the fan, you -- and all your employees -- are prepared for it. If a nervous manager is still checking in every 2 minutes instead of asking how they can help, it's your job to MAKE THEM STOP.
This strategy is how you prevent most failures, and make the ones that happen manageable. Your strategy is how you create failures, and then make them worse for everyone involved when they happen.
> This should have been caught before it was an emergency, through standard processes such as integration and unit testing.
Things always go wrong in the development cycle. Problems happen. Conflicts between individuals occur. I was giving an example. Are you asserting these issues don't arise in the workplace? Or that my example is of a rare situation in software development?
> If your coworkers and managers are angry during an outage, or asking anything other than "what can I do to help", they're not helping.
People sometimes do unhelpful things. Sometimes people even make emotional decisions about what to say to others. Are you aware of the fallibility of man?
> Those coworkers need to be instructed in proper workplace behavior.
They likely will be. Have you met engineers? A lot of them could use some workplace etiquette training! Until then, you need to do your job - you can't just go home because it's stressing you out. This is something that happens in every single workplace. If you can't handle it, you aren't a very valuable employee.
> This strategy is how you prevent most failures, and make the ones that happen manageable. Your strategy is how you create failures, and then make them worse for everyone involved when they happen.
This, and the rambling before it, is completely off-topic. I have no idea why you're rambling on and on about software development practices when the issue I'm addressing is that social stress always exists at work. If you feel my examples are unsatisfactory, but don't have any issue with my core argument, then you aren't being helpful. If you have an issue with my core argument, please address it instead of nitpicking my examples.
Your examples are all cases where someone creates social stress by reacting poorly to avoidable, stressful situations.
Your goal seems to be to hire people that will put up with these avoidable stresses.
My goal is to build an mature workplace where we avoid creating unnecessary social stresses by applying collective wisdom and experience to plan ahead.
This is one of the many difference between engineering and hacking.
Back in my early 20's I applied for a job (which I was basically invited by the manager to apply for) and was extremely nervous in the interview. I got the job and turned out to be one of the most experienced in the department. They later asked me why I was so nervous in the interview, I said I thought I just really wanted the job.
I've come to enjoy technical interviews. How often do you get to spend a whole day talking one-on-one with people in your field who you can learn something from? I enjoy giving technical presentations to groups of 1000 people or so and been not nervous at all. But I have to keep water with me because some reflex makes my mouth get really dry. So I'm fine with public speaking (the most commonly cited phobia).
On the other hand, the prospect of sitting down to pay a bunch of paper bills or do taxes makes my body want to hyperventilate. If, in a place where there are only people around I know well, and I remember an event in my past that involved a social interaction in which I acted suboptimally while at the same time someone is crinkling a plastic bag, I will startle to the point of crying out "Aah!". It's true :-)
My only point here is that people are complex critters who change with time and environment. So attempts to think of others as a box of predictable glands and cortexes (cortices?) can easily turn into a pseudoscientific projection of one's own preconceptions. (I find it very helpful when thinking about myself, however.)
Has anyone ever tried asking candidates to design their own interviews?
Lots of real phenomena occur during an interview. For example, you could go one level deeper and describe the molecular mechanisms between adrenaline and whatever 'receives' adrenaline. That would sound something like:
"What happens in job interview is that adrenaline binds to a variety of adrenergic receptors, which triggers a number of metabolic changes" [1]
It'd be just as real, also clearly connected, but also quite clearly not what the conversation is about.
I feel the problem is that, few of us actually being neuro-scientists; While saying something like "people don't perform well under stress" is something that is close enough to common knowledge, instead saying "adrenal glands will effectively reduce the output prefrontal cortex" leaves the reader with a whole bunch of unanswered questions:
* Does a stressful interview necessarily actually increase adrenaline?
* Is there actually a link between incresed adrenaline and reduced prefrontal cortex output?
* What role does the prefrontal cortex play in answering interview questions?
And while these may all have good answers, it just throws up a whole bunch of mental '[citation needed]' annotations all over the whole argument, that can just be skipped entirely if we aren't talking about neuro-science. And if we insist on talking about the bio-chemestry, we should probably go into the details.
>One thing you're testing for is how well their brain works with an excess of adrenaline floating around in their body.
Unless you are specifically hiring people for a death march kind of project (which I find unwise anyway), targeting people good at high-stress situations doesn't seem very productive.
If your programming projects are a series of high-stress, high-adrenaline runs, either you work for NASA or you're doing it wrong.
As a company you should always advertise if you want your employees to work under a lot of stress. I would for one never choose to work for such groups.
I'm not sure if I'm that guy or not... but anyhow, here's the hiring process that's given me some good results.
Note that this requires that your team does pairing.
Interviewing answers three questions: Can they do the job? Do they get along with the team? And, can they adapt when the game changes?
Before your phone interview, let them know that you want to do a short screen-sharing and pair-programming exercise via Skype, in their own personal development environment.
Start with a chat over Skype. Talk to them a bit about their preferences and experiences at other companies. Are they comfortable to talk to? Do they have opinions about the stuff they've worked with? This is important -- A Java developer that doesn't have a vocal preference on checked exceptions or static typing probably isn't very experienced.
Then give 'em the FizzBuzz.
At this point, do you want to hire them?
If so, it's time to invite them into the office.
When they come in, at the very least you should buy them lunch and pay for transportation. They're taking a day, not paid, to come and interview with you -- show some courtesy.
Have them sit down and pair with three of your more senior engineers for a few hours each, both as a pilot and a copilot.
At the end of the day, take your group of three, and see how your interviewee did:
1. Did the candidate demonstrate the expected level of knowledge and contribution?
2. Would they feel comfortable with the candidate as their direct boss?
3. Would they be upset if this candidate went to work for the competition?
If somebody can get through the phone screen and a day of pairing, they're probably competent, and likewise want to continue working with the people they paired with.
I don't know if this is a Silicon Valley thing or what, but the idea that someone would basically do a day's unpaid work on projects they know nothing about with three different people is just about the most foreign thing I can think of.
That being said, I've never done any pairing so maybe it would work better in that type of environment.
A job interview is going to take at least a day of your time anyway. What does it matter if that time is spent programming, talking over coffee, sketching on a whiteboard, or whatever?
Pretend it's not tech, and Starbucks interview consisted of you working as a barista for a day, or Goldman Sachs wanted you to trade stocks for a day, etc...
I will tell you, back when I bartended in New York (10 years ago), it was pretty common to spend a day or two being "tested" as you made drinks for customers, until they decided you were worth hiring. You wouldn't even keep tips, since they'd test you out just on drinks made for tables, or whatnot.
Nope. No wage, no tips. In my case it lasted two nights only, and was paid the third night (and making the big bucks on weekend nights within a couple of months), but I'd heard of it being as long as a week at other places.
The worst part was, you wouldn't even know how long the trial period was -- they'd just ask you, last minute, if you could come in again for another trial. If it's a place you wanted to work at eventually, you'd say yes.
Starbucks isn't going to bother to fly a potential barista to their location, have several managers assess their ability to perform as a barista, and buy them lunch. As someone looking for a job in the tech industry right now, if a company was willing to pay my transportation costs to interview with them and buy me lunch, I would consider that a fair deal. Even if I didn't get the job, I would be out a day or two of my time (during which I'd be able to scope out the city they were located in anyway) and they'd be out anywhere from hundreds to a few thousand dollars. If they got any value from my work while being assessed, I imagine they would want me on their team.
Having a random applicant "work" a day for free is a pretty crappy proposition in most industries; I suspect tech isn't much different. You gave an example of asking a job applicant for goldman sachs to "trade stocks for a day." Are you kidding? Do you realize how much money a bad applicant could lose in a day? If it's different in tech, it's only because it's (often) easy and painless to roll back an applicant's mistakes.
There are a few ways to approach a job interview. The one I like is to say, "I'm a professional working in this area; those people are professionals working in this area; I should learn what they're up to and we should see if it makes sense for us to work together right now" and right now is key, because sometimes the timing or the particular projects aren't a good match, but the people are and you'll want to stay in touch in the future.
One nice non-coincidental advantage of treating it this way is that "success" is a lot easier to achieve, so the interview's going to be a lot less stressful.
So now approach it from that perspective. You could spend a day (or two hours, or two days, or whatever interview duration) chit chatting, talking to HR people about benefits, describing past projects and whatever. But time is really short and the important questions are: what are you actually going to be working on? and are the people you'll be working with any good? Are their experience, personality, skills, etc. going to set you up to do well, help your career, get stuff out the door, and whatever other concerns you have. How are you possibly going to learn any of that without talking to them at length and in detail about their projects? And the best way to "talk in detail" is to pretend you have the job for a day and get up to speed.
So are you now "working a day for free?" Not really. Because, if the people you're interviewing with are people you'd want to work with, they've had a lot longer than you to think about their problems and know a shitload about them. If you contribute anything of value that day, they probably would have gotten as much value from talking their sticking points over with any bright, engaged outsider to get a fresh perspective.
In the rare occasions where the applicant really does provide high value consulting for free for a day, he or she should have known better than to interview there in the first place. If there really was no way to know in advance, just accept that you had a dud of an interview and go on with your life. You've still learned whether that particular company is one you'd want to have a business relationship with in the future, which is valuable, and whether any of their employees are particularly good or bad, which is also valuable; and, if the company has the right values, there's a good chance that they'd want to hire you for contract work. So even those interviews still tend to work out.
tl;dr - when you interview for a job, no matter what, you're working. You'll get the most out of it as an interviewee if you take it seriously as work and try to get as close to the day-to-day job as you possibly can. If the company materially benefits as well, even better.
Exactly. An interview is work, on both sides of the table, and I see no point in distancing that work from what we actually expect the candidate to do once they get hired.
Puzzles, whiteboard coding, and so on don't tell me whether or not somebody can actually build software.
If the goal was to get free work, we wouldn't sacrifice staff time on the interviews. We'd give the applicant a challenging "homework" assignment and ask them to do it on their own time.
Well with a "regular" interview, it's pretty unlikely that the company is going to make any money off of the direct output of the interview. When helping them develop a feature or product, it's slightly different.
I'm in academia. I've had job interviews where I've literally gone office to office discussing people's research projects and helping them come up with ideas. It was fun, and it never occurred to me that I was being taken advantage of. Worst case scenario, they might think of me as a potential consultant down the line.
I guess it depends on the position, but the point still stands. Given that you spend x amount of time interviewing, what does it matter how it's spent? (with the obvious caveats that if they spend, say, 3 hours screaming at you as a stress test, you probably don't want to work there, etc etc).
>I don't know if this is a Silicon Valley thing or what, but the idea that someone would basically do a day's unpaid work on projects they know nothing about with three different people is just about the most foreign thing I can think of.
Yep. That basically amounts to an application fee.
If you think it costs you something to be interviewed (thus your suggestion about an application fee) it certainly costs the company at least that much if not more. They are paying their staff to work with you on something that likely will not produce direct value for the company (usually a throwaway project or trivial task)
Plus they will spend time before and after the interview discussing your qualifications, performance etc. It comes out about even.
If the job doesn't work out, they lose small% of potential productivity, while you will lose 100% of your income.
Companies have numerous employees, but you only have one employer. You bet everything on your employer, they only bet a small part of their hand on you.
The company is expected to have costs when hiring, it isn't appropriate to push them onto the applicants by having them work it off.
What if a company gave you an itemized bill after your interview? You're still exchanging something of value for their time; is that reasonable or ridiculous?
Seconding this; even for internships I've not once paid for travel costs. Everyone from small, YC startups up to MS/Google paid for cab fare, flights, and hotels.
Most jobs are not provided by SV companies or Microsoft. Most prople are not in a career that typically hires from across the country at their own expense. Paid travel is a luxury enjoyed by a small segment of job candidates.
Expanding the scope of discussion to all jobs is disingenuous. It's pretty clear this thread (not to mention this forum) is fairly software centric.
Regardless, companies across just about all industries that hire from remote talent pools will pay for travel (consulting, finance, government, NGOs, banking, accounting, manufacturing, insurance, you-name-it).
Whether or not a given company expands its talent search to include workers in distant locations is bound by several factors, none of which are unique to "SV and Microsoft".
I doubt there would be anything wrong with saying "we maintain open source project X and we only hire people who have made submissions in this project that we have accepted. If you want to be considered for employment, you must first get involved in the community."
I'm curious if you wouldn't mind sharing a few more details of your experience pairing a candidate in the office with your senior engineers.
- Are the candidates able to help make meaningful contributions to whatever the senior engineers are working on without having the background and domain knowledge?
- Do the employees have to spend a significant amount of the 2-3 hours with the candidate bringing them up to speed on what they are working on, just so that the candidate isn't totally in the dark? If yes, does this totally sideline the employees day/flow?
- What do you do if the employees don't have several hours of non-stop coding lined up for the day?
I don't doubt this method works well - but I'm curious how you work through some of the stickier issues of pairing a total outsider with a fully-up-to-speed engineer and then evaluating the outsider based on what was produced in those 2-3 hours.
- No
- Yes
- You have to put other things on hold and come up with something non-stop code worthy, which may be frustrating.
There's a solution I like though - work on a known bug or new feature in some open source project, potentially of the candidate's choosing, but preferably not one either the candidate or interviewer is already a major contributor to. That way both people come to the problem with a similar low level of expertise and have to work together to come up to speed (which is a great real-work-like exercise). It also makes it obvious what to work non-stop on for that time. Bonus points for being way more useful than a make-work wheel reinvention project and for not being free work for the interviewer's company.
I havent done pairing interviews but I have tried giving candidates some started code and having them develop a solution based on that.
One piece of logistical advice here: keep different kinds of keyboards/IDEs available. I use a mac and I interviewed a candidate who wasnt comfortable with the mac keyboard. In an interview situation where a person is already slightly nervous the last thing you want is for them to be worrying about keyboard shortcuts!
The purpose of pairing in this context is not to see a meaningful contribution; rather, it's to see how the candidate interacts with a codebase he's never seen before, and developers he's never worked with before. Can they spot mistakes? Recognize patterns? Read and write code? Does the candidate know the ecosystem well enough to suggest methods or libraries?
Most importantly, pairing tells us whether or not the candidate can communicate effectively. Are they terrified to offer suggestions, or overly arrogant and controlling? Can they explain clearly? Do they clarify when the other half of the pair doesn't understand?
For example, many moons ago, I did some pairing with a candidate that gave me a suggestion on how I could make a section of my code both smaller and easier to read by combining a couple of iterators in an obvious way that I hadn't noticed. We ended up hiring that guy.
The getting-up-to-speed part happens as they pair, with a lot of explanation happening as they work on the system, and it's another important part of seeing how they work -- can the candidate understand new concepts quickly? Do they stop and ask questions when they don't understand?
This type of interview is a performance hit for that employee for the day, but this is fine, as I consider hiring to be very important, and losing a day or two of engineering time over the course of a month is a price I'm willing to pay -- keep in mind that the phone screens eliminate a lot of people.
Admittedly, I've never had a day when my guys didn't have work to do, but not all interviews involved the company codebase.
I make it a point to carve space out for my team to do miniature research projects, in a fashion similar to the way HP used to work, or Google's "20% time". These are timeboxed, and at the end, the result needs to be made available to the team; and yes, "this didn't work because of X and Y" is totally acceptable. This always pays off in terms of education, and every now and then you get something useful out of them, but they're also fair game for pairing during programming interviews.
At the last place I worked our interview was split into 3 sections; Technical, Personality, and Critical Thinking (mirroring your three questions). While it wasn't a programming job, it was a support job with a wide array of third-party and in-house software, written in a variety of languages. Because of the size and mobility, they preferred resourceful generalists over hiring people who were extremely specialized. When I was interviewing, the whole goal of any puzzles wasn't to find the answer, but insight into their process of assessing the situation and seeing how reasonable their through process was. If they were familiar with the puzzle, it was obvious because there wasn't any thought process before answering--which isn't a problem, just move onto the next question.
The problem with hiring (anyone) is that it's really like "speed dating" - even having multiple interview sessions really doesn't give a greater insight into the candidate. The idea of "umbrella of cultural fit" is what mostly happens with hires.
37Signal's approach of trying them out for size is a good approach, but not practical for everyone else.
... someone being paid to be a programmer who can't code. But I assure you this is the case. It's shocking but true.
This is a shockingly true statement - it's almost beyond comprehensive when witnessed first hand.
What is shocking is the fact that in companies where devs have no say in hiring, managers hire those people and then ignore the fact that they do nothing (or try to promote them out of development) to try and cover up their mistake. It is incredibly demoralizing working ten hour days trying to deliver working code while someone else is producing nothing, getting ahead, and going home at 4 pm. This shit is by far the single most enraging thing in all of software development. That's why I 'keep it real' in interviews.
Concerning the viability of firing under-performing individuals, how likely is that? Are there any implications with indirect costs such as unemployment benefits? (I ask because I do not know - any insight on this would be greatly appreciated.)
I ask difficult interview questions and require actual code to be written on the spot. If the questions can't be answered I let everyone know it. The viability of firing under-performing people is zero. Managers just want headcount and don't care about ability. They play the political game not the build-and-ship-software game.
Are there cost implications? I should think so since the fakers cost lots of money and are not producing any software.
Unfortunately, this happens far too often. However, I have completely bombed on "difficult interview questions" and "code to be written" - merely due to the pressures felt in the interview(s).
As said elsewhere, identifying the "fakers" is extremely difficult - perhaps that's why people who were referred tend[0] to be more successful? (But how do people outside get in?)
[0] Merely a guesstimate based on casual observations.
Many companies are terrified of being sued for firing and will resort to passive-aggressive behavior instead. Most stuff like this is settled out of court so I don't know what the real odds and risks are but anecdotally I've been in a few orgs that weren't shy about firing at all, and didn't suffer for it.
Practical is too much of a general term, but I do agree with your statement that it's the most prudent choice versus the possible downsides.
My usage of the term not practical meant that the employer is spending time and effort on something other than generating revenue. And that also means the potential employee is (potentially) working without pay (or other benefits).
The statement "without pay" is an assumption since the book, "Getting Real", does not explicitly state compensation.[1]
I've worked to two programmers who could not code. I mean literally not code. You cannot imagine how frustrating this became before the managers finally got rid of them.
While terrifying, I think the real danger is hiring someone who can code, but can't do it in a maintainable way. I'd much rather have someone not write a single line of code, rather than have them write 6 months of unmaintainable spaghetti code. Zero contribution is bad, but it's much worse if they have a negative multiplier on the future of the team (if what should take a day to do takes a week to do because of unmaintainable code).
How can this even happen? I mean, isn't it obvious in the first week or two, month tops? If not how can they hide their cluelessness for months until they are laid off?
I've seen it happen a couple times. They get everybody else to help them, so their assignments get done. They invite the CIO out to lunch every couple weeks so they're insulated there (helps if they already know the CIO).
Out of desperation, their immediate supervisor assigns them some non-programming task like kicking off a special report every morning. The manager resigns and now they're the only one with institutional knowledge about how to do that task. So the new manager isn't going to fire them. Besides, the CIO speaks highly of them, so they can't be all that bad…
Month one and two they are 'ramping up' Month three and four they pick something inane like what kind of font the web site should use and schedule lots of one on one face-time meetings with higher ups. The higher the better. Month five and six they are now telling everyone their life goal is to be a project manager or that they are training to be a project/program/product manager. Their work is sort of like a project manager. They did the font project. Remember the font project? Simple game theory prevents managers from acknowledging the truth that they are loafing because that might make all management look like all they do is loaf. The guy's direct manager is overjoyed that all the higher ups are paying attention to the font project.
Aw, c'mon, I want to be asked how to move Mt. Fuji and what would happen if I dropped into a blender (that one's easy: I'm pretty positive that I would blend).
For Mt Fuji, how you'd do it is less important than your first question(s): "Why?" and "What constraints?"
If it's to make space for a hyperspace bypass, I'll just let the Vogons handle it. If time is critical, how much mining equipment and how many many megatons do I have access to? What's the size of the evacuation zone? Is glowing gravel OK? There are some companies in Appalachia that have plenty of experience removing mountains, can I call on them?
Trivial puzzlers like:write a fizzbuzz program, swap two variable contents without using a temp variable, how do you count the stars in the universe
Just encourage people to game the system. It's impossible to get real insight from them, because you can't differentiate between those who worked it out, and those who have read about it. If you're in an interview by now and someone asks one of these cookie-cutter questions like: how much storage is required to store satellite images for Google Maps? or How would you count the starts in the universe?
If you're good at problem solving but haven't though it through before you might come up with a general ballpark solution, or you might freeze under pressure of the interview of course.
If you're a slick liar and have been reading codinghorror, you'll have an answer down pat for this category of questions before you even enter the room, one which you cribbed off the internet, as you will for all the other common categories.
If you're so incompetent that you're neither, I guess you'd be weeded out - not a huge win - I wouldn't expect such people even to get to interview, which is a vastly expensive process (in time and money).
In contrast a real world problem, if the discussion is led properly by the interviewer, offers the opportunity for real in-depth discussion about the sort of problems you're going to be expecting this programmer to tackle every day - what trade-offs should they make when designing data structures to balance normalisation and speed, which libraries would they use to solve a particular problem, which problems they have seen in the past which are similar and how did they solve them, etc etc. and it also deals with cultural fit. There is not a clear dichotomy here between logic (puzzles), and wishy-washy cultural fit (a chat about code) - an informal chat, if held properly, can provide far more insight than a set of questions.
This depends entirely on the sort of job you're hiring for of course, but if you're hiring a web developer then a good background in interview teasers is not really on the list of desirable attributes, but a good understanding of real code is. I guess if you don't control the entire interview process and/or work at a large organisation, this sort of filter might become appealing because of the standard of interviewees you receive, but that's a reaction to a failed process.
One hiring project which I really admire is the stripe CTF as it selects for a certain type of person that they're obviously keen to find. It's a puzzle, but one so original and unique that it's very hard to game, and also one which records the users' attempts in real time (if they keep logs).
As much as people say "well, everyone knows about Fizzbuzz by now," lots of people still can't do it.
"Swapping two variables without a temp" is a bad question, because it depends on knowing the trick.
"Find the 2nd-highest element of this list" is a basic question, and I can change it up it a zillion ways such that anyone trying to just learn all possible permutations will end up doing way more work than someone who can just figure it out on the fly.
I think the swapping two variables has a bit more going for it if they do not know the trick. Explain the concept of invertible functions. Hopefully they would be able to come up with plus and minus - even if you have to walk them through the
a=f(a,b)
b=g(a,b)
a=g(a,b)
From there you can say that works fine with pen and paper but why might it go wrong in a program? So how do you avoid integer overflow? If the signs are the same then use subtract first else use addition first. After we have done that. After this step assume I have returned to the main flow. How do I check if I should use addition or subtraction? If a is bigger than b I must have added so use subtraction. Writing the algorithm this way is trickier than xor. The problems to be solved are to do with understanding concepts
Next you have the xor solution but more important than getting the answer to that might be to see if they can see that the two functions are like encrypt and decrypt where a is the message and b is a key and a + b is the ciphertext c - so that f(m,k) is encrypt and g(c,k) gives the message and g(c,m) gives the key. If they have had any exposure to cryptography then xor should present itself.
If they give xor straight off then you can work backwards.
You're missing the point, and this is how people end up at these "puzzlers". The point of fizzbuzz or the simple questions is to weed out people who can't code at all. If you transform it like you do above, then you're testing something else: how well people know low-level tricks like swapping variables using xor.
Those tricks might be valuable in your workplace, and may make excellent interview questions for you, but they are a very very different type of question than fizzbuzz, at least in purpose, even though they may look superficially similar. In a fizzbuzz type question, you are not looking for brilliance, you are looking to just weed out the people who can't.
Exactly. This is precisely the problem. It's not that any particular question is a "bad" choice, per se, it's that interviewers can fail to realize what they are actually testing for.
I had never seen the XOR algorithm and after seeing the parent post grabbed a pencil and paper and tried to come up with something. Took a couple minutes and I consider myself a middling programmer.
It's impossible to get real insight from them, because you can't differentiate between those who worked it out, and those who have read about it.
The fact that they're involved enough in the industry to know about fizzbuzz is still a strong signal. It will only come up if participate in some sort of developer community, which, I suspect, is something that the completely incompetent don't do, as a rule.
Once again, it's not designed to find the top talent. It's merely a high pass filter that removes the hopeless from the set of people to consider.
But even if fizzbuzz is something that the hopeless start learning by rote, switching it out for any other trivial problem will still work. "Count the number of 'a's that come after 'g's in a string", "print the lyrics to 99 bottles of beer", or hundreds of other problems like that are just fine for the task.
It will only come up if participate in some sort of developer community, which, I suspect, is something that the completely incompetent don't do, as a rule.
A great thing a small company can do to find people is sponsor a users group of some sort. If your company does Java start a JUG, Ruby start a RUG, etc... For the price of some pizza you can informally meet people that by the nature of showing up give a strong signal as people you may want to talk to when looking for your next employee.
Once again, it's not designed to find the top talent. It's merely a high pass filter that removes the hopeless from the set of people to consider.
I'm sure it is useful in some situations, particularly in big companies with a broken hiring process. I imagine 37Signals don't find it useful as they have a more effective screening process, and people coming to interview simply couldn't possibly be that bad as they're referred or they've seen some code they wrote.
If you are really faced with candidates who are incompetent (i.e. HR has filtered candidates for you based on CVs or similar), I think I'd try to automate some kind of online quiz they can go and take before they're even allowed to apply (which can include fizzbuzz or whatever little puzzles you want, checks for cheating etc) - then you can spend the phone screen or interview actually talking about interesting stuff which is in the problem domain they'll actually be tackling and finding out what they'd do with it. I just suspect these questions can be gamed and are not necessarily weeding out all the undesirables, just those who haven't done any homework on your hiring process.
I've seen people hired who talked a good game and could have passed these trivial tests without giving away that they'd just read up on likely ones, but were lacking in all the skills necessary to bring projects in on time and competently - that sort of mediocre candidate who lies about experience and knows the basics is far harder to stop.
It's interesting how different people react to IQ tests or little programming tests though - some people hate the implication of jumping through hoops, and that ability/knowledge can be reduced to such simple set of tests, and resent being subjected to them. Others really don't care and find them interesting or at worst trivial. I'd find it kind of weird trying to guess in an interview if someone who asks for say 'select the second highest number in this list' just wants a quick solution like list.sort[list.count-2] (which in the real world would be the first choice until or unless you are likely to hit major performance issues), or if they want you to write a trivial sorting algorithm for them and explain that you're familiar with the content of an undergraduate CS course.
It's also interesting seeing the range of different interview questions posted here as examples - some are trivial filters as you say, some are more involved, and some require knowledge of basic undergrad stuff, so I'm not completely sure that all of these puzzles are intended as completely trivial, there's always the temptation to elaborate them into some kind of exam (like those posted on HN a while back with maths questions) for esoteric knowledge which is frankly useless in most people's job.
Google has a pretty good reputation for tough filters and still, questions at the Fizzbuzz level are enough to blow many or most candidates away. If you're not asking anything like it, you'd be amazed at how many otherwise intelligent-sounding candidates just. cannot. program. at. all.
Does that mean I fail the test :) A sort then select for the second highest seemed the simplest solution off the cuff (perhaps not the quickest), and if the code is fast enough, there's no point in optimising. Of course you could write a loop and check each value against a stored memo - your O(n) right?, but why bother when the library sort is fast enough for the purpose? If you're dealing with genuinely huge data sets obviously this becomes important, but then often the data will be in a db anyway.
As the person who posted that question, if you gave me that answer right away, I'd say "good, but can you do better?"
"Sort and take the second highest" is a fine first answer. I'd expect most good candidates to start with that. I'd also expect them to say "but I think I can do better" on their own.
This is where I find questions like this a little confusing. To define better (at least in real world situations) the question requires more context. How many records? How are they stored? What are the characteristics of the storage? Which libraries are we using to store it and how do they behave (and in the real world there might be a few)? What are the other constraints? If we request a sort (say in a remote db) will that be quicker than the latency of requesting data in batches and iterating?
Presumably you mean better given infinite memory to store all the records and a very large number of records? Because for most small collections of say a few thousand, a sort then select would be better than an iteration IMHO as it is simpler and the intent is clear if performance is acceptable. Perhaps I have misunderstood the intent of the question though, I'd be interested to hear what you feel is the ideal answer.
I think "do better" is a pretty obvious description, but if the candidate is having trouble with it, I'd say "you have a few million records, and your current mechanism is too slow."
I have patience for candidates, but not an infinite amount. If they keep on trying to come with reasons why they don't need to improve their answer, or try to drown me in a sea of bullshit, that's a no-hire. I'm screening out assholes and prima donnas as much as I'm screening out people who can't code.
Upon being told that "sort and take the second element" isn't good enough, I'd expect the candidate to come up with a linear walk through the code, keeping track of the two top elements.
I'd then ask the candidate to generalize it to "the #n element" instead of "the #2 element," which will probably require a new data element. Keeping those n elements in a sorted list is a fine way to start. A really great candidate would ask if they could re-order the elements as they find them (doing a partial version of a quick sort) or use some variation of a heap-sort with limited space.
Ok, I think that clarifies your intent, the question is purely hypothetical (many records, infinite resources) and designed to elicit the platonic algorithm given no constraints save time. You surely recognise though that 'improving' depends on the constraints which were not given? I find it curious that you characterise questions attempting to relate your problem to real world constraints as 'a sea of bullshit' :) Surely no one tries to avoid answering by asking for clarification of the constraints, wouldn't you normally face this problem with records in a db, not all in memory directly accessed? I'm not sure this interviewing game would suit me.
If sort first was too slow and it had to be done in memory, I would have iterated once and kept the results in an ordered list but not sorted the records during iteration as you have suggested, didn't think of that. Thanks for clarifying.
I think this - I mean the "meaning of better" being obvious or not - is a major difference between most of industry and academia. In academia code readability matters less, it's asymptotic performance matters more and constants can be safely ignored. Reimplementations of things tend to be rewarded, because it shows the student's understanding. It's completely different in the "real world".
And this is why looking at actual code someone wrote is a good idea - because there are things one cannot check during the interview, like how well the candidate structures the code, how well he writes comments, how does he manage the complexity of a growing codebase, how quickly he can learn a new library or tool and so on.
In this particular case I'd probably know that "doing better" means algorithmic complexity, but I would never come up with "but I can do better" remark myself. It's also possible that, when asked to "do better", the first thing I'd try to do would be to write unit tests and docstrings. Is this unacceptable? Would I be seen as an asshole or prima donna?
Also, there's an issue of me being able to come up with a "correct", "better", faster in terms of Big-Oh algorithm. I am not a genius, I know this. I am not and I can not hope to become Dijkstra. I'm also not a computer scientist. I worked hard to be able to read and understand and then code algorithms presented in papers, but I'm damn sure I couldn't come up with even the simplest of them. For example just last week I had to implement a scheduling algorithm called "highest response ratio next". I was able to look it up in the sources, understand it and implement it in a product, but I am sure as hell that I would not invent it during an interview. Does it mean that I'm not qualified for a job? Would I be seen as dumb?
Posts like danielweber wrote above make me nervous. It's probably not what he had in mind, but it seems to me like he would not hire me - and I work in "the industry" for almost ten years now and I have been programming for twenty. I can brush this off as irrelevant - I have a job that I'm proud of and happy with, but what a young, aspiring programmer has to feel when he gets told that he's no good, no hire, because he can't invent algorithms on the spot?
The algorithm design is all said out loud. If I printed up a page of code and handed it to the candidate and said "make it better" I might expect them to work on style.
So I don't mind answering questions during the interview process. The candidate might be nervous and I should be clear about my expectations. I don't like mind games like picking a fight and making sure the candidate responds the "right" way.[1] I would hold no points against them for asking on clarification for what "better" means, as long as I didn't they they were trying to be difficult and accepted that faster was better.
I'm sorry if you don't think I would hire you. Maybe I wouldn't, but that doesn't mean you aren't a good coder. But I also think you shouldn't be intimidated by "algorithm invention." You probably do this a bunch already. In the order you write your code you are probably already doing implicit algorithm invention. I think you can write the code to figure out the biggest number in a list if you sit down to do it, and once you do that finding the second biggest just involves keeping track of one more thing and doing one more test on each pass through.
Often the question is figuring out what data structure you should be using.
Oh, I don't actually expect anyone to give me the perfect answer of how to find the nth-biggest element in the list. In fact, if a candidate did give the perfect answer, I would say "this person is more qualified than me, please hire." I'm not capable myself of giving the perfect answer even though I've thought about it a few times and it's essentially a question I've practiced.
Ok, I thought it was being written as a code, even if just on paper or a board.
"accepted that faster was better"
The point we - grey-area and I - are trying to make is that it's not always the case, or rather that the "betterness" of faster implementation is, in practice, dependent in large parts on other characteristics of the code. If someone deleted slower implementation along with tests and docs and replaced it with faster, but undocumented, untested implementation I would not consider it a change for the better.
And I think you would agree with this in general (of course there are situations when a change like this is necessary!), and I think this is just a misunderstanding on my part - I was thinking about actual code, an implementation, all along while you were talking about description of the algorithm with words. In this case - with one additional assumption that memory complexity is irrelevant or not changed, but that's nitpicking - of course faster is better! :)
"I think you can write the code to figure out the biggest number in a list"
Yes, of course: (foldl max (first lst) (rest lst))
And I can even explain what foldl is and how it works.
"you shouldn't be intimidated by "algorithm invention.""
I am not "intimidated". I know my limits. I wouldn't be able to invent quicksort during the interview and I wouldn' be able to invent this: http://www.jstor.org/discover/10.2307/2346806 even if my life depended on this and I was given all the time of universe to do this (but I still coded it).
This also depends on what we mean by "inventing". Does using some previously learned technique or adjusting other already known algorithm count as an invention?
But from what you say I gather that you wouldn't want me to invent quicksort on the spot - and then it's ok, it seems like I still have a chance of getting a job! ;)
http://www.jstor.org/discover/10.2307/2346806 was not something someone came up with after a few days at it. They had deep understanding of the statistics behind the algorithm, and an almost unfathomable knowledge in the domain, not to mention a large network to help them. I'm sure that most intelligent people would be able come up with it if they had the same circumstances working in their favor.
As a rule of thumb, if you feel that the correct answer depends on these details, ask questions to pin it down. Or, even better, just pick what you think is the most relevant for the company that you're interviewing at and (briefly) say what you're using as working assumptions.
It depends entirely on the person asking the question. Some people probably don't care, but there are some fans of the most efficient possible solution.
At my current job, we used FizzBuzz when we were interviewing potential senior devs. One of the candidates who looked good enough on paper to talk to could not complete FizzBuzz, and in fact spent 20 minutes struggling with it before I decided to end the screening.
It's certainly possible that some of the people who completed FizzBuzz are not good programmers, but it seems pretty clear that the one person who failed it cannot possibly be a good programmer.
Having people go through FizzBuzz (which generally took 5-10 minutes per person) saved us the trouble of a longer interview with more people for at least one clearly unacceptable candidate.
Come to think of it, swapping two variables in place is a horrible question - because you can't really do it! A statement like
a = a + b
requires the (invisible) presence of a register where a + b (or xor or whatever) is placed right before being written back to a. But then you might as well use this register directly for the swap instead of dabbling around with one-to-one functions.
That isn't a "puzzler", in any way, shape or form. It is quite literally "write the most basic, simple, trivial program possible". There is no trick to it, no gotcha, no special information that makes it suddenly go from hard to easy. It is always easy, and is a practical test of actual programming knowledge.
I didn't mean to imply it was hard. As to a test of programming knowledge, I'm glad I'm not in a position where I'm exposed to people who would fail such a test :)
I get it. That said, isn't there concern about candidates just cramming on the top 100 programming puzzles just to be able to ace such interviews?
I take a bit of a different approach. I want to know how they think. Pose a problem and collaboratively work through it on a white board. No code. Just state diagrams, flow charts, whatever.
Not looking for a solution. Looking for the thought process.
Then put up a piece of code. Ask the candidate to explain it and critique the code. How would you improve it? I'd expect them to ask questions like 'How do you define "improve"?'
I tend to be skeptical of the top 100 programming puzzles because I have no way of knowing if they spent the last two weeks training for just that or not.
You can even take the approach of posing a problem that has, on first inspection, nothing whatsoever to do with their job description. A seat-of-the-pants example would be to ask someone being hired for back-end web development work how they would go about designing a small multi-tasking OS for an small 8 bit micro-controller.
Crazy? Maybe. If the candidate launches into a description without prompt you know that this guy (or gal) either has experience outside of the web realm or is inquisitive enough to read and get into other stuff. If they do not, then you get to see how they go about stepping into something they know nothing about. Do they ask you questions or do they cave?
I don't think there's a single answer to this problem. If you need to hire programmers by the hundreds then, by all means, puzzles are almost a must. If you are after a few uniques, I am not sure they are worth it.
"Not looking for a solution. Looking for the thought process."
This is the part I can't understand. When the person is on the job, the opposite is in effect. If you tell the candidate to solve a hard problem, you don't wan them to just spit out their thought process, do you? No, you want them to solve the problem. Why the opposite for interviews?
If I'm an employer and you are working for me, I am asking you to solve a problem because there is some value in the result.
If I'm interviewing you, chances are, the result has no intrinsic value - I already (hopefully) know it. What I want to know is "Can you solve other similarly difficult problems?" In this context, being able to come up with a solution to the problem isn't as useful in answering that as knowing how you solved it.
Probably because you have, at best, a few hours in an interview to determine whether or not someone can solve the hard problems. Many hard problems take more than a few hours to solve, so the interview process serves as a gauge of how likely a candidate can do this on the job; of course after other vetting methods were applied.
> whiteboarding code by itself is not a great filter. It should be used in conjunction with a multi-faceted interviewing approach that involves testing fundamentals, the ability to construct a relatively simple algorithm, the issues of working on a team and on a production code base and systems design.
This I can heartily applaud. However, I take issue with this:
> tests like the FizzBuzz test [1] are, in my experience, remarkably effective [...]
I'd never heard of the FizzBuzz test before, just tried it.
You may as well have asked me to sing the solution out loud. It was that bad.
Paper turned out to be a completely unfamiliar medium for coding, despite years of whiteboarding solutions with colleagues. I instantly felt like I was using a totally different part of my brain. I practically lost the ability to write with a pen and significantly lost the ability to code. My heart rate rose substantially with fear of the unfamiliar. This tiny test took me ten minutes, and when I checked it later I found a syntax error.
And that was in my favourite chair, at my desk, at home on a quiet sunny Sunday morning, working in a language I know inside out. Under pressure, with strangers, in an unfamiliar job interview room? I'd have just fallen apart completely.
Back at the command-line? 30sec in a one-liner in a language I'm still learning.
I will not be adopting this method to filter hires, having self-identified myself as a false negative. I am never out to embarrass someone in a job interview - I want them to show me their strengths! If they prefer it (they usually do), I'll give them a real computer and ask them to shine using that.
I don't know if we've gotten there yet, but Fizz Buzz specifically may not always be an effective weeding-out tool: once enough people have heard of it, they may prepare specifically for it. Perhaps a better approach would be to come up with your own small test of comparable difficulty, sort of like a blog that has a custom but otherwise extremely hackable way of testing if you're a real person.
> Not everyone can produce real-world code. Most of what we do for a living belongs to our employer. Side projects, particularly in the US, can be an issue for IP reasons (California is somewhat of an exception here)
Says who? My side project that I do on my own time doesn't use the same language as my day job nor does it tackle the same problem domain. How can this be construed as an intellectual property issue?
Says the courts in many US states who have upheld ridiculous contract clauses where an employer claims ownership of everything a worker makes, even on his own time.
It's entirely possible that your contract has no such clause, but they are disturbingly common.
That depends on the agreement you signed. It's common for the invention assignment agreement you sign when you start working at a software company to cover anything you do, even on your own time with your own equipment, if it "relates to the Company's business, or actual or demonstrably anticipated research or development of the Company", or similar language. If your employer is a large company with its fingers in many pies, this could apply even if the work has nothing to do with what you yourself do for them.
Your employment agreement may include a broad intellectual property assignment clause. If it does and you don't live in a jurisdiction that has legislatively restricted IP assignment clauses the programming language and problem domain of your side project is unlikely to matter.
I'm sorry but writing code on a write board, ever, is stupid. I'm happy to write out architectures on a whiteboard, but code, never. Asking someone to write code on a whiteboard is the equivalent to asking a UI designer to chisel wood to demonstrate their ability to develop a web UI. It's rediculous.
I'm sorry but you are incorrect. It is not equivalent. It would make perfect sense to ask UI designer to draft a design on a white board. Same applies to coding. You don't get the point. It's about seeing how the applicant thinks and communicates under pressure. The thinking out aloud part is more important than the white board part. Everyone knows this, except you?
You can determine how someone thinks without trying to code on a whiteboard. I've interviewed many people for tech jobs recently, I've never even considered asking them to code on a whiteboard.
Writing code on a whiteboard is massively obnoxious. The times I've been asked to write code on a whiteboard I've spent more time screwing with the 'interface' than I ever have spent 'thinking'. e.g. trying to keep a line of text level, figuring out how to get brackets to line up, and erasing lines to reorder or writing lines off to the side with an arrow pointing to where it should go. A whiteboard is the worst possible interface for creating strictly structured text.
You aren't testing how the person thinks, your testing how they deal will a situation you've artificially made as awkward as possible, why? What does this actually indicate to you about the applicant?
If you want to quiz a candidate on how they think when coding then hand them a laptop that is already setup to duplicate its display to a projector. Or give them a simple 'take home' project. You'll learn vasty more than having them regurgitate algorithms on a whiteboard.
The basic question is whether it is a good idea to ask questions which humiliate a sizeable portion of interviewee simply because this maximizes your personal utility?
I don't even know for certain if such an approach does maximize your time versus finding-good-programmers ratio but let's assume it does.
The thing is you are willing to maximize this ratio by doing things that treat some portion people who have ability as if they don't (even it's also treating people without ability fairly)?
I'd claim that doing that sets you for up an adversarial relationship with everyone in the workplace - the people hire you can feel they're under-the-gun. I strongly suspect that the "almost no one can code" crowd are the people who live in these adversarial environments, which produce bad code through all the emotional and organizational games this implies.
I think people are up voting this one based off just the message, rather than the content. The message is important, I guess. Plus a lot of 37signals staffer's and friends use HN - they will likely be up voting this based off a Facebook post or similar. ~10 of the 21 points being users affiliated with 37signals is very likely.
It seems more like they make a risk-free blog post parroting general HN sentiment every couple of weeks for the karma/publicity. Not that fresh or honest if you ask me, and hardly pragmatic.
He says on twitter. To gain more publicity from HN. Just because he hates that there's some people here who call him on his bullshit, or even if he hated HN for real, that wouldn't mean he doesn't use it for marketing.
If they actually, honestly didn't want their inane spam posted on HN, they could easily achieve that simply by returning a non-response or redirect or error for any requests with a HN referer header. They do not do that because they actually do want to continue getting traffic from the inexplicable portion of HN that insists on posting their crap all the time.
The question I always like to ask and reserve a good chunk of time for is along the lines of "What non-technical hobby or interest do you have?" and then "If you had all the technical resources you could dream of, how would you now bring something new to your hobby?" and once we've gone through that the real question is:
"Given that when you're duck hunting the key is to shoot ahead of where the duck is, and that a lot of the suggestions have been the next iteration applied to the hobby/pastime... what's the third or fourth iteration?".
Basically I want to see the excitement and passion about technology, their pastime, and how they think, how they identify problems, solve them in numerous ways, go beyond just today's solution. And I get wrapped up in it too, I also get excited by some of the stuff that comes up.
With a great candidate... the interview ends with us both inspired and a thousand new thoughts floating around the room.
The vast majority of candidates cannot leap beyond the first iteration, and some struggle to even see how anything within their hobby could ever change.
It's fairly irrelevant, the last part is the important part... very rarely I do find someone who stubbornly declare that they have no other interests at all, but then I just propose for them some pastime or other that may fascinate them.
One interview was someone into rally driving and who was dreaming up HUD units in his car, to allow him to see his progress in real-time compared to those who had already raced, and to report tons of sensor based info back to the press hut.
Then there was the rock climber who wondered whether rope tension could reveal the likelihood of the tether point giving way.
Good hires raise interesting questions and then set to figuring out the many ways to answer them.
The majority of hires I've made are technical, but their pastimes are not generally "When I go home I code".
About a year ago I travelled to the south west (of UK) for a job - A full day of interviews, product ideas, I nailed the whole lot. Apart from the obligatory coding test. I have 15 years in this game and I have coded OSS or commercial code every day or week in Python, the language of the test, since 96.
It was a codility.com test - I had not actually been expecting it but, kerpow. I rewrote my own abs() because I forgot such a thing existed, the problem itself is still blanked from my brain - and every passing minute it got worse.
After an hour and a half, the interviewer took pity on me and drove me to the station.
I actually sat on the train took the codility demo test. 100%, 3 minutes. Signed up, took more. I could do them. Just impossible to know what went wrong but it went badly wrong.
Ultimately it worked out well. I would have had to live weekdays down there and my marriage probably would not have survived it. Now I work 15 minutes walk away and give my kids breakfast each day after we sit on the sofa and watch Mister Maker.
For me that test was a wake up call.
Firstly, I run my own business - being at the mercy of one boss is rubbish.
Secondly, these tests are rubbish for deciding who to hire - but they are OK for programmers as a form of continuing education.
Thirdly, the exercises at the end of each chapter of SICP are much much better form of education.
Forthly, I was once asked how a large corporate IT shop should handle a reduction in workforce. I said march everyone into a room and those who cant code FizzBuzz get a pink slip
Even given my experiences above - I still think that is the best option. Sometimes a guy just dies on that day. Its unlucky. But sometimes it works out for the best.
I never heard of codility so I went to take the demo test. I couldn't pass that in English, much less Java. Lol. That's not a programming test. That's a math test.
I do tests for prospective candidates, but my approach is slightly different.
After the interview I email them a small unfinished program. I then give them a set of instructions on what I would like them to do to finish it off, and they can then do it at home on their own dev setup; with the internet and any other resource they would normally use for development.
I give a coding style guide, and in the areas where some specific technique is required I make that explicit.
The app has a few bugs in it, and some of the techniques asked for are slightly leftfield.
I instruct them that if at any point they have any questions then feel free to email me - and they won't be marked down for asking questions.
In fact I mark candidates up if they ask questions.
The task shouldn't take more than 30-45 mins for a decent coder.
I find this to be a really valuable approach to seeing how a coder works. It shows how they work with others' code, it shows they can follow instruction, it shows they can communicate when unsure rather than hiding away and it gives a good guide to their style of coding.
It's a far less confrontational approach, and gives the candidate the best chance of showing what they can do.
In the interview I tend to be much more interested in the person, because I'd rather have a coder who is slightly less capable that I can train, than a coder who is amazing at everything but won't fit in. I'll still discuss past projects and dig deeper where necessary, but I like to find out how they influenced the outcomes of projects rather than get too deep on algorithms and the like.
I think in addition to the measures he says, if you're hiring for a web position, it's good to make sure they understand how the stuff they're using functions. If they don't have a good grasp of how http works / url parameters / that kind of thing, then they can have some nice looking code which seems to work, but has faulty assumptions that can be security and bug nightmares down the road.
Of course, you could always take smart people and train them - but seriously, who does that anymore?
I've never actually seen that happen. This is what happens:
1) Large companies will import H1Bs and/or open development offices in India and similar.
2) Startups will have the founders learn to code (this often ends well), or they will outsource the development work (this often ends badly).
3) Companies will hang outside universities at graduation/trade shows and try to grab promising students (this is mainly restricted to top tier universities)
You may have good examples of it happening though?
We hire folks who have either a CompSci background or "skillz". If you can walk the walk a piece of paper is just signaling.
At 2600hz we have folks with Masters Degrees and folks who learned to code the hard way, but it's important to understand that it's easier to start with a foundation and build than it is to start from scratch. That being said, old habits die hard and it can be difficult to unlearn them.
Ultimately, referrals are almost always the best source of candidates. The best way to interview is with a conversation. You can learn just about everything you want to know if you ask the right questions.
Of course, you could always take smart people and train them - but seriously, who does that anymore?
The sad truth about this is that I think many companies are afraid an employee will leave shortly after receiving training. They fear that a competitor will entice away the employee without incurring the training cost. Of course, one way to avoid this is to make the work environment desirable enough so that no one wants to leave but that takes effort.
This imply that the job market has given in to the consumerism habit of the rest of society. We all want to have everything right now, but the gratification of earning last longer with learning than with "being given". If that all makes sense.
Hiring smart people and training them is definitely more of an Ops route than it is a development route. That's how I got my job as a Linux admin, I worked in tech support and learned a bunch then got promoted. I was already ahead of the curve anyway, but I was definitely not qualified to be an admin before my stint in tech support.
The best interview I ever had, I was given a problem to solve with a system, a whiteboard and 20 minutes.
I sketched out a quick ERD, and explained the high level architecture of the system. No code was written -- it was assumed that I could implement said system if I could design it.
The worst job interview grilled me on low level algorithms (I'm not a CS grad) and wanted me to sketch out a quick sort. Of course, I can describe a quick sort now as a result of said interview, but since it was all C# I doubt we were going to have to implement our own sort algorithms.
Another terrible interviewer asked me about projects I did in college. I finished grad school 3 years ago, and have 8 years professional experience...
A. Friggin. Men. I just wish they'd make it easier for us to weed ourselves out. Just come straight out and ask for CS grads only. None of that "or equivalent experience" nonsense. I also don't have a CS degree. I have the bastard stepchild degree (CIS). :-)
Just ask for CS ONLY and we can easily avoid you and make this easier on all of us.
I've been on both sides of the hiring process - and most recently was going through the interview process again as I sought out my next opportunity. One thing that stood out to me was the way companies would approach the white boarding or coding challenges. While I agree with the articles issue with coding up a specific question (especially when it is yet another fizz-buzz - even if it has a twist to it) - the puzzle style interview can provide some kind of insight in to the thought process behind someones work. One company in particular really stood out by having a great white boarding exercise that not only was entertaining, but offered a decent amount of a challenge that it really made them a frontrunner in my final decision making process. A good coding interview can be beneficial not only to the hiring company, but can also give decent insight in to how the company works for the interviewee - and that is something that should be equally important if you are trying to attract talent.
On one hand I would love to - and on the other hand I don't want to give this companies interview question away out in the open. If you want, you can email me (email is in my profile) and I would be more than happy to share it directly!
At ITA Software, we found in-person quizzing to be a lot less useful an indicator than looking at puzzle (or other) code someone had written "offline".
We also found puzzles to be a useful talent magnet. If you're Google (or perhaps 37Signals) you don't need a talent magnet. If you're a small company nobody's heard of, one can be quite helpful.
Not that it matters anymore since this was pre-Google, but I swear to god, someone actually asked me the manhole cover question when I interviewed at ITA in 2010. This was for a DevOps type role, so maybe they were less well organized on that side of the house.
At first I laughed because I thought he was joking until he I saw he was serious. He followed that up with something about stacking weights on a seesaw.
I remember actually working on some of the puzzles that were posted by ITA in the Boston MBTA subway. I didn't submit them (since I wasn't interested in working at ITA) but they introduced me to interesting problems. For example, there was a problem related to generating rebusses* whose solution contributed to the solution to another problem I was working on.
I think it's the opposite. If the first thing you confront me with in your interview process is a puzzle and I don't know anything about your IT department, I'm going to assume that all your managers are pointy-haired bosses.
This topic keeps coming around and it always makes me wonder why people have such a hard time hiring good programmers.
I say because the best places I've worked at already have a "system" in place for their front-end people. You can still be a marginal developer and do well if you follow the system. It takes a few months, but once you know how they write their code and the standards they use, it's pretty easy to write sustainable, reusable code.
This also makes it considerable easier to think and work outside the box. You can take a few of the better programmers, tell them about a new framework or approach you want to use on a project and let them loose. If it's a success, you merge these new ideas into the existing ecosystem. Simply repeat as necessary.
The industry really puts a lot of hard work into finding and hiring good candidates, but it's not the programmers who are important. It's the system you're plugging them into which matters.
When I came to Google for my onsite interview, in the group of people scheduled for that day there was this guy with an alpha-nerd attitude including a Rubik cube that he kept playing effortlessly -- he could completely scramble the thing and then solve it in seconds almost without looking at it. I was never able to solve more than two faces of Rubik in my life :) fortunately no stupid puzzles at all in the interviews (at least not in mine), I got in. The Rubik guy, never saw again. FWIW.
Key word here was "front-end". Algorithms aren't usually featured as often there. Before we break out the broad brushes, lets not paint all programming jobs as requiring the same skills.
[1] Eric Lippert summed this up quite nicely:
>> Does not having knowledge in Data Structures really affect one's career in programming?
> Well it certainly will prevent you from getting a job on my team. But like I said before, programming is a huge field. There are lots of kinds of computer programming that don't require knowledge of data structures.
Over time I've made interview challenge problems progressively easier, as I've found that the trickier ones don't give any more information than the easier ones. My challenge now basically amounts to 'read this doubly nested for loop and tell me what it does', equivalent in difficulty to fizzbuzz.
If you can't solve that, even in the pressure of an interview, then I'm sorry, but you have no business working as a programmer, and I'm offended that you even applied.
The last interview I gave for a senior dev I tried making up some custom discussion questions to give after the basic tell me about yourself stuff. The questions boiled down to:
1. Junior developer code review where a save method included a call to a list method "because it's more efficient" than making a separate ajax call. Explain to the junior developer why that's not necessarily a good idea.
2. UI wants a tree view of categories where each category has a name and a parent category. The number of categories can be arbitrarily large. Discuss problems you might run into using a database agnositic approach (e.g. Hibernate) and what solutions you'd propose.
3. Prototyping a new app, two potential clients in the same domain have widely different ideas of how those domains should be represented (i.e. obviously similar domain class names, very different properties). Discuss ways of modeling domain classes such that each client sees the data in their preferred format.
All in all, I think the interview went pretty well because the discussion questions led to discussing all sorts of related topics that gave me a good feel for the types of problems the candidate had run into and if he had success in dealing with them.
It was great giving an interview without arcane things like explain the yield keyword in C# and how it might be used.
As Cletus wrote in his detailed comment, "this is the issue that will just never die." We discuss company hiring procedures here on Hacker News over and over and over, because most of us have applied for a job at least once in our life, and those of us who are building up a start-up have to think about whom we would hire.
The point in the submitted blog post by 37 Signals, "The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written, talking through bigger picture issues, and, if all that is swell, trying them out for size," basically says that you should do a work-sample test to hire a programmer. And that's what research says. The more a hiring manager understands what a worker will do on the job, and the better the manager appreciates what a new hire may grow into doing after a few years on the job, the better the manager can devise a work-sample test to identify a good worker. There is a research base for this. Work-sample tests aren't perfect hiring procedures either--nothing is foolproof, and every hiring procedure goes wrong with both "false positives" and "false negatives"--but you can improve your odds of hiring a good worker by using a work-sample test routinely whenever you are hiring. As a job applicant, you can select better rather than worse bosses and co-workers by aiming most of your applications at companies that explicitly use work-sample tests as part of the hiring process.
With the help of several Hacker News participants, I have written a FAQ about company hiring procedures, revised over a few months of edits to fit the typical recurrent HN discussion of this issue. See
for the latest posting of that, with full references to the research literature and legal cases about hiring workers in today's world. Feel free to contact me through my HN profile
if you have suggestions for improving that FAQ before I post it to my personal website.
P.S. Even before I saw the prediction at the end of Cletus's comment, I planned to make the prediction untrue.
EDIT TO RESPOND TO FIRST REPLY ABOUT PUZZLE QUESTIONS: Yes, if a company in the United States insists on using puzzle questions as a hiring procedure, and justifies using those puzzle questions by saying that they want to see which applicants are "good problem-solvers," or "able to think on their feet," a rejected job applicant just might be able to subject the company to a very expensive lawsuit based on employment discrimination, unless the company has prepared beforehand a validation study showing that those puzzle questions have a bona-fide relationship to work requirements. I would not advise a company to take that risk, especially when the legally safer alternative of doing a straight-up work-sample test is available. The law is different in other countries, and as a reply in this thread points out, in the EU it is generally legal to use straight-up IQ-type tests in hiring processes, although those are underused by private companies in Europe, according to the sources in my FAQ post.
ANOTHER EDIT, TO LINK TO AN UNBELIEVABLE ANECDOTE ABOUT HIRING:
In August 2012, I heard a story from a hiring manager of programmers about the hiring procedure he uses as an initial screen for applicants who have degrees in computer science: "Write a loop that displays the numbers 1 to 100." That sounds awfully easy, even to me, but he says that the great majority of his applicants with accredited CS degrees fail that screening test. My earlier telling of the full anecdote
seem pretty nearly unbelievable in what they imply about how clueless many CS grads are, and yet I think the anecdote is a true description of reality. Cletus too mentions in this thread, with considerable agreement from reply comments, that many people hired as programmers cannot actually program.
All this teeth-gnashing regarding programming and logic puzzles are simply to cover up the simple fact that most people (even around here) are insecure about their programming skill, and more-so their intelligence. The fact is that these types of questions are indeed backed up by the very research you cite, namely that a work-sample test and general mental ability are the only reliable predictors of job performance. But this is exactly what programming puzzles are (and to a lesser extent logic puzzles). A programming puzzle is simply the intersection of a work-sample test and an IQ test. And of course, logic skills and programming skills go hand-in-hand. Instead of writing endless blog posts and internet comments about how useless these tests are (trying to convince a potential future hiring manager not to use them on you), just get better at them! You'll be better off for it.
Of course, not all programming jobs (most, in fact) need top 10% programmers with gifted IQs. It is important that jobs are honest with themselves regarding the level of talent they need for their next sexting iphone app. But this too is a part of the problem. Everyone believes their shitty app or website is going to change the world, and thus need world-class developers to help them realize their vision. Egos will be the death of us all.
> A programming puzzle is simply the intersection of a work-sample test and an IQ test.
No: a programming puzzle is the intersection of a work-sample test with one question from an IQ test. IQ tests get a good handle on your intelligence by testing you over a wide range of near-trivial questions, each known to similarly measure intelligence, and then adding the scores up. By giving you so many opportunities to succeed or fail, they don't fall victim to the "I just couldn't figure this one out" problem. Whereas programming puzzles do that all the time.
Now, if your interview consisted of 200 programming puzzles, each of which should take roughly 20 seconds to solve, then you could say it's getting a fair handle on someone's IQ as measured through the lens of programming. But throwing two 10-minute puzzles into the interview does nothing to help you measure anything. The "scores" you get are black-and-white threshold filters on a set of numbers (IQ) that are almost totally middle-grey.
> a programming puzzle is the intersection of a work-sample test with one question from an IQ test
In the worst case this is true. But ideally the question will have various components, and various levels of difficulty. I'm reminded of a blog posted here some time ago that had the candidate split a string based on a provided dictionary of words (or something to that effect). The best candidates produced maximally efficient code including memoization or dynamic programming. The worst failed to implement a basic "split on the space" functionality. Besides these two extremes there was a whole spectrum in between. The point is to not give questions that are Pass/Fail, but a question that has a spectrum of possible answers allowing one to find the limits of the candidates knowledge.
That still doesn't help the candidates who just don't see the trick.
What if you know memoization and dynamic programming, but it just doesn't occur to you that day, that to correctly split on spaces (not adding empty tokens at the beginning and end of the string, etc.) you need to use a state machine?
You could give the candidate that "step" of the solution and continue on to see if they know the rest, but the candidate will likely be feeling a loss of confidence. They'll think "hey, if I couldn't answer the 'easier' part, why should I be able to answer the 'hard' part?" and then their mind will lock up even harder when they search it for the next step.
This is why IQ tests are organized as a series of orthogonal questions. No question is dependent on any other question, so you can just move on quickly from a failure and prove your worth on another question.
In fact, on a usual IQ test, you're even given five or six "presentations" of each question, each slightly different--like math homework questions with different numbers inserted. It happens quite often that you'll get stuck on, say, recognizing how one specific pattern of shapes go together, but will be able to see a different pattern instantly.
Interviews do the opposite of this; they are presented as if intelligence is a linear quantity, and if you fail on any part of a puzzle, you couldn't possibly be able to solve a "harder" part. In other words, interviewers are using the "error" theory of intelligence, rather than the "bugs" theory[1].
>That still doesn't help the candidates who just don't see the trick.
This may be true, but at the same time candidates who DO see the trick get positive points. Because a lot of these puzzle questions simply require you to think outside the box, if you're forgive the use of that phrase, and honestly if you have hard programming problems to solve that is a killer skill.
I have (once) had the problem where my mind locked up on a puzzle question in an interview. (Disclaimer: Usually I'm really good at them, but then I usually get most of the IQ questions right too...) But I got the job anyway.
The point is that HOW you use these questions is at least as important as the content of the questions. If you ARE willing to work with the candidate on finding the answer, and you make it clear that it's not a make-or-break question, then I think it CAN be (at least) a positive filter for awesomeness.
Yes, sometimes you could let someone fall through the cracks that way. Nothing about the interview process is perfect, though, and if what you really need is someone who is awesome, then letting someone get away who IS awesome is much better than serially hiring people who AREN'T awesome hoping that you find one eventually.
That said, and as others have pointed out in comments: 99% of the projects I hear about on HN would NOT require awesome. They just require competence. And solving IQ puzzles isn't how you detect competence.
If your point is that these sorts of tests don't have any statistical power, I would whole-heartedly agree. Certainly you have to be careful with the types of conclusions one draws from the results of such tests. But they are not useless. The probability that a candidate will work out after having solved the problem with the best and most efficient solution is certainly much higher than if she only gave a middle-of-the-road solution. They are useful as positive filters.
And just to be a unnecessarily technical: if you accept the premise that the combination of appropriate CS knowledge and intelligence increases the likelihood that a candidate will successfully answer the question, then the fact that they answered the question successfully increases proportionally the probability that the candidate has sufficient CS knowledge and IQ (good ol' Bayes' theorem). This then increases the probability that the candidate will be successful on the job (citation in ancestor comment). That was a long winded way of justifying these types of questions as a positive filter.
I didn't say they're not a positive filter; just that they're a black and white positive filter.
Usually, hiring isn't about "passing or failing" candidates; it's about ranking candidates relative to one-another, and hiring the "best" ones (under whatever utility-function you think will help your business succeed.)
And putting people into "really impressed me in solving a puzzle" and "did not really impress me in solving a puzzle" does not much help in ranking people. It just gives you two buckets, one full of people who were both intelligent and lucky enough to get that particular trick that day; and one full of people who might or might not be intelligent--more intelligent than the people in the other bucket even--but who didn't get the trick that day. Personally, I'd rather not allow "luck" into my hiring process if at all possible.[1]
And this isn't so hard! I don't see why there is such a strong digging-in against the notion of changing away from "one or two long puzzles, that each give a single bit of informational entropy about the candidate's capabilities" to "hundreds of trivial, short puzzles, that together give a strong, statistically-sound measure of the candidate's capabilities relative to the other candidates measured."
I imagine the answer is that "the long-form questions give me a chance to get to know the candidate better, and see their [cognitive, and social] approach to problem-solving." But the OP (this guy: http://news.ycombinator.com/item?id=5264735) never mentioned knowing someone's approach to problem-solving as a dependent variable for their workplace success. IQ tests--relatively ranking someone's general problem-solving ability--is a dependent variable for workplace success. Work samples--showing, technically, that you can actually do the job as asked--is a dependent variable for workplace success. That's it. Rely on those, not your gut.
---
But, if I may go on a bit of a rant here...
This implies that convincing your interviewer that you'll be a nice guy who stays on-task, doesn't complain about problems, works long hours because they're "dedicated to the company", will talk down their successes (that is, be "humble") and won't demand fair compensation for their abilities, and is otherwise the "Agreeable, Conscientious"[2] kind of person that schools are expected to produce[3] and HR departments want to consume... is not a dependent variable for workplace success. There's no correlation. People can be great workers and produce their best work with their boss hating them and their coworkers thinking they're an arrogant jerk. And people will not just put up with them--but even approve of their behavior--if they get the job done better than someone who isn't like this. (Anyone who's ever watched House will probably grasp the concept intuitively.)
In fact, in software development at least, I suspect that this "not submissive to workplace domination" attitude can even be positively correlated with success--it's what gives people a sense of ownership and responsibility over project components, makes them passionate about making things right instead of just working, getting coworkers to refactor their own APIs so it doesn't hurt to consume them, etc. And I think that at least some software shops know this--it's, as far as I can tell, the original meaning (before it got diluted to meaninglessness) of looking for a "rockstar" developer. A clearer term might be developer primadonna.
But these types of people--unless they're lying--interview very badly, since, implicitly, an interview is an hour-long test of meaningless workplace submission. This probably isn't a bad thing--if you have the type of organization heavily infected with stress addiction[4], then this type of person--who will avoid the gametalk[5] that powers reciprocal dopamine transactions--won't fit in well anyway.
But if you're looking to build an organization full of people who care more about winning[6] than about "being a company", my advice is: skip the interviews. Just hand them the IQ+work sample test, send them into a room, and get them to do it. Maybe we could even turn this process into some sort of widely-recognized certification [a developer's journeyman certificate, basically] so the interviewed would only have to put up with it once. That would reduce the hiring process, simply, to "let's go out for a pint." And that sounds, in a healthy industry, like how things should be.
---
[1] For the same reason, I don't want to know what school you went to; luck [of parentage] significantly determines what schools you'll naturally end up at, with or without trying, but when people know what school you went to, a peculiar kind of nepotism/cronyism appears, with Harvard interviewers hiring Harvard graduates even if the Ohio State University candidate is otherwise equivalent, etc.
I appreciate your comment. I didn't realize you were actually advocating giving a bunch of small programming questions in place of one or two involved programming questions. I totally agree that this is a far superior way to test programming ability. Actually developing such questions would be rather hard I think, it seems it would be very easy to degenerate to trivia. Perhaps a set of questions similiar to that Quixey one minute programming challenge that was posted here a while ago (http://challenge.quixey.com/)?
I actually have a very ambitious idea to completely disrupt tech hiring. You have echoed much of my rationale. I don't want to go into detail, but the premise is basically to remove ego from the hiring process and make it far more statistically sound. Of course, this is also the very reason would very likely fail. Everyone loves to employ their pet "interviewing hack" to find world-class developers (who happen to be a mirror image of themselves). Most people will not give that up lightly.
The "loss of confidence" part, I don't see why that shouldn't be taken into account. If a candidate can't recover from failing, they're not likely going to do the same when they fail in their job. It also may indicate that the candidate is not teachable if they can't understand or have a hard time with your hint. Hints are a good way to progress an interview.
It isn't that people are insecure about their programming skill and intelligence. It's that they are irritated by tests which don't measure what they're supposed to. If any insecurity is involved, it's about ability to ace unrepresentative tests ranging from trivial nonsense to outright hazing.
You advise readers to "just get better at [the tests]". That's practical. But it goes right back to the problem, that many of the tests mostly index whether you prepared for that test.
Everyone talks about how many applicants are terrible at programming, or how many employees are mediocre. But at least offline, we rarely mention the complementary problem, that many companies are terrible at interviewing and hiring.
We talk as if being on the other side of that desk, in a position of power, means you are automatically good at interviewing. It isn't so. When a company treats candidates abusively - or takes bad hires who are extra slick, cheap, submissive, or similar to themselves - that isn't because of bad applicants. It's a failure of that company.
We can recognize that a defensive and condescending attitude is toxic in a programmer. But the same attitude in an interviewer is completely accepted. They're on top, applicants are on bottom.
> It's that they are irritated by tests which don't measure what they're supposed to
I disagree that programming puzzles don't measure what they're supposed to. Programming puzzles are a microcosm of what programming entails day-to-day. They test the same exact mental faculties. Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request. Doing these type of "low level" tests then allow you to include components that do measure one's mental flexibility and creativity, i.e. intelligence.
Sure, we can give a candidate a test that exactly measures what they'll be doing on the job. But what if the job changes? What if we need them to use a language or a framework outside of their comfort zone? This is the reason these tests are not specific to particular tools. You want a programmer that has the intelligence and command of computer science concepts to quickly jump between various tools with little effort. Comparing strings and reversing linked lists do absolutely measure this capability. No, its not testing whether you memorized how to do these things, its testing whether you have the mental flexibility to figure it out on the spot.
Comparing strings and reversing linked lists do absolutely measure this capability.
This depends entirely on the quality if the test. I suspect for many of these ad-hoc tests thrown together for an interview, the quality will be low and the overlap with other tests high, meaning they can easily be passed by someone who googled 'top 100 programming interview questions' and read through them.
Comparing strings or reversing linked lists are not actually things that most programmers would/should ever do themselves or even nowadays bother to learn or remember outside of a CS course, because the code has already been written in every standard library, but they are things that programmers might still be required to learn in order to pass interview tests, and you would thus be testing at many your candidates on whether they had memorised a few of the problems likely to come up, not on if they could work them out on the spot.
Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request.
I don't know what strcmp is in the way that you're using it. Does that make me a bad web developer?
Something orthogonal to this came up recently on Reddit, and it was observed that in the real world (anecdotally), 95% of professional programmers don't know what the Lyskov Substitution Principle is.
I think that most interviewers will happily give you the definition of strcmp if you don't already know it. If you're not a bad web developer, you'll happily come up with some code on a whiteboard that meets the definition.
Anecdotally, perhaps, 95% of professional programmers don't know the definition of the words, "Lyskov Substitution Principle". That's not the same as not knowing what it is.
Two weeks ago I interviewed at a subsidiary of a major web property, and "[w]hat is the Lyskov Substitution Principle?" was on a written test they gave me.
I would have told them that I had no idea what that was, but then told them what the "Liskov Substitution Principle" (named after Barbara Liskov of MIT) was. I mean seriously, if a company can't spell, do you want to work there? :-)
I think this is basically true. But a point worth making is that a "puzzle" problem is a much heavier thing than lightweight tests like FizzBuzz. If you hand someone a puzzle you're pretty much guaranteed (absent the rare, truly gifted candidate) to watch them struggle at a whiteboard for half an hour. FizzBuzz (or "reverse words in a sentence", or whatever) will tell you the same thing in a minute or two.
Basically, the Venn diagram regions for "can code fluently" and "can solve puzzles" have a ton of overlap. So test the first one, not the second.
You make a good point. The question is how much of an overlap is there? Is there value in adding a component of raw intelligence to the question (say, applying a data structure in a unique and non-obvious way)? It is reasonable to expect that the act of programming fluently entails a sufficient level of IQ such that there is nothing more to be learned from attempting to craft a question that tests intelligence directly?
Two bits of information could potentially answer this: does intelligence as a predictor of future job success drop off past a certain level? Also, what does the distribution of IQ look like for those who are fluent coders?
I would consider myself a very good programmer compared to the hundreds I've worked directly with - think good quality, clean, well tested code delivered on time that delivers outsized value in production.
I am however useless at these mental gymnastic and logic puzzle things.
Having a high degree of programming skill is weakly correlated with these kind of interview questions, if that.
> most people (even around here) are insecure about their programming skill, and more-so their intelligence.
> Instead of writing endless blog posts and internet comments about how useless these tests are ... just get better at them! You'll be better off for it.
This doesn't hang together. Will getting "better" at programming tests & logic puzzles make you a better programmer? Make you more intelligent?
If not, doesn't that illustrate precisely why people seem to have a problem with these tests? Namely, that they are primarily a measure of preparation & studying for the exam?
It depends on your strategy for "getting better at these types of problems". You can cram and memorize all the algorithms you can, and that may allow you to pass some filters that you otherwise would not have. This strategy has its uses: besides being offered more jobs, you now have knowledge of these algorithms in your head and thus more tools in your toolbox.
Ideally, the candidate would learn about these "exotic" data structures, understand their properties, learn the cases where they are useful, and most importantly develop mental heuristics to recognize patterns of problems and appropriate classes of solutions. This will also help you pass these filters, and in the process will absolutely make you a much better programmer.
Speaking for myself, I have purposely avoided studying specifically for interviews. My reasoning was that by relying on my existing knowledge + logic, I would give my interviewers better insight into my thought process and approach to problem solving.
If the benefits of spending time with this material are as significant as you say, perhaps I will revise my strategy (or, lack thereof).
Reading the summary you've linked to, isn't the implication for this thread that certain puzzle type questions, if they do a reasonable job of measuring general intelligence, are useful predictors of job performance? (With the big caveat that in the US they are potentially illegal.)
The popular response here of "I'm never going to solve these puzzles in my day to day" seems to wholly miss the point.
The claims about U.S. employment testing are completely wrong. This is how it works: there is an unconstitutional insurgency (called an antidiscrimination agency) that counts the number of negros on your staff, and if you come up short, they steal from you to prove how badass their homies are.
They generally leave high-tech alone. Firstly because high-tech will hire a pitch black Indian teenager with an outlandish accent if he can code. Secondly because there are not enough negro programmers from the projects to fill a bus.
In the unlikely event they do come after you for IQ testing, you what the large companies do: keep right on IQ testing, and appoint a director of racial sensitivity. His job is to use "sensitivity" and "nuance" to balance out the staff's "intangible factors". That's Newspeak for running the negro quota and making sure the quotees are selected for a pleasant personality and any sort of useful talent whatsoever.
The best questions to ask are the ones that pertain to actual work.
Think of a problem you had before and solved or are attempting to solve. Ask the person being interviewed how they'd handle the problem. Do they have good process, intuition, and problem solving skills? Are they clear about communicating their intentions and explaining their decisions?
This isn't a trick - it's what the person is going to be doing there, in and out, every day.
Completely agree. I recently had an interview where I was asked to solve a job scheduling type problem. So I see the potential graph, remember topological sort and bingo!
Right after that the interviewer told me they have a lot of code that manages assets that have priorities between each other and that this is the kind of problem they solve regularly.
It was a good question: it wasn't too hard as to take hours to solve, it was a decent test of how I was able to choose an appropriate representation for the data and it applied to a real world problem they solve.
I consider myself a proficient developer, however, in situations where I'm required to solve an obscure puzzle while someone is staring at me, generally makes me uncomfortable...no matter how easy the question. I interviewed for an internship position at Microsoft, where I was required to write some bizarre code on graph paper while the interviewer literally stood over my shoulder. Horribly uncomfortable, I barely made any progress, and was subsequently denied a position (obviously). After the interview, I went home and solved the question in a matter of minutes, wrote the code, optimized, and fully tested it. While I choked during the interview, I saw fellow classmates fly off to Redmond...classmates that I knew weren't better developers than I was, simply because I had worked with many of them on projects several times. That's when I knew this whole puzzle interview stuff was nonsense.
I have a similar story with Microsoft when I interviewed with them my senior year also. I feel like I nailed everything except for the puzzles, and also got denied. My favorite one I "failed" on was:
Given a node of a linked list, delete that node. The expected answer: Copy the next node's contents into the given node. Then delete the next node. Just simply never occurred to me to use a memory copy in a linked list. And this is despite the fact that I could immediately reason about the solution he gave me and point out a flaw in it (deleting the last node is impossible).
Great point, and not something I had thought of. Just makes the "expected" solution even more odd... If you're exposing nodes enough to get an operation to delete a particular node with no context, then the given solution just invalidated a completely different node, but not the one you asked it to delete. No, the one you asked it to delete is still valid, just with different data associated.
1. It's not a normal operation for a linked list. A normal linked list will expose operations to deal with items, not nodes.
2. There was an expected answer, and that answer was something other than the normal linked list delete operation of "traverse the list to find the node containing the item, while also keeping a trailing pointer to the previous item so that you can remove that item's node."
Even if you are exposing nodes, and you do implement the given operation, then TheCoelacanth is absolutely correct that that operation just invalidated all references to the node after the one you thought you had just deleted, but not any references to the node you thought you had deleted. (Though those references are now pointing to a different data instance.)
Assuming (for arguments sake) out of everyone who applied you where the most ideal.
In that case since you were not hired, either you or the people responsible for hiring made one or several mistakes. Most likely it was multiple mistakes on both sides.
None of those mistakes were "ask a potential hire to do a simple demonstration of ability". Instead it was mistakes (on your part) like getting nervous and choking (something everyone does)...and mistakes on their part like not making the candidate feel relaxed, not providing enough guidance, and placing too much importance on one aspect of the interview...etc
As someone that just went through a whole slew of interviews I would say I agree with this to some extent. I'm a frontend developer and while CS principals are important, asking questions that are not practical to everyday development, are counter-productive and do not give you a view into how that person tackles a real problem.
Things like "write a function that performs merge sort", are bullshit because you would never do this in real life, largely because languages have sort methods that are already extremely performant. Plus a lot of languages will already be using some variant of merge sort and exposing it as a "sort" method.
If you are given a problem like "create a tabbed news component" or you're asked to do a small project (nothing longer than an hour), I think you can still get a good understanding of how people solve problems. This is less stressful, less "gotcha" mentality, and a fair approach.
Merge sort is a bad example - it can actually be useful in real life, because the library sort functions are limited to what will fit in memory while merge sort is not. It's also simple enough to keep in your head without needing to look it up. I've had to code it up on the job more than once.
Shameless plug: I've been working on this : http://initialround.com to allow applicants to complete an interview at home and in a browser, to eliminate the pressure of the "quizzing cage". I'd be interested to hear people's take on this approach.
My current position did something similar with an existing FiddleJS they had. When it was time for the initial screening, we spoke on the phone while I forked the Fiddle and "whiteboarded" in real-time. I thought it was great and I urge you to pursue your project further.
Thanks! I've done similar things with online text editors, and wanted the ability to compare applicants side-by-side. I was also thinking this might be useful for recruiters who don't necessarily know the code, but just want to pick questions from a preset list to get started.
[I] got asked how to do something in JavaScript on a white board. The specifics are vague, but it’s crystal clear how stupid it made me feel and how little it had to do with the actual job.
So did I. (Well, it was C then, but same idea.) The prospect of scrawling code on an inapplicable medium while people stared at every move is, of course, irritating.
Instead of going to the whiteboard, I pulled out my notebook computer, fired up a code editor, and told the several guys present to continue the interview _while_ I worked on the challenge. Time was saved by multitasking, I did the work in a sensible normal manner, everyone was comfortable with the situation, and they were happy with the resulting code.
Testing with excessively clever tests and puzzles will attract a mindset of creating more complex code and than needed. Architecture a problem in a way that solve the problem elegantly but remains open and flexible is far more a tougher challenge than an algorithmic challenge.
There's a fine line between testing a hire to make sure someone's behind the steering wheel vs. communicating that you want things solved in interesting ways.
I've been interviewing pretty much non-stop since last April. I feel like I've become somewhat of an expert when it comes to interviewing processes.
First off, as I've become a better programmer, I feel this fact is best shown in my opinions. When I first started out, if you had asked me my opinion of PHP vs Python, I would not have been able to say anything coherent. Now that I've worked with both technologies, my opinion is much more coherent.
Instead of asking puzzle questions, ask candidates their opinion. "What is your opinion of Mongodb?" I've spent enough time in the trenches with Mongo to have quite an opinion of that technology. Same with Postgres, Django, Javascript, Coffeescript, etc.
I think hands down, the worse type of interview questions are the ones where you're told to solve a problem, but you can only solve the problem in a certain way.
Its become a interview idiom for me. One recently was to reverse the words in a string. In comes "This is a string", out comes "string a is This". No problem. In python this can be done with one line:
the_string.split()[::-1]
It takes me 5 seconds to write that out. The interviewer telle me "good job", then he asks me to do it again, but this time to do it without using any standard library functions such as split or reverse.
At this point, I can almost with 100% certainty that this company will be a terrible place to work. It wouldn't piss me off it it wasn't for the fact that like 90% of all interviews I do end up like this.
I've through about this a lot, and what I've come up with for an explanation as to why I can;t do these types of questions is because my algorithm writing process is very subconscious. When I'm writing code and I come to a problem that requires a lot of thinking, I usually stop, do something else and let the problem float around for a bit in my subconscious. I got my best ideas when I'm in the shower, on my bike riding around time, even while reading a news article. When people are watching me (especially strange people I don't even know) I don't do my best thinking. At this point I'd do anything to be able to think in front of people. Because its keeping me from being able to get jobs.
I think you're wrong about the 100% bad place to work just because someone asked you to reverse a string without using library functions.
I would think your computer science education would have prepared you to plan out such an algorithm. Sadly, its pretty had for a place to confirm you have subconscious skills by asking a question and then waiting for you to come back the next day with the answer once it popped into your head.
First off, I'm self taught, I don't have any computer science education. Secondly, it wouldn't be too hard to give me an actual problem that the company is actually having, and letting me think about it for a few days, then come back with my solution. Why must all interview problems be made up?
The question is not hard if you are allowed to use extra memory: just tokenize into words, put them into array and reverse, then join them back into a new string.
It is harder to do it in place. I actually did not believe it was possible until looking up the answer on StackExchange. That solution involves a "trick" which one might not figure out under pressure.
You mean ' '.join(the_string.split()[::-1]). Which is a totally reasonable thing to do, but doesn't exercise the same kind of thinking as doing it letter by letter. Can you solve it letter by letter, even now that you're not in an interview?
Doesn't exercise the same kind of thinking as doing it letter by letter? So what?
Solving this problem in haskell doesn't show the same kind of thinking as solving it in PHP, whats your point?
You basically saying you have to solve this problem they way I say you have to. Thats a sure-fire way to tell if the company is worth working for or not.
My point is that there's a difference between the knowledge of how to call a library function and the knowledge of how to write one. It appears that you lack both.
To change it up a little bit, has anyone here been asked simple debugging questions? I've interviewed at numerous places and I've never been asked the question why a certain piece of code is breaking or how to fix it. The questions could be designed to be self-contained and just complex enough to get a feel for how the candidate thinks. My consulting job usually takes me to places where a project is in trouble or failing. It would seem that quickly understanding broken foreign code would be a huge asset. The closest question I've been asked is: "Here's a Singleton implementation in Java. What can go wrong?"
I tend to think debugging is a pretty useful indicator of how well someone will do - the best developers I've known were really good at debugging, and I've seen very few people be good at debugging and bad at programming - so I usually try to incorporate a debugging question or two into my interviews.
I like to ask debugging type questions in sort of a role-play format. Usually I'll start with a symptom that an end-user might see and then allow the candidate to ask me for any additional information (as either the end user or a system admin giving them log files and other things) to help them narrow down what the problem might be.
It is a little tricky to make it possible for them to make progress without knowing anything about the hypothetical code base, but I like it because I think troubleshooting problems is a decent indicator of someone's skill and knowledge: Do they have a good approach to narrowing down where the problem is? Do they know what kind of information to ask for (for Java devs, I like to ask questions that lead them towards asking for thread dumps or heap dumps), which shows mastery of a language/platform? Do they ask good questions in general?
I've also seen people use printed stack traces and ask the candidate walk through what the stack trace is telling them and what the possible root causes might be - this is more useful when you want to test someone's knowledge of a specific framework.
Handing someone an application in a broken state and watching them work through the process of debugging it would seem like it would be interesting, but I've never had the time to prepare this for an interview.
Basically, most of the questions I see asked in interviews are contained in Levitin's algorithms book. The greatest potential for someone to excel at those trick interviews is to be fresh out of college.
I walked out of my first interviews, because 'I do not like programming games.' The first was a 4d matrix puzzle with more mathematical/theory roots than programming/logic.
My third interview started with the above quote and I was hired on the spot -- I believe they took it as a rebellion of every fresh out of college kid wanting to program video games.
I have a long-winded blog post about this very subject awaiting some polish.
I've done my fair share of interviews. I still don't like them. I've narrowed it down to basically two things:
1. There is a lot of highly subjective, mystical advice about what "works," in a technical interview and what doesn't. Big problems, small problems, trivia, puzzles, white-boards, no white-boards... hardly any of it is thoroughly studied and built on solid theory. It's all conjecture and personal bias.
2. How the candidate is feeling and where their head is at on the day of the interview matters. I can go on about cache alignments, bus errors, segmentation faults, C++'s lack of order in evaluating function arguments, why template class declarations must exist in the same compilation unit as the definition, why CLOS is amazing and the computation model of recursion is brilliant (or at least why I think so). Yet on a bad day I can (and have) botched perfectly trivial, benign problems.
However at the end of the day you need some sort of process. I just think that the current practices are not good enough. It seems to me that companies are probably turning away perfectly fine candidates without realizing it. There's pros and cons to every approach I suppose. But I still think trivial problems are dumb.
I agree that the hiring situation in our industry is less than ideal. The irony is that, for all of the data we use and tools we make for others, our hiring practices often rely on few tools, limited data, and lots of gut feelings about what is important.
Take the process of advertising a position and moving candidates through the hiring process - the tools there are almost universally crappy and haven't made a lot of progress in the last decade.
Also, as far as I can tell, very few companies attempt to record any kind of data about how candidates are hired and how they subsequently performed on the job (even tracking job performance is a tough problem) - and it certainly doesn't exist industry-wide.
There is a ton of money in this area - why hasn't there been much progress? Is it because it is an unsexy problem, or is it because it is genuinely impossible?
I agree wholeheartedly here. I know many developers (myself included) who would fail a FizzBuzz test or any instance where they're asked to write code on a whiteboard and yet can produce some really clean, efficient and above all well-written code. How about instead of asking people to write on a whiteboard or do Computer Science 101 tests, how about you give them a computer, a problem and ask them to solve it?
Don't forget it's not always how smart they are, it's their work ethic and whether or not they'll be a good fit for your team. Personalities that meld well with already hired employees are an essential must, no point hiring an introvert for a position when the team are extremely social people who have Friday afternoon beers.
Here is a tip: hiring a developer? Ask them to write a blog application, a task management system or a real world problem they will encounter if hired by your company not some ridiculous test that could stop you hiring a great developer. I consider myself to be pretty good, but I don't have a CS degree, I suck at mathematics and can only do simple math, yet I am able to produce exceptionally great code especially in instances where deadlines are tight and sometimes unreasonable.
I'm sympathetic to this - I don't have a CS degree and most of my last jobs have been through references from people I've worked with before, so I haven't had to answer any tough programming questions in an interview in quite a while. I'm sure if I tried to interview at Google, Amazon, Microsoft, etc, it would be a little tough for me.
At the same time - any reasonably experienced developer should be able to whiteboard something as simple as fizzbuzz without a problem. The more complex the problems, the more you worry about the whiteboard effect, but you should be able to at least walk through your thought process and come up with some reasonable attempt.
Having a candidate build something outside of the interview process probably isn't bad (as long as you follow up by going through the code with them), but I think something like a blog application is both too big (unless you really constrain the feature set) and too simple to work in some cases. Something that requires a bit more thought and a bit less time would be better, IMHO.
This is a complicated issue, but in my opinion the google/facebook style of interview is the one that gives you less false positives, even though it gives tons of false negatives. In other words, very few people who can do it are bad coders (false positives), but many good coders can't do it (false negatives).
So, as companies like Google and Facebook have so many applications, they're much more concerned about false positives than false negatives.
Just my 2 cents on this issue - writing code on a whiteboard isn't always about the code you write.
It's not about the actual code you produce in the interview, it's about talking through your thought process, your ability to reason while under pressure, and your attitude of whether or not "this is beneath me - wah!".
I've been in several interviews where I had to write code. I made some mistakes, some things I plain out didn't know (for my first job, I had no clue how to write any JavaScript, and I explained that it was just something I didn't know how to do, but that I'd be willing to learn it as required). I still got hired from all of them. So, if you think coding tests during interviews are about producing flawless code, they're not. They're to see how you think - and that is profoundly important as a developer.
I would also argue that a developer's existing skills are the least important factor when deciding whether or not to hire someone. Every developer works as part of a team and has to collaborate and communicate with others in that team. If you want to hire a developer, you need to see how well they communicate complex technical problems to non-technical people.
I used to give interview candidates a little background lecture on our problem domain then ask them for input on whatever I was working on that week. E.g. If you were trying to parallelize this code, how would you structure it? I never asked fizzbuzz style coding questions, because I think they're stupid and I probably couldn't code a linked list on a white board (I rarely code on whiteboards you see).
I probe for weakness in 5-6 different areas of computer science in the phone screen - keeps me from bringing in weak candidates.
Then in the on-site, I give them my Macbook and ask them to solve a few simple problems with some flat files I conjured in a language of their choice that's installed on that machine (C/C++/Java/Python/Ruby/PHP/Perl/shell/etc). You get an idea of how they solve problems, think about solutions, fix bugs, avoid common errors, and what they look up on the web (I let them use a browser to lookup apis and so forth - it's a good way to weed out "cut and paste" programmers). The people who do well finish quickly and we chat about the company with the remaining time - the people who do poorly use almost all the time on the first problem, make a bunch of mistakes, and usually don't understand the proper data structure for the job.
There are a few programmers that have a proclivity towards "clean, perfect" solutions, but FizzBuzz is ugly.
Making FizzBuzz work at all is just stupidly simple; but the moment you ask yourself about the little details, you notice that you either have to make an if tree, a bunch of if-elseif's, or some other hinky thing that ruins the "purity" of this simple little program.
Alot of people take the perspective that if you go
if(i % 15) print "fizzbuzz";
else if(i % 5) print "fizz"; //or was it buzz?
else if(i % 3) print "buzz"; //or was it fizz?
else print i;
then you've found an optimal solution and you can wash your hands of it, until you think to yourself: "but an if tree would use less comparisons! Should I be readable? Or take the efficient route? But if I'm thinking about efficiency. wouldn't a lookup be faster and simpler?"
I suspect when presenting a problem like this, the interviewer filters out two kinds of people: the people who "can't code", and the people who are obsessive.
The entire point of FizzBuzz is to filter out the people that say they can code but clearly cannot. This is a surprisingly high number of applicants, unfortunately.
I have sometimes in the past been obsessive to the degree you're describing, and the consequence was that, practically speaking, I couldn't do the thing I was obsessing about.
The entire point is "is this person capable of the most basic fundamentals of programming (conditionals, looping). That's it. We don't filter out obsessive people with it, they demonstrate that they understand conditionals and looping, what's the problem?
Someone who is obsessive wouldn't like their off the cuff solutions to FizzBuzz. They would think of a solution and stop themselves from writing anything at all.
How would you tell the difference between someone who doesn't want to write their first three solutions, to someone who can't think of even one?
Microsoft's "implement strcmp without the use of libraries" doesn't have this property. strcmp has an easy and clean solution.
FizzBuzz doesn't filter anyone, the people using it do. It is not difficult to explain the purpose of the exercise, and that easily resolves the problem for people looking for "clean" solutions.
Lots of practical problems don't have clean, elegant solutions. Maybe part of it will be a beautiful algorithm, but some of the logic will probably be fairly clunky. You have to be able to solve those problems too.
Yes. I think you are seriously underestimating the intellectual capacity of those people. Telling them "it doesn't matter how nice it is, or clean it is, or generic, or reusable, or maintainable, it is just to see if you can do basic introductory level code" solves the problem. Anyone who still stands there like a moron writing nothing is useless as a developer, regardless of the reason they stand there.
I was invited to a company for a game development position. One part of the interview consisted of a fairly challenging mathematical/algorithmic problem. It wasn't impossible, but it's certainly not something I can do in ten minutes at a white board with some people taking notes about my every movement.
The other part was being shown intentionally messy and broken code. Not code that'd fail to compile (although it sure looked like it would), but something thrown together in a completely insane manner that I couldn't possibly make sense of in 15 minutes. Much of it boiled down to, "What would happen if you call this function and use this variable before you actually define them and then do x, y and z?", and the way this language did it was quite different from other languages. I mean, yes, I did learn a lot about the language from that interview, but I'd really hope that's not the kind of code they'd be throwing at me or expecting me to write at their company.
Plenty of the gigs I've had have been mostly trying to unscramble really confusing code that mostly-works. If my codebase has a bunch of strange crap in it that I need fixed, I certainly will bring examples of it to the interview.
When I interview others, I use a couple of basic whiteboard show-me pseudocode questions (how would you reverse a string, how does a linked list work, give me an example of a callback). It seems to me that if they know their stuff, they can answer those. If someone can't get through any of them, or at least have a valient stab at it, chances are they don't have the background I want.
By the same token, I've been interviewed myself where I've been given weird shit questions like others have mentioned; not the blender one, but shit along that line. That's nothing I personally do without some serious contemplation, not to be achieved in 5 minutes in an interview. Anyone who does so likely knew the answer ahead of time or is extremely puzzle-oriented. I think the real test is to examine the puzzle-solving processes of the candidate, but as a rule I don't think much of those kinds of questions.
Ok... I was under the impression that 37Signals hired mainly remote workers. Could this account for the lack of a white board?
Personally, I use a whiteboard when hiring. It's not the deciding factor... but it's a part of the bigger picture. As part of a small start up I need to know right then and there if they know what I need them to know.
I think using a whiteboard to test someone's design knowledge is probably a good idea. I like to draw diagrams on a whiteboard when I'm designing a solution to something. However, using a whiteboard for a CODING exercise is stupid. When was the last time you wrote any significant amount of code on a whiteboard? Give someone a laptop if you want to see their coding skills. Even better, have them pair-program with one of your developers.
Even though there are plenty of online whiteboarding tools, I'm pretty sure 37signals can afford to fly someone in for an interview. To say the lack of asking whiteboarding questions because they work remotely is absurd.
Coming up with good questions to ask in an interview is an effort that the hiring team needs to invest in. This involves a top down reflection of the job along with a vision for the job. It requires to ask him/her self before hand "What sort of tasks is this hire going to be doing" , " What language skills does he need" , " Does he need to reinvent a sorting algorithm to get this job " .
Well this rarely happens , to start of folks (generally) tasked with the hiring process are not the right "vision" folks. They are highly task oriented individuals aka MS Project specialists.
A real test would have to encompass among many things.
- Do they are ask the right questions ?
- Are they capable of picking up new languages, ideas , technologies while defending or questioning changes in an articulate yet respectable demeanor.
- how they handle things when they get to a wall.
That's what I was discussing with my friend a few days ago. He had an interview with Google last month and it took him a month or two to prepare for that. he did not pass the interview, but he got an interview with Amazon this month. He was busy with other things and he could not study for the interview. He mentioned that he cannot remember most of the things he prepared and read for the last interview. I am forgetful too, and I guess 90% of people forget details about algorithms and how to answer a puzzle very fast if they don't use it everyday.
So the question is what is the value of a skill or knowledge that fades away after a month or two? I know people who got hired by Google or other big companies that forgot all the information they acquired before the interview. Cause most of it is useless for the actual job. Many logic questions or puzzles are well known and people tend to know the answer before the interview.
If I had to interview someone; I would assign him a project from the company that is part the of the job he is going to do, leave him alone in a room with a computer for a time period I expect someone to finish that specific project. Finally, I come back to see the out come.
It is good for the applicant, as there is no pressure on him (except some time constraints maybe which is not a big deal), I do not put him on the spot and he does not have to deal with his boss while answering to a question. He does not have to dig unrelated information before the interview, and he can show off his programming skills if he has any.
It is good for the company, as I do not have to spend a day interviewing and prepare for that, I will know right away from the code if the guy is a good fit or not, I can see his programming skills, I know ahead of time how long in average it takes for an internal person to do the job, so I can measure his performance relatively. I do not even have to evaluate the code in front of him, I can invite multiple candidates at the same time, put them in separate rooms. Get the results, evaluate them and invite them for the second round if I liked their code.
Well the good news is this problem should be solved soon since we spend so much time focused on it.
I don't care for puzzles. Just conversations. I skim resumes looking for glimpses into personality more than looking for skill sets. I have a guy that went to Yale on my team. I didn't even know that until he mentioned it the other day, yet it was on his resume. I missed it because I don't care where you went. I don't care how fast you can solve a puzzle. It's put up or shut up and show me what you've built when you come here. If you can do that in an interview there's a great chance I'll hire you. The last two hires whipped out apps they'd built and then we went over how they accomplished it. To me that's more valuable than any other process, having a portfolio or sample ready when you show up.
I was interviewing with Microsoft last semester for a SE internship. At the end of the interview with a senior engineer, we were just chatting about various topics. This is one topic that we talked about. His response was something along these lines:
"Of course we do not write code on whiteboard everyday. Of course we do not write code to detect if a graph is acyclic (my interview question) on a daily basis. Of course we do not put our engineers on the spot with these types of questions. However, when the time comes, may it be only once in 10 years, we know that we have hired the right people because these questions show your thinking abilities - not if you know the answer or not."
Btw, I didn't get an offer, but I'm going to Amazon where I had very similar interview process as Microsoft.
Ok, this is true story and has nothing to do with my credibility. Few days ago glenwood systems, a core product developing company conducted a recruitement drive in my college.
There was me, a coding geek with all my passion for one thing, and there was him, who is a puzzle freak.
They selected him past the first two rounds full of puzzles and math, finally rejected him knowing his programming incompetence. And they missed someone who really could help. me.
There are a ton of books out there teaching you how to crack interview puzzles and programming riddles. There's no book telling you to get your hands dirty with projects and code. The author is right. Hire the one who did things, not the one who learnt to crack puzzles.
Not employed people for years, but I never put much value on ultimate skills. I used to think long term. So, give me the the right human being, with a reasonable level of knowledge, and very quickly they will be more than good enough. A little more time and for me, they become pretty much perfect.
I want the right personality. The ultimate skills will follow. So, I just chat to candidates and employ those who I feel I and my people can work with. I never left it so late that we needed instant skills. To me that would be poor management.
Prior to that, I have tested and done many of the things people cite here, but most of the time I got awful employees who turned out to be good at doing interviews.
I won't present a résumé, a github link, a linked in profile, or do nonsense tests. I always send a full web-application I built for code review, do a technical demonstration and provide sample work on their codebase. Any other route is a waste of time.
Programming is about problem solving. What you want in a programmer is someone who knows how to approach solving a problem in a structured repeatable manner.
What getting someone to solve small problems like FizzBuzz in the interview gives you an idea of how then approach the problem. Can they decompose the problem into smaller chunks ? Can they then refine their solution into a better solution ? Do they know when to stop refining before the solution becomes difficult to understand ? These are the skills you want to asses from a potential programmer.
Programming languages, platforms, frameworks come and go but if you don't know how to approach solving a problem then that knowledge is ultimately worthless.
It's the perfect, super-simple puzzle that can be solved quickly, but let's you see how a programmer might approach a problem, how they like to code, and their go-to language, etc.
It's not like you are trying to stump the programmer. Any programmer with a lick of skill should be able to knock out the FizzBuzz problem in their language of choice.
It's actually kind-of fun the first time you do it.
Personally, I'd watch the programmer's face. If they smiled after realizing the problem is not as clear-cut as it seems, they probably enjoy programming.
Some of the best hiring strategy I've came across is indeed "test driving". Take a challenging problem your team has, slim it down and ask candidate to do some quick prototype during a weekend. No interviews, puzzles or whiteboarding :). If you were going to fly down a candidate on-site for a day of interviews instead, it's going to take same time anyway considering logistical effort on candidate's part.
Test-driving, however, is unfortunately neither scalable nor "leak proof". As soon as one candidate gives away your test drive question, the next candidate could easily cheat away inflating their apparent awesomeness compared to other better candidates.
However it would be incorrect to write down all whiteboarding interviews as "evil". Like any other interviewing techniques, it really depends on how you do it. A good whiteboarding question that allows candidate to use CS fundamentals to solve fairly non-trivial problem that you also needed to solve for your current work is a great question. There are 3 major reason why this doesn't work as expected at some large companies:
1. Interviewers can't think of good whiteboarding question and fall back to commonly asked popular puzzles. These interviewers are also often the ones who have one "favorite" question that they would ask everyone. This is absolutely #1 problem why whiteboarding interviews devolves in to secret puzzle marathon. The best questions that interviewer could ask are actually the ones that they needed to solve themselves as part of their work recently. This keeps questions relevant and refreshed regularly. It also allows to compare candidate with themselves and follow the philosophy of hiring people smarter than you. It's however hard because interviewers themselves needs to continue doing something interesting regularly.
2. Interviewers provide feedback to each other during the loop. It's not coincidence that if 1st interviewer gives "hire", all rest tend to do same at companies where there is no clear policy of not communicating feedback before end of the interview.
3. Interviewers don't follow up candidate response by extending question, building on to next level perhaps open ended scenario, asking auxiliary details such as test cases, complexity etc.
Eh, being able to understand stacks and queues, big-O, etc and problem solving are important things, but otherwise agree that software construction skills are important as well as understanding tradeoffs bottlenecks, networking etc.
At my old company we'd tell interviewees to bring a laptop loaded with there usual dev tools. After a couple of typical interview questions they were then instructed to create a small program using whatever language/tools they'd like. They had "unlimited" time and sat at an empty cubicle. Candidates also has had full access to the Internet. It was amazing how several people could not even complete the simplest program without needing extensive assistance.
I wish this were the case more. I graduated with and electrical engineering degree, but over the years I've spent the bulk of my time coding up DB, big data stats, ML, etc. to help me do my job. Trying to explain I'm more than capable while I falter with simple puzzle based question is an impossible gauntlet that has kept me from even considering a career shift. (Granted I'm a much better analog designer than I am a java code.)
I think the skills you need to throw together a nice RoR app are different from other positions.
I wouldn't want on critical infrastructure code alongside someone who can't reason through the performance implications of their code on the fly. You can waste a hell of a lot of time diagnosing and fixing performance problems because someone built an entire module around an inappropriate data structure.
It's one thing to ask people to right syntactically correct code. It's another thing to ask them to write pseudocode in whatever language they please and understanding their process.
At the end of the day, as Hal Abelson said, computer science is not about computers, or science. It's about being able to formalize process.
Kent Beck tweeted a very fitting comment: 'Programming contest problems shouldn't be algorithms, they should be like "set up continuous deployment of a multi-region Django app on AWS"'
Because this is much more than sysadmin skills involved. You've got to get a scalable application written first. It doesn't need to be complicated, but it needs to be robust and keep data consistent... etc.
Also, this type of skills is badly missing to fresh grads.
Script is code too. But I don't think you get there with plain sysadmin skills. How do you sync up the data between the regions? I have worked on a similar case not long ago, and there were definitely more than just "sysadmin" skills involved. I suppose Kent Beck also went through the same exercise. Pointing to the deficiency of usual contests is a good thing. In any case, if all sysadmins can do continuous deployment, cloud management and worldwide distributed apps, then I've definitely met the wrong ones before.
Comments are closed on the article and I wanted to thank David.
I'm one of those who freeze up like a deer in headlights when asked to do a quiz. But have a degree in CS, have made several enterprise level packages by myself, and have been programming for 31 years -- so am fairly confident that I can program.
(If you need help posting a comment, feel free to use any of these samples: “You make todo lists, you don’t need real software engineers”, “Math is actually really important, you know!”, “Google is worth one gajillion dollars and they use quizzes, so there!”)
All great and well spending 20-40 hours trying someone out to see if they are a good fit, but this is only practical once you are down to a very short list of final candidates.
Nothing is said about how you actually get to this final list and that of course is where the real challenge lies.
Reminds me of this supposed exchange between William Faulkner and Ernest Hemingway.
“He has never been known to use a word that might send a reader to the dictionary.” –William Faulkner (about Ernest Hemingway)
"Poor Faulkner. Does he really think big emotions come from big words?" —Ernest Hemingway (about William Faulkner)
>>Excellence in the grammar is also irrelevant.
It took me a long time to grasp the fact that there are very many intelligent and skilled people who have dismal grammar and spelling skills. It still troubles my mind when I encounter examples of it.
is there any well known data/studies on selection methods for programmers?
I can understand the reasons why not to use algo problems as a filter, but I would have thought simpler forms of intelligence testing would still be quite useful.
I think Token has the longest list of that sort of thing. I'm sure he'll be along shortly to post it ITT. Dig through his comments here if you can't wait. http://news.ycombinator.com/threads?id=tokenadult
Agreed - but I guess if you're 37signals you just attract the right people and you don't have to worry at all. But if you're not then maybe some of these riddles are just good heuristics.
It's too easy to prepare for the common ones. I have a list of them which I share with my students before they graduate. Sometimes we work through them for fun.
These puzzles and IQ tests are as pervasive as they are irrelevant. Nowadays, you even need to pass one to get some government jobs. Here's the one to pass to get a job in the EU agencies:
One of my career goals is to get a programming job without having to show that I can add a leaf to a binary tree through recursion (again).
I don't mean this as flippantly as it might sound. I used to think it was kind of fun and exciting to be at the whiteboard, thinking on my feat, dealing with various curveball data structures and algorithms questions. I certainly always read up on this stuff prior to interviews, and from what I've heard, I'd be advised to do it again if I get an interview at google or something.
But the last time I did this left me depressed. I realized that I've done so much of my work in obscurity that people are still asking me about binary trees. I want to be very clear that I don't blame them! I also started to realize that I have a lot more to offer than my ability to traverse various tree structures, and that the interviewers seem unaware of this. But it's up to me to show them, not on them to just take me at my word.
The last time I went through a massive, all day technical interview, I didn't get the job, partly because I didn't review my data structures and algorithms book (again), but probably also because I came off as cranky and irritated. I was busy that week, probably should have put it off.
Just earlier that week, I had taken several clients from pharma and semiconductor companies through some beta testing of our software, refining the model, getting feedback on use, optimizing the code base so it could give answers more quickly. The company I was interviewing for created supply planning, forecasting, and inventory management software for various large businesses. As a math major with an MS in Industrial Engineering and a background writing software, it was right up my alley.
My interviewers showed no interest in anything other than my ability to code or a couple branches of math. In fact, some of them seemed so inexperienced that I'm not sure they were aware that this kind of experience even exists to be asked about (Oh, you work with clients? Our project managers do that). At lunch, one guy asked me how to swap two integers without creating a third integer. This is after a full morning of technical grilling on math and programming, with a lengthy afternoon to follow.
Fortunately, I'm now in a job where I am encouraged, not just allowed, to contribute to open source, do presentations at conferences, try to build communities, and so forth. I would not consider any job that did not have this element (and now that I have a job like this, I'm not looking anyway). But if I were to look around, I would vastly prefer to be hired for my known and widely used code base than for my ability to detect cycles in linked lists, implement a hashing function, or print out all possible permutations of a string (with no duplicates!).
"The last time I went through a massive, all day technical interview, I didn't get the job, partly because I didn't review my data structures and algorithms book (again)"
Which means you're not using that stuff on a daily basis. Which is fine. You're probably doing other things. Development is more than just data structures and algorithms. Yet, that's all many interviewers want to ask about.
Here's my guess: you probably reviewed that stuff prior to job 1. You got that job, due to short term memory, then you went to work. Now you want to go to job 2 (or 3 or 4). And you find yourself having to "review" material that doesn't stick to short term memory, but I'm sure you know damn well how to find it and interpret it if you needed to. No doubt at all.
I interviewed at several companies about a year ago, and one had a practice that really stood out. I was impressed by how organized it was, how well it seemed to be a good test of a job candidate, and how it gave me many opportunities to show my qualifications. It was a whole day, but they flew me out and put me up in a nice hotel nearby.
The day was broken up into sessions of about 1 hour each (maybe a bit more). In each session it was me and 3-4 other people, but the people rotated, so I got to meet lots of engineers, a PM or two, the hiring manager, etc., and they all got to meet me. A few people showed up more than once.
The first session was basically an introduction. I think it was shorter than the others. No quiz-style questions, but stuff about my background, etc., and opportunities for me to ask questions also.
Next I did a presentation at the whiteboard describing some project I'd worked on. They had told me this would be a part of the day, so I could have prepared (though I didn't). I talked about my recent startup, describing some of the data model and machine learning tricks I'd developed. They asked lots of questions, some to clarify, others about my reasoning, how it would scale, if I'd considered technology X, etc.
In the next session was me at the whiteboard again, but here they described a new feature they'd like to add to their site, and my job was to sketch out a rough solution. So partially this was about my ability to ask clarification questions, and also about my big-picture design skills. Again they asked lots of questions, and once we even got into a specific SQL query (where I opted to use an EXISTS with a correlated sub-query). So this session seemed a lot like the first, but more impromptu on my part and probably more practiced on theirs.
Then they took the whole team out to lunch at a nice Thai restaurant (~15 devs plus the director of engineering and the PM). This part was relaxing and fun. Of course this was still part of the interview, and wouldn't be relaxing for everyone, but still it was a good way to feel out personality fit for both sides. Also there were too many people for me to be the focus of attention, so that was a nice break.
Finally there was one afternoon session where they gave me a laptop with a toy Rails app, and had me add a feature (change a relationship from one-to-one to one-to-many, or something like that). There were two devs watching me code, and I had to do the whole MVC thing: write a migration, tweak the controller, edit the view. I got to talk through the changes as I did them, so they knew my reasoning. It made me smile to slip in a few fancy vim moves; I don't know if they noticed. But it made me realize this was also a test about some basic mechanics, too. They let me choose my editor/IDE, so it was a chance for them to watch whether I knew my tools.
Then there was a final session for follow-up questions etc. This was fairly short I believe.
All along the way I had lots of opportunities to ask questions of my own, which I appreciated.
There were no puzzles, no trivia quizzes. Personally I'd add a FizzBuzz test to the initial phone screen (like Han Solo "They hardly asked me any questions."), but otherwise I plan to completely rip off this template the next time I hire someone myself. Maybe it can help some of you!
I'd never work for a place that gave me a fucking puzzle to solve. I'd probably smack the interviewer upside the head for wasting my time.
The best way to do it is pay the person per task as an independent contractor until you're sure a full time position is what you're both looking for. If they're helpful to you then you keep calling them and if not everybody keeps their dignity, no big deal. Don't humiliate potential employees right off the bat with stupid games.
I'm going to avoid the cultural flamewar around whether companies should do this.
I don't mind it. I've interviewed for enough jobs and had enough acceptances and rejections to learn something: it's not a big deal. If you don't know an API function, you don't know it. The likelihood that you won't get the job because you didn't know that one question is unlikely.
In fact, part of the evaluation is whether you can handle not knowing the answer with grace. The fail-outs are the ones who use their cool or seem to think the question is stupid, not the ones who get it wrong.
Once you realize that you don't need to get all of the answers to pass an interview, it gets easier.
My gripe these days is that places I've interviewed with now that I have a few years of experience are still concerned about big-O notation and a lot of the more academic questions. I feel like I shouldn't have to go back to my college notes to prep for an interview for a job that I'd really be using my current experience for. I think it comes down to other developers not spending time really thinking about the quality of their interview questions. They just do a search for 'programming interview questions' and call it a day.
My most recent interview was with Amazon. I was asked the dumb big-O runtime stuff as well as how I'd count all of the stars in the universe. It pissed me off, but I did like the question I was asked about how I'd design an elevator system and then asked how my solution would scale. That's not too far off from what I'd be doing, albeit in an abstract form.