Let the person code WITHOUT YOU BEING THERE.
Interviews are stressful. The author says "My heart rate jumped" ... his heart rate jumped because his body released a bunch of adrenaline in reaction to a stressful event.
Adrenaline prepares the body for fight or flight, and part of this preparation is re-routing blood to critial systems ... muscles, heart and lungs ... and things like digestion and higher level cortical function are not required for fight or flight.
Programming requires higher level cortical thought - which is disabled during a stressful situation.
Give the person a programming challenge, tell them to code it on a machine (or the whiteboard for the inferior option) and then leave the room and tell them you'll be back in 10 minutes.
Let them relax, then their brain can think.
Pretty much any time I feel like I'm being evaluated by someone, my brain goes into an overly defensive, self-critical mode where I spend more cycles trying not to screw up than I do actually performing. It's generally a disaster.
Strangely, I discovered my cure many years ago when I was able to crank out Mozart's Rondo Alla Turca in front of an entire wedding party. Under just the right level of inebriation, alcohol seems to dampen the self-critical part of my brain just before it starts affecting much else.
Unfortunately, it takes quite a bit of alcohol for this effect to kick in. And sadly for me, this hack is a bit harder to pull off in a job interview situation without coming off like a high-functioning alcoholic. It certainly helps with the phone interviews though...
Also, it is very important to "manage" the process, because the candidate may get stuck and feel frustrated, thus performing worse, or may settle on a sub-optimal solution when they could actually do better. Both these issues can be solved with some interactivity, using hints and probing questions, which cannot be done if you're not in the same room.
But, there is also a large part of the developer population that has the same reaction that the author did to "paper" coding, to any pair programming, including pairing with devs they've worked with for years. Developers in this population, who know that they are in it, generally will decline any job that actually involves large amounts of pair programming so you should reveal that you do lots of pairing early in your hiring pipeline.
If you do not actually do pair programming a lot in practice, this technique is no better, and in some ways worse, than "paper" programming.
My policy on any programming test is that it should be take home, non-timed and non-observed. Unless your environment has real, hard requirements about "how" someone writes code putting fake constraints on that part of the filtering process only biases against potentially great hires.
As an example: I recently applied at Trello. I nailed their programming challenge to be able to submit the resume, and I gave a thorough explanation for my solution in the submission. I got a call from HR and they scheduled an online interview where I would be sharing a code window with the interviewer and I'd have to solve a succession of problems. Shit. All of these teasers were easier than the initial problem I solved to get the interview in the first place, but with a spotlight on me and an implicit time limit that I could feel ticking past with the interviewer's silence, I went down in flames. I have no doubt that I'd be perfectly capable for the job itself, but that doesn't matter because I couldn't dance the right jig.
I have since accepted employment with a company whose interview process cared more about what I have done and what I can do, rather than forcing me to participate in an unnatural process.
I'm not sure where I stand on this, but I've heard many hiring managers complain that great phone and work sample candidates often fizz out entirely when tested on the sample in person - and not due to jitters, but due to e.g. not being able to comment their own code or explain one of their functions.
I will say that I think a "homework" style coding test has always felt like a more fair assessment and been more enjoyable for me personally. I think a homework style work sample is probably one of the best methods for hiring, as long as it 1) won't take longer than an hour or so and 2) is technically exhaustive of typical job functions.
That said, I know brain teasers are hated, and I dislike them too, but I really think most programmers should know standard algorithms and data structures. I don't think the current model of whiteboard coding is perfect, but knowledge of algorithm design is great for optimizing performant code.
I personally have no problem with candidates getting help on submitting their programming problem. I view effective help getting as a fundamental tool in a developers toolkit, so that in and of itself doesn't bother me.
I think you can do an onsite code review, where the candidate presents their code sample, teaches you how to use it, explains the tradeoffs, responds to criticism, etc. This should be just like what you are doing for your "real" code reviews (if you don't do "real" code reviews, how do you know that any of your colleagues understand the code they write?)
Over the years, I've grown a bias against even caring about that second step, because I have found so few people that do well on the programming problem part and then do poorly on the second part due to anything other than stage fright, but I understand I have an extreme position on that.
How would you feel about, say, providing a candidate with a chunk of [source] code and 30 minutes to read through and grok it. Then they can lead a discussion/review about their thoughts on its structure, and so on. This can move in to their preferred implementation, or just focus on consequences and risks etc. Essentially switching positions from the usual stage 2 where you're reading code and leading.
I know reading code is in some ways easier than writing - but to me understanding foreign source is sometimes harder than writing a fresh tool in your own style etc. Being able to distill intent, aim etc.
We are well into the area where it is my biases talking and not any sort of well researched finding, but I don't believe in developers who can turn on/off writing good code. Either a developer writes good code, or they don't. Further, I take getting my next job very seriously. I take hiring my next colleague equally seriously. I'd hope that this seriousness (in combination with me asking for a reasonable amount of work sample) would encourage everyone involved to put their best foot forward.
> How would you feel about, say, providing a candidate with a chunk of [source] code and 30 minutes to read through and grok it.
I have never tried this so don't have any experiential recommendations. My instinct is to think that this would be problematic, not because it is wrong to judge someones ability to learn foreign code, rather because the evaluation step would be very hard to filter for someone who learned foreign code quickly, from someone who is great at talking about stuff generally.
If I wanted to judge someones foreign code ability, I would as part of the code project, expect them to interact with a provided set of foreign code. I'd set explicit boundaries about how much they could or could not change it and then judge the candidate on how their new code interacted with the legacy code as well as what and how they changed the legacy system.
> I know reading code is in some ways easier than writing
I don't think that is in the least bit true. I think evaluating a candidates ability to interact with legacy code is a hugely valuable thing to filter on in most environments, it's just very hard to design a fair and repeatable work sample problem around it.
Do you think developers need any grounding in foundation if it isn't explicitly part of their job description?
Finally, do you administer the homework test via a third party like HackerRank or develop your own problem and simply email it to them?
I personally have found my computer science academic background invaluable in my career, and think that foundational computer science is useful in nearly any software development position. That said, some of the very best developers I've worked with do not have that foundation, so I do not think it should be used as a selection criteria.
I have historically used email for work samples and then fed them into whatever code review system the company is using. If I were setting up a big hiring pipeline right now I would probably due research about outsourcing the mechanics of collection/testing/code review collaboration to a 3rd party. That said, sending candidates through your real workflow can expose problems on your side, which is a nice side benefit.
At the end, there's too much factors you can't uncover before hiring, that at some point more testing in meaningless. My experience is that everybody has a preferred way how to work and a preferred topics, and allowing them to do just then yields most benefits.
If a candidate is unwilling to invest that much time in our process, that is perfectly within their rights and I respect that. But I'm not going to sacrifice my time for them either. If I need to cut time commitments from my hiring pipeline I will do it at the interview component. I would gladly trade 4 hours of work sample time for a half day of onsite interviews for instance.
As far as eliminating obvious misfits, I'd argue that a mechanical suite of tests (I like to provide say 70% of them to the candidate) gets much lower false negatives and much higher true positives than any "simpler" test you can devise.
The Guerrilla guide is a very good starting point btw, but I think it misses the central problem, which is, interviewing is not a reliable way to filter candidates for software development jobs, no matter how you do it.
Sounds weird right? But the thing is, I'm not hiring rocket scientists so I don't need the best of the best, just someone who knows his stuff and isn't afraid of saying "I don't know (but I will find out)" when that's the case. I also take great care in planning work and defining deliverables so as to reduce the pressure on my colleagues' workday and there is hardly ever a situation where we need to rush to meet a deadline. If or when that does happen, I find that the culture of trust and a certain amount of friendship does wonders for the team's spirit.
I've worked in research since 2002. I always solve the problem. I'm the one you go to, after the other teams ran into trouble (I work rather hard to position myself as that).
I take the details of your issue, and offer to come back to you. Then I brainstorm and come back with a proposal. I will probably discuss the problem with several other specialists, one or two at a time.
Ideas occur at a Poisson rate with time spent thinking. I find running helps to focus the time spent thinking.
My most creative problem-solving ideas I've had while peeing. I think it is the interruption.
These screening interviews will rule me out, with high probability. Fine.
This sounds like a problem that you could put your mind to solving.
Yeah, I can do that. Mmmm. And maybe I pass. But other companies won't make me do that...
So, if the aim was screening for people that want to work for Google above all else, then, yes, the screening totally works. But... Google would downsize you and your project coldly if there were financial reason to do so, so...
The other part you should consider is that Google and others of their whiteboarding ilk will likely pay you much more than other companies, except maybe finance. So if other factors don't prevent you, maybe it's worth jumping through the hoops for the pay upgrade.
Those two sentences conflict heavily with each other. In research it is very rare to solve open problems. So either you are researching trivial things or your definition of "solving the problem" includes assuming it can't be solved.
At the end both of us came away with an understand of the others skills and how we worked as part of team.
I've since used a similar exercise when interviewing people as I've found I get a much better sense of what the candidate can do than placing them in from of a whiteboard.
1) Always, ALWAYS before writing a line of code say that you are trying to solve the problem first, by whatever means necessary and only then will you refactor.
2) Solve the problem by any means necessary. O(n*n)? Not freeing up resources? Allocating an array that is 100000 elements long just in case? No problem. Solve the problem.
3) Explain what you did/how you did it
4) Ask (if there is time) to refactor the code to be more idiomatic. Keep explaining that you wanted to solve the problem first, so later you can think of the right way to organize the code, and not the way around.
I never had an interviewer complain about solving the problem in that order nor have cut me off (even when it ran out of time) when I suggested to refactor the code in the end. Also solving the problem first, then using idiomatic code takes a bit of the stress off since you don't have to go and write and rewrite each line everytime you see something wrong.
 During the first part, I completely, totally, forgot about SQL joins. No, really.
That's always made clear though. I guess it goes to show that there's rarely one approach to this sort of thing.
I start with the problem, whatever it maybe, then I 'solve' it by whatever means necessary. This lets me iterate very very quickly over various solutions if needed without much concern about pretty code, even if it is a simple api call, parse json, display something in the view. When I have that working, I refactor, sometimes I rewrite, I make the UI nice, I write tests, etc.
In an interview, I believe you either want to see if the person can code/create and algorithm or you want to see his thought process on solving problems. If the latter, you don't need a whiteboard and a conversation with someone will work as well. If the first, I would do as I mentioned in the previous post. The O(n!) question for me wouldn't be important. I would probably reply "Yes, I know, let me just finish this part"... "Back to the O(n!), you are right/wrong, but seems to work this way, what we can do is change that for loop into a something/something reducing it to O(2n). And rewrite that part of the code.
When I was interviewing people, after a quick phone screening, we would ask someone to come over, we had a quick chat with them (30 minutes maximum) and then had 2 quick exercises for them to do. 1) given a simple piece of code (200 lines or so) we would ask them to refactor it, based on some parameters like future extensibility, not being linked to some implementation, etc. Second one was a written spec, with 3-4 unit test written that would verify the spec, and we would ask for them to write the implementations (max. time 3h, good candidates finished it in 30-45minutes, bad ones couldn't finish it in time). With those two questions we were quite easily 'judge' the competence of the candidate. The ones that did very good would go home with job offers (or would have one when they got home), bad ones were rejected, and intermediate ones, depending on team feedback would be invited back to discuss the code so we could make up our minds.
ps: I actually wrote the exercises and everyone already at the company did them as well and passed.
On the other hand, I'm on the opposite end of the spectrum from the author I suppose, I actually really enjoy interview questions. If anything I wish I would encounter those sort of questions day to day, but sadly real programming is much less demanding intellectually for most of us.
My first solution to the first part of the problem was laughably complex (but still correct). I soon arrived at a much simpler version the employer was likely looking for a few moments after.
Immediately the interviewer should've known I was affected negatively by being put on the spot. I am an introverted person after all, and I don't program on a giant whiteboard with someone constantly looking over my shoulder. So if my job is to work with someone new every day on a programming problem, someone I am not entirely comfortable around then I guess I'm off the list?
The second part of the problem was more complex, and I eventually worked through it, with some leading questions by the interviewer.
I did trip up once or twice. Oops, that one part of the program was O(n/2). I did not arrive at that one correctly.
I demonstrated I'm comfortable with multiple languages, can reason intelligently about a program and come up with some solutions, iterating on them while demonstrating understanding of big-O notation, but the employer cares about none of these things even if they tell you beforehand that they do. That's pretty much a lie. The employer is simply lazy and has decided the best metric for weeding out candidates are toy problems on a whiteboard.
It obviously can't do too much for the psychological side of the problem, but I believe that getting the interview environment as close as possible to your natural coding environment is definitely a needle-mover when it comes to candidate comfort during interviews.
As for whether or not coding problems on the phone or the whiteboard are a good idea, in my opinion they are when applied properly. There is no way I am going to hire someone for a software development position without seeing them write some code, and there is no way I am hiring someone for an embedded C position without knowing that they understand how pointers work. There are not that many reliable ways to do this in a controlled environment. We know that everyone can use google to find answers - I'm more interested in where people are starting from.
There is a balance though. I do not ask interviewees to solve brain teasers; that sort of problem only serves to increase stress in the interview. Most of the problems I ask are simple, such as printing out multiplication tables or simple string manipulation, or linked list type questions for embedded C. When doing these on a whiteboard, I am very clear up-front that trivial errors like missing punctuation or spelling do not matter. I am interested in seeing how the candidate approaches the problem logically, and it is surprisingly obvious what sort of candidate you are dealing with when seeing them attack a problem.
How often do you watch your currently hired developers as they actually produce the code? How does the best of them handle you standing over their shoulder judging their typing technique?
I'm unfairly giving you a hard time because you seem to have a sensible approach to hiring, but you should definitely know that watching people code, impacts the code they write. I've also found little evidence that people that are good at coding in front of someone, are necessarily better at the final product, than people that need some space to code.
1 - I don't think "cheating" is rampant. Obviously opinions will vary, but if you think that you are getting a lot of cheaters in your hiring pipeline, I would suggest the problem is in the funnel part not the evaluation part.
2 - I view getting help on a programming problem to be a good thing. I suppose in certain environments you need to make sure that your hiring candidates are not collaborating with others, but I'd be surprised if that was common. The idea that it is bad to seek outside help on work problems is some weird vestige of our education system, not an actual day to day issue with most developer jobs.
3 - It is much more important to find out if a candidate understands the code they've submitted and thought about the trade offs they've made (or didn't) as well as if they can explain it to the people around them. This is true regardless of how the code was produced. We suss this out with our colleagues all the time, and at least in my professional career its never been via watching someone work. We have tools like code review, discussion, documentation etc to do that.
You do need some knowledge up front and immediately accessible. However, almost no one agrees what that body of knowledge should be per position (and it's never going to be the same across positions or companies). Part of that is because you need to be able to practically communicate to someone about the things he's going to be responsible for. The other parts are actually doing the job and reducing the ramp up time.
The more accommodating positions will need less up-front knowledge (e.g. hiring any junior developer). They're okay if you don't know a few basics but you need to demonstrate something. You can't be too accommodating or what is the difference between him and someone from another field with no knowledge?
You also can't require too much knowledge up-front, or you just end up hunting for purple squirrels. Everyone (that is sane) already accepts that there is some ramp up time.
'Hitting the ground running', which is what people with a lot of up-front knowledge should be able to do, should be a luxury, not a requirement. Asking it of someone who isn't prepared is asking for a technical debt loan from the mafia loan sharks. The reason hiring managers seek people who can do this is because it means more work done in the same amount of time.
The conventional way to break into the industry is a CS degree and an internship, but that's not set in stone (contrast: medical doctors/scientists/etc). However, the industry still accommodates the self-taught crowd and other quirky origins because the body of knowledge we require to work in this industry is just so diverse and not agreed upon at all.
Whether a candidate can reason about the correctness of their code without using a debugger is a useful data point though. Both in a more abstract way and by manually tracing at least part of the program instead of stepping through it with a debugger. If you struggle to manually trace code in your favorite language, that raises some questions in my mind.
Fast forward to this year and and I've breezed through every interview with Google and other companies. I spent months preparing for the interviews, but the preparation was one of the most rewarding things I could have done as a developer at my stage. I feel far more competent as a developer with a solid grasp of algorithms and data structures. I enjoyed the preparation so much, that I started reading white papers on the more cutting-edge cs theory :)
On top of my personal projects, I've had great internships, which treated me like a full-time engineer. I shipped production code regularly during those.
Every skill takes time to develop yet most people think they are good at giving interviews with zero experience, research, experimentation or methodology. Allow a typical junior employee to start interviewing candidates and it will go right to their head as they are given their first taste of power over others in a professional setting. It takes time to learn how to interview properly, figure out the specific attributes you want to screen for a particular role, remove your own personal biases and plain get over the fact that you have the power over someone else's career for probably no good reason.
My philosophy on interviewing is that as an interviewer you want to avoid screening for anything that is not reflective of the actual job duties and to also avoid anything that the candidate is expecting to be asked. If you ask typical programming questions then you are likely screening for candidates that have the most time to prepare for interviews which screens out the lazy but also screens out the no-nonsense doesn't like to waste time types. Ask questions that get to the core of what interests them and figure out if they will be motivated enough to learn what they need to get the job done. Then get back to them quickly!
Once they pass that gate, then we do an interview with our dev team. That way everyone gets a chance to talk to/ hear the candidate and get a feel for if they want to work with them. As part of that interview we do a group coding exercise, usually using Pex4Fun (http://pex4fun.com/default.aspx?language=CSharp&sample=Chall...). Basically, we give the interviewee time to start working on it, and if they get stuck, then start white-boarding the problem as a group and solve it together.
Again, we want to see how the person works and whether we think they will fit well on the team.
After the interview is over, we take a vote and if the result is not a unanimous yes, then we pass. It's tough, but I've had about 80% of interviewees tell me that it's the most fun they've ever had in an interview. So far, we've been very happy with the results.
Did you ask them this during/immediately following the interview? That may... skew your results.
Then again, if they're offering that insight without being prompted, that's a very good sign. I think the execution of this strategy, and the general tone of the people involved is very important.
I also go out of my way to make sure I communicate with the candidate. We take the voice vote immediately after the interview, and I call the candidate within 24 hours. If there are any no votes I make sure to tell the candidate why, and try and give them pointers for specific areas they would need to improve on to work here and try and point them at resources they can use to improve.
We're in a relatively small market, and it pays to be nice to everyone. Anyone we interview I am pretty much guaranteed to run into at some sort of dev event. And just because someone wasn't a good fit today, it doesn't mean they won't be the perfect candidate next year or two years from now.
First I talked to the CEO who liked me and said: "Now your gonna be asked a few technical questions by our lead developer." The dev came in, we both hated a bit on Oracle and he told me to draw an UML diagram about a system of classes he will tell me. When he told me I didn't need to go into detail that much, the CEO looked impressed and I got the job.
We then developed RemoteInterview.io Tests . It's a screening platform based on coding tests where candidates solves the challenge in offline mode whenever they want. Some candidates still don't like to have time limits but I personally think it is important to have a deadline for every task because in real-world, no project can have infinite time.
If it is "you need to complete this problem in 2 hours from the time we on our server observe an arbitrary event", that would be a complete non-starter for me. A) those deadlines quite simply do not exist in the real world and B) it adds a whole level of stress to some candidates, much like the stage fright stress, that is unnecessary.
If it's something like "design this project/prototype" then a week should work.
However I absolutely agree that phone/whiteboard interviews are not sufficient for determining a candidate's fit, because it doesn't say much about the person's work ethic, time management, and ability to manage large projects - which is a thing that many "born gifted" smart people struggle with, in my experience.
Like an architect who chose synchronous HTTP as a protocol for a service that later required the ability to cancel long running operations in progress (he unfortunately didn't realize why hitting the browser stop button wasn't the same as cancelling the process on the server).
Or a senior dev who thought that a JSP scriptlet getting the current time would show the client's time, instead of the server's time.
There is no guarantee that people who ace brain teasers will necessarily be able to figure out technical issues, although they are often excellent at sounding like they have figured out technical issues when they leave the details for the people who get the real work done.
no pseudocode allowed. I'm not sure why you
should want that....
For example, if a candidate is asked to write a bubble sort and they write: Take the input array, go through every element comparing it to the one after it and, if they're in the wrong order, swap them. Repeat until you get through without any swaps.
Did I see the candidate think about off-by-one errors, like missing the last item or exceeding the array's bounds? Did the candidate think about common edge cases, like zero-item and one-item arrays? Did the candidate think about whether the function would work if the array contained two identical entries? Did the candidate check their loop will eventually terminate? Does the candidate know how to write a working computer program, rather than describing an algorithm in vague terms?
With the pseudocode answer I can't tell if the candidate did those things, because the solution doesn't show me what I need to see. And yet, if you've asked them to use pseudocode, they've done exactly what you asked of them.
That's why my preference is to say "use a real language and tell me the tests you'd perform on it, but I'm not fussed about semicolons and exact names of library functions." - it tends to get you a solution that lets you tell if the candidate knows how to program.
I'm curious if anyone else has had the experience of a pair programming exercise for their interview?
My experience with them as a good tool in the hiring pipeline? Less compelling. Any work sample evaluation is better than techniques that don't involve work samples and pairing seems to pain less developers than on paper coding does. But unless you actively pair a lot during your day to day work, then paired programming is still fundamentally different than the kinds of work you are actually looking for and there are lots of developers who work poorly in paired environments.
I feel this takes some of the pressure away but has the benefit of having an insight into how someone works technically.
- Conversation about the company/position, not even a screen, usually more of a sales job, but done by someone who can answer actual real on the ground questions about the position and the hiring process.
- Standard take home programming problem.
- Code review session.
That is all. My own personal bias has come down on the code review session isn't actually important either, but I get looked at like I have 3 heads when I suggest 0 interview style settings in a hiring pipeline.
There aren't. If there were, they wouldn't be split on the axis of "programming in a stressful situation".
Even in development environments where front line stress can be a part of the job, its not full time, and the good developers work on 1) reducing the amount of time the organization spends in those modes and 2) removing human performance from the triage out of those modes. The reason for that is that under stress performance is not actually repeatable. Everyone breaks eventually and pushing the eventually back is not a cost effective way to mitigate the risks.