I conducted a couple hundred interviews for my first FAANG employer, and I was constantly amazed at the percentage of candidates with years of Microsoft or Facebook experience on the resumes who apparently did not know how to program. I always thought, 'huh, guess I know why they quit after 3 years, amazing that they all lasted this long."
Then I interviewed for another company and utterly bombed. It became suddenly clear to me that I had been an idiot. Of course nearly all of those candidates were perfectly good programmers. They had had shit interviewing days, probably mostly due to nerves, but they probably would have mostly been perfectly good employees. How frickin' arrogant I had been for concluding that people who couldn't solve incredibly high stakes algorithm riddles on a whiteboard in 45 minutes with enough speed and flair were somehow not qualified to be my coworkers.
I was on the other side - I flew out to FB during a vacation earlier this year. I got there, could explain the solution but something was off and I kept losing the thread when I went to convert it to actual code.
The interviewers didn’t know I’d gotten a call from my cat sitter that morning and one of my cats had to go in for an emergency check up. (He ended up being fine but didn’t know that at the time).
Later that week I got the rejection from FB and then, 15 minutes later the positive response from Google. Really made me realize how much impact the day has on an interview outcome.
A pro tip for anyone reading this who ever ends up in a similar situation: just postpone the interview. I promise you that everyone will understand. Everyone gets that these things are a little goofy[1] and wants to set people up for success and not doom them to failure.
1. Yes, there is a larger and more complicated discussion to be had about making the whole system better. But that isn't gonna happen in the short term. This is advice for dealing with the situation as it exists today.
Yeah I didn’t realize how distracted I was until I was there and just blanked.
I also have real problems with traveling to interviews since my other half has issues being alone at home hence doing the interview while we were visiting her family. Even then me being gone was highly stressful for her.
Google was willing to let me interview with folks local to us which really helped. I’m curious if more companies will conduct remote interviews in a post Covid world too.
If you explain extenuating circumstances to your recruiting contact, you probably can get a chance to reinterview sooner than the usual cool off period (especially if you did well in the earlier rounds).
It's worth emphasizing, as a hiring manager: By the time you're in an interview, I want to hire you. I'm sitting there hoping to write "Strong hire" on my feedback form.
I think this is true as a hiring manager, but isn't necessarily true for an interviewer, who may be a senior engineer applying a very strong filter to the "who would I want on my team" test.
I'm senior enough that if I rate two people strong hire, I'm capable of getting more budget. This has in fact happened recently -- we couldn't decide between two great candidates, so we offered and hired both.
Yes, and in some cases, the company needs one person now, and another one a bit later. Then can be nice to have a list of people who did well, to contact again
Same. If someone makes it that far, I (and several others) have already decided that they stand out. So I'm secretly rooting for them the moment the interview starts.
I would LOVE to get my 2 hours back in a day because of a last minute interview cancelation. The only person who would be truly irritated is the recruiter, who's job depends on meeting candidate quotas. Even then, they want you to be successful.
Trust me anyone reading this, if you are not feeling well, or something is distracting you, please, please, do feel free to reschedule your interview, even last minute.
Once you go through the process, and it's a no-hire, that is recorded in the system and you won't be allowed to try again for some extended period of time. At my company, it's 1 year.
The obvious counter to this is: Which will give the worse impression: Canceling for a good reason or showing up and being distracted and performing poorly? Companies like the usual FAANG are nice enough to merely give you a 1 year embargo. In other companies I've worked at, you interview for a team and if you do poorly, the team will simply never interview you again (although you could interview for a different team at the company).
Also: Do you really want to date someone who doesn't care that your cat may have a life threatening problem?
Finally, even if it's not that serious (cat emergency): Do you really want to work/date someone who is that susceptible to this bias? I know it's the norm to fall for first impressions, but for me, it's also a signal of problems I'll have with them in the future.
BTW, my cat had a serious health problem once. She needed lots of immediate care for a few months (at home, away from work). And the vets were telling us that even if she got through it for now, it will come back soon and we should really consider euthanasia - the condition would eventually kill her - not many cats would last a year with it, and chances are her she would be in pain for much of it.
And this all started a few days before I started my new job.
I went straight to my new manager (before my official start date), and explained to him that the cat was my priority - I could take a leave of absence or whatever was needed, but I needed to figure this out (either euthanasia, or time off for treatment options, etc) and could not be anywhere close to 100% at work.
Fortunately he was sympathetic and said I should not worry and do whatever was necessary.
The job turned out to be crappy, and I stayed there longer than I should have entirely because of this.
For a small company it matters, but for a FAANG interview the person rescheduling your interview is completely disconnected from the people making the hiring decision so it would literally not make a difference.
I rescheduled the second date with my now wife at the last minute. Worked out well for us, but I was so tired from work that there’s no way that date would have gone well (busy period of a project at the time).
I think knowledge workers will have to learn more about human psychology in the future.
I know I go thru waves of high productivity and low productivity. That productivity seems to be closely related to the ability to concentrate in my environment, sufficient sleep, how closely I've followed y daily rituals, fewer stressors in the past few weeks, etc.
While waiting at UBER reception area I got a call from a "tax office" telling me about suspected tax fraud transactions (it was a fraudster call but imagine me at that moment) a minute before I was called in for an interview.
Yeah, I was really kicking ass at that interview (NOT!)
The same is true for design/ux jobs interviews. Well, at least for me. I got laid off a couple of months ago, and have been interviewing pretty steadily since then. I have yet to receive an offer.
I already know where my strengths and weaknesses are. I'm fine with 1 on 1s and all that, but when it comes to the technical part of the interview process like a take-home design project or a live design challenge, I usually fail.
I know I have the skills. My work has always been well-received, and I've always received good marks on reviews and feedback, but I haven't been able to translate it to the interview process.
I've tried to assess where I go wrong, and I think I panic and put too much effort into trying to figure out what they want to see or come up with something innovative rather than just doing the work and following my normal approach.
The stress of the time crunch (and sometimes not knowing anything about the product or problem) only add to my inability to improve in this space.
--
When I've sat on the other side of the table, I know that most interview processes aren't that well-structured or set up to actually evaluate the candidate. Even when there's a good process, it's often too easy for one person to influence the results.
So sometimes I tell myself that it was a crappy interview process and it just wasn't set up for someone like me to succeed. But I still end up stewing over it and feeling depressed for a few days.
The depression hits those hardest who put their identity into what they do. Often, I don't realize how dependent I am on positive feedback until it hits. Lately, I've tried to remember in those moments that I am a diverse person and that if I looked in the mirror and saw the worst developer ever I would be OK
If you’ve been looking for a while I think it’s totally normal to get some deep depression regardless of how you identify. Interviewing is stressful and draining and feels incredibly arbitrary. Also, at least in the US, your employment is absolutely tied to your quality of life; unemployment benefits are a joke and you’re on your own after they run out. I’d be depressed as well.
Not saying I disagree with you though, I think identifying heavily with your job makes it even worse.
That's a good point. I probably put way more into my identity as a designer than I should or care to admit. Thanks, gives me something to think about and work on.
Hang in there dude, I think the coronavirus and remote working really messed with the traditional hiring process.
The fact that you are getting steady interviews speaks for itself that you have the design chops and experience. I think people don't know how to interview design candidates well as is, removing the physical cues you get from an in-person interview makes it even harder to assess a designer. Also, it could just be companies are flying by the seat of their pants and a lot have reduced workforce with this second wave of infections.
High probability it's not you, it's them and their broken process during covid-19. Keep your design skills sharp and keep building your portfolio and use this time to work on concepts that you are passionate about that haven't come up in a 9-5, that will help you until the right opportunity comes along. You got this!
>I panic and put too much effort into trying to figure out what they want to see
My approach for take home challenges is this. Often I complete them and never even get any feedback from the firm beyond a form rejection letter. If that is going to be the case anyway I might as well make something I'm proud to have in my portfolio, even if it doesn't line up with exactly what they are asking for. That transforms the work from something that I'm doing for them to something I'm doing for me. I know that's hard when you are unemployed and have bills to pay, but if at the end of the day you have some work that at least meets with your expectations you can retain that bit of your pride.
This made me think, what if we post take home assignments on github. Along with company name and status - accepted/rejected. This is a testimony of your work as well as of hiring company - if code is good and you are still rejected, that is the red flag to developers.
This.
Adding up, front-end engineering has an identity crisis now.
A friend of mine is a web dev who started back in the days when Javascript was for form validation.
When JS frameworks and SPAs took over the web, he adapted himself. But being a UI engineer he cares about the craft's visual aspects, than the technicalities of JS.
Although he has a proven record of his works, he is now dealing with interview questions like "what are the problems with closures" and "when you do use function.apply()" and "immutability".
For a live design challenge I recommend slowing yourself down and asking for more input from your interviewers. If you appear relaxed and asking for input from others then you will seem more open to both collaboration and constructive criticism. My two cents and good luck.
Have done over 1,000 interviews for a FAANG competitor that IPO'ed in the past 10 years. Completion time (in minutes) of a coding puzzle is essentially the only objective measurement taken during these coding puzzles, and is sadly the primary datapoint fed into the hiring manager's decision. The hiring managers usually adjust the cutoff based upon whatever quota they want and/or if they're expecting people to leave. I've worked directly with a VP of Engineering and that's really all he wanted to hear.
The whole process is missing a feedback loop to the evaluators, resulting in the OP's situation. In my own situation, it took several years after that mess of being forced to shotgun candidates to pick apart what in the experience was actually predictive of success and what wasn't. To this day, the most successful hires from my perspective were those who just really wanted the job and wanted to contribute to the product; definitely no strong correlation with "leetcode" ability.
It's not just that the coding quizzes are poorly administered, but that the hiring mangers and leadership industry-wide have been obstructive to getting evaluators feedback (for starters, what did a candidate's other offers look like? where were they 3 months after the interview?) and especially tight-lipped on sharing interview questions. That last bit is particularly obnoxious because people monetize their insider knowledge either through things like Rooftop Slushie or they take a more Gayle Laakmann angle. (Just to be clear: while many questions are "protected under NDA," many questions are also stolen from other companies-- interviewers asking them are already violating their own NDAs when asking them, or potentially when feeding the question bank of their new employer).
What you can do is go to your manager in your 1:1 and ask for interview feedback. Ask what happened to candidates 3-6 months after the interview, even if they were hired (perhaps not just to your team). Start to get feedback, and then work from there. You'll probably do better interviews, and at least curtail some of the information arbitrage culture in the C-suite.
Occam's razor is: algorithm tests are specialised IQ tests, and that version of IQ is actually a very reliable indicator of performance (compared to the other signals available).
I'm not exactly sure I buy it, but it seems like the simplest explanation.
It's not just nerves either. All interviews, but especially FAANG ones, are random. The interview might be racist, sexist, etc. They might have gotten divorced that day. The question they ask you is something you haven't heard of but they spent their PhD on. You can even do everything correctly, but the interviewer doesn't like the way you did it.
The leetcode stuff is just masochism. You will spend X months beating DS&A into your brain until someone can pullout a random question and you immediately know what questions to ask, what steps to take, and how to write it out on a whiteboard.
It's difficult to remove that randomness from the process. As I see it the only reliable approach for both sides, given that structure, is volume - you just do/conduct a lot of interviews and evaluate/get evaluated on the top percentile.
I agree with you for the most part, but Google goes out of its way to force the interviewer to just hand in an emotionless, gender-neutral description of the candidate's code and coding performance to a hiring committee that makes the decision
This kinda bit me in the rear when I applied to Google a couple of times. As a nerdy American male, aged 22 to 40, with above-average people skills (for nerds) I got along great with everyone I talked to. But the hiring committee would get a sloppy page of half-functional garbage code, along with a note like "Seems like a great candidate". I found another FAANG company I fit into a lot better anyway, so it all worked out great
Additionally, companies doing common pool hiring are worsening the situation even further. One bad interview day or one failure to solve complex riddle means that you're not good for any teams within that company until cool-off time passes which is typically from 6 months to 1 year. The system of cool-off period is equally brutal.
I've heard HR optimizes processes that limit Type I errors (false positive) at the expense of Type II errors (false negative).
I've often wondered if this is because, from the HR perspective, false positives are much more costly. I.e., Those who would be helped by limiting the false negatives are project managers who are too downstream in the process for HR to care.
You only need to fill a position with a person that is good enough.
If you reject someone who would have been a good fit but still hire someone good eventually, that’s ok.
A bad hire is not fun, not for HR, not for the hiring manager, not for the colleagues and ultimately not for the person who got hired. Especially if they had to move for the job, possibly with family.
A bad hire has massive negative consequences for many people. Of course you want to avoid that, if that comes at the cost of not hiring someone who would have been great occasionally, that’s unfortunate but acceptable.
I don’t disagree but “good enough” comes with its own risks. That may be fine for an organization that has a lot of inertia and is biased towards the “maintaining” side of the house. However, I think it would make it make things more difficult if the intent was for bold, transformative change.
The bar for "good enough" can of course be quite high.
If you're hiring people at the very top or with rare skillsets, like building browser engines, building databases or people whose skill is at author/contributor level to relevant technology (for the job in question), you're probably not going to go with the usual process anyway as there is less uncertainty.
Fun fact: Google & other tech giants have poached a lot of game engine designers to optimise datacenters at truly obnoxious salaries, because their fundamental skill is making complex systems of interacting entities run as efficiently as possible. It's causing problems in the games industry because they can't compete.
Understanding this is the most important part of interview process, in my opinion. Large companies want to minimise bad hires, small companies are more open to odd CVs and novel approaches. Market yourself appropriately.
Not trying to be snarky but isn’t it intuitively more expensive assuming you’ll try to fire and then rehire correctly? It’s like asking “why does it cost more to go through a process twice rather than once?”
I think the problem is HR itself. Few companies should be large enough to warrant full time HR, but even small companies need many.. HR are specialist in expanding the disconnect between people the company thinks it needs and people who are willing to work for it.
You say that, but look through this thread. HR has a solid understanding of what a given company's large-scale objectives are when hiring. As evidenced by most replies in this thread, technicians on the ground do not.
The evidence is that it is a hard problem that is not solved. No amount of people who can't do the task demonstrate the existence of experts who can. Google and Microsoft have shown us examples of failing at simple HR tasks for years or decades with budgets for HR that dwarf a small companies entire payroll.
What virtually every attempt shows is that it is better to hire people who are roughly qualified and see what happens, managing the results instead of filtering. Having an HR strategy for hiring leads to creating biases (i.e. requirements based on your current work force) that gets you a group of overly similar and therefore collectively incompetent work force.
What some of us do now is at home exercises with no deadline and setup the interview to do a code review of it after it's given back.
It gives so many advantages:
- people will have some fun trying to solve a real issue / build a little software
- they will show how good they are at stack overflow, general methodology, cleanliness, test coverage, deployment and even git (how many we see who can't even gitignore their project files, make a build script that works, or test for real)
- horrible candidates who can't even cheat will not be able to lie their way around a fake resume, cheaters will bomb the code review quite fast
- candidate will also discover their future team mates general work attitude during the review and if the reviewer is good, will lock the candidate faster than "how many billard balls can you put in a Boeing 747" type of interview.
- candidate will even LEARN during the code review => you get away with something out of the interview even if you fail
Since I've been interviewed that way, doing my little project on my wife's pregnancy bed smiling like an idiot at solving the challenge, then fell in love with my manager showing me new tricks during the review, I'll never do any other kind of interviews myself.
I recently did one of these where I was paid at half the hourly rate of the job offer. There was an initial unpaid assignment that was difficult but totally related to most engineers' day to day work. That took less than 30 minutes. Anyone who passed that got the paid assignment which took a few hours.
If I had options I wouldn't even do a 30-minute assignment. I'd just go with someone else. I'm reading about employers giving assignments a lot in this thread, but what do you do about the fact that the people you really want just don't have a reason to waste that time?
There was no interview. Not even a phone call. After I submitted the 3 hour paid assignment they hired me by giving me access to Slack and we just started working together. One of the smoothest and easiest "interviews" I've ever done and I got paid for everything except for a 30 minute challenge that helped solidify my understanding of git.
> people you really want just don't have a reason to waste that time
I'm "really wanted". In the past week I have had to say no to two clients calling me for follow up work. Where do you think I wasted my time?
It was great and I enjoyed working with them. That's why even though interviewing is broken at most companies I enter the process assuming the best and with an open mind. I'm disappointed 90% of the time, but you don't want to throw out a gem just because you usually dig up worthless stones.
That's why I think take-home exercises are a much better way to evaluate candidates. Give them the exercise and enough time and you will figure out:
1) How good they are
2) How much they actually care about the job offer -- the more detail and the more passion they put in the exercise, the more into the job they are.
Why not drastically different approach - ie. instead of active technical interview process on company's side, just put up few dozens of open source projects/libraries that are being used inside company and leave it as technical interview sandbox.
Keep brownie points on pull requests to unlock "you gained <<passed technical interview>> badge" a'la stack overflow or do one off evaluation based on contribution when candidate applies. Stack overflow contributions can be equally taken under account to gain points and arguably they have badge system that does the job for you already.
Github could introduce something similar and wipe out "anxiety interview" completely, if they wanted to.
There is really no reason to do technical interview when you can inspect open source work. For people who don't have contributions yet, open source libraries/projects by the company should be made available.
Unless you are looking for a "senior developer with x years of experience". I have a live, hobbies and family. I don't care to spend most of my free time fixing code in some obscure library I'm not ever going to use only to stand a chance to get a different job in the future.
How in the world did current developers EVER get ANY job? Who gives developers the chance to grow these days?
It's a "foot in" problem, I feel. Shit companies have shit interviews and give out shit workloads to MAYBE get a foot in the door.
You can follow along or be proactive, for instance start a new github account just for interviews, do code wars challenges, write a blog and just link your writeups, github and blog to applications. Along with your CV with major projects etc. of course.
Most people I know who work at big firms got calls from friends or friends of friends to interviews, so spending time socializing the local hacker circles is probably going to be worthwhile as well.
If you have life, hobbies and family, you should still be able to schedule 1-4 hours a week to job hunting, much more if you don't have a job of course.
It's not pretty or much fun, but it's certainly doable.
Not the worst approach, and as long as it's in the spirit of cooperation might even be good.
UX people could tune or at least comment on the interface, security people could fuzz or pentest the library... however, for a lot of actual problems in actual open source libraries it requires a crapton of internalization -- again, for free. Granted, it's for open source but it could be argued it's even WORSE approach for the company, since they're getting free benefit from expertise and work.
Also, I'm a private person, I have splintered online identities, I'm not going to give my 'hobbyist' github account in a professional context, and I'm also certainly not going to make a github account just for interviews and applications.
I'm lucky enough to have connections that will probably keep me employed as long as I like, and the tales of interviews and free work that's required to put up just to get a chance seems insane.
4 hours... okay, I can sort of see that, I might set myself a limit of 1 hour and tell them this was my approach, this was how far I got. I've heard of 30-60 hour workloads given to applicants to 'top five' companies and it's just ridiculous. As are the stories of interviewers who have no technical skills themselves and are just looking for a canned response (Google, looking at you).
I was going to say that the best approach I've seen are companies that put up interesting puzzles and security challenges on their website. The technical skills required aren't _that_ high, but certainly require an amount of ingenuity, persistence and just plain being interested in tinkering that they feel suits their company profile. The applicants can solve or try to solve these challenges, do writeups and probably get interviews just based on those.
Granted, I can also understand someone looking at 50 companies all giving them 4-10 hour workloads to MAYBE get an interview might feel frustrated and overloaded. It's a classic egg-and-chicken problem. If I were to start looking for a job from scratch I'd probably solve a bunch of code wars challenges and security CTFs, make a blog and just link my writeups and blog to applications.
Doesn't that tell you that the method of hiring is flawed in the first place?
If you care to weed out bad candiates, what help are 100% perfectly fine take-home exercises? The only thing that tells you is, that all of your candidates are viable in the first place. Just go into an interview with THAT assumption and ask the right questions.
Why do so many companies assume that people who try to get hired for a certain job don't meet the basic requirements? That gotta be the rule not the exception? That's just a bad faith assumption.
Just make sure your hire fits into your culture and give them sufficient probation time to grow into your codebase.
Could you provide a bit more info? How long were the exercises? Were the applicants evaluated more based on their results or possible writeups? How much did their CV weigh compared to the take-home exercises?
Where engineering fields have a disproporationate number of individuals with ASD (née aspergers), and where anxiety plays a large part of the syndromes, it's no surprise that many, many good engineers are terrible at interviewing.
It's also sort of ironic how studies repeatedly show high preforming teams, are cognitively diverse teams. Even though research shows this, the hiring strategy of most technology companies, is hiring new employees with the same skills, and traits, as current employees.
This is a critical point that a lot of hiring misses. We talk about building great teams, but the hiring requirements are static and never seem to consider who's currently on the team and what mindsets or backgrounds are missing.
For three years or so I taught week long corporate training classes at Apple, Microsoft, Facebook, eBay, PayPal, Cisco and Intel. Every week, week after week, real rocket-science level Android and iOS deep-dive in to the guts of the operating system classes dealing with firmware and device drivers and file systems. In the classes we didn't build apps, we built operating system components. Small to large classes, thousands of software engineers from senior and up, over the course of those three years.
I would stand up in front of 80 or more engineers, go in to esoteric aresa of the operating and talk eloquently and at length about the structures and code found there, and live code on a projector, answering questions about the code and debugging the code of everyone in my class as they followed along. Day-after-day. 40+ hours a week. It was intense. I became an independent advisor to teams and a consultant for some of the groups on how to bring up Android on new devices, create device drivers, and so forth.
Some of those classes, I wondered how some of the people I was teaching was actually able to hold down their job.
Months later, after I stopped teaching, I interviewed at Intel and PayPal and Cisco. I bombed every interview. I was, by all accounts, unable to program my way out of a wet paper bag.
I call these high stakes interviews "the perfect dismount".
You can feel the pressure mount - any algorithm that doesn't run on the first try or second (with a couple of quick fixes) and you're being mentally discounted.
The same for silence - if you're thinking through something, take abnormally too long, you're discounted.
The Perfect Dismount. A 10.0 is what's needed to land a FAANG job.
I just pulled out my laptop and if they aren't cool with the solution I google'd in 30 seconds, they can test my critical thinking in another manner. That was my critical thinking and resolve under pressure.
These are nothing but shit test.
What are shit tests? When somebody fucks with your head to see how you will react, what you are experiencing is typically a (series of) shit test(s). Everyone has been shit tested, gets shit tested and will continue to be shit tested; We use shit tests to make value judgements about people, likewise they can be used to determine how people cope under pressure. The underlying mechanism of shit tests is to test your mettle.
I much preferred our style of (technical) interviews (I took on a few dozen over the years). First there's a review of the CV, Linkedin, and github, if that looks good, a first interview - just get to know the person, very casual. If that clicks, second interview is the technical interview - we did that by giving the candidate homework, 4-8 hours worth of tasks that are close to their day-to-day work, usually something with a REST API, some basic math/arithmetic, and a front-end / forms, and plenty of room for the candidate to play with.
It was received pretty positively; especially early on (2012-2015) we had a lot of candidates indicate they had never done anything with JSON or REST before. (later on they mentioned having done nothing with XML before, lol).
And we got to see some interesting solutions and creativity; one guy we hired did it all in J2EE, while we were a Spring fanclub. But the code was sound, he showed that he understood how it worked, etc.
The conversation during the technical interviews were entertaining as well.
I learned this lesson a few years back when I interviewed for the CTO role of a educational software company - the main part of the interview was preparing a presentation on how to launch a new product which I had to prepare and present to the management team.
That went very well and I was feeling pretty good.
Then the CEO had a quick chat and happened to ask me a very simple technical question and my brain would not work - I completely failed to do it even though it was trivial.
I think that's how I discovered that mental context switching is a real thing - I had spent a few hours preparing and giving a presentation and then when asked a simple technical question those parts of my brain were completely offline. I think the shock of not being able to do it had a big effect on me and I went to bits (possibly the first time I've ever done that).
The best advice I ever got about interviews is to figure out ways to prove yourself wrong about what you think about the candidate. Although it's good advice for a candidate who seems too perfect, this advice really shines when someone is bombing.
That advice seems like unnecessary mental gymnastics.
Just take an unseen technical problem, solve it for the very first time in front of a colleague, then double your time as the expected time for candidates in an interview.
When you're under time-pressure, you actually spend more time thinking about how much time you have left, rather than thinking about how to solve the problem.
It's not mental gymnastics. The point is to challenge yourself as an interviewer to reveal a diamond in the rough. Some people suck at interviews and are great at coding. I've hired a lot of them and we've made a lot of money together. I've never had to fire anyone either so I'm not getting false positives either.
> Just take an unseen technical problem, solve it for the very first time in front of a colleague, then double your time as the expected time for candidates in an interview.
That gives very poor signal in my experience due to the time pressure. Your process is now selecting for people who are good at taking tests instead of selecting for good engineers.
I've done my first programmer hiring for my startup several years back.
Thankfully I've came across Guerilla Guide to Interviewing[1] article by Joel Spolsky, and that helped greatly.
I've interviewed 6 candidates within a week, and hired one. Never regretted of the outcome.
I've set up a test code so: an input variable, comparison variable, and an empty function.
Function takes input variable as an input, and output of which is compared with comparison variable.
Interviewee would be asked to fill in the body of the function, and play with it until function gives the expected output.
I've set up a computer with two screens, put IDE on one, and Google on the other. Two chairs, and some coffee.
First I've chatted with the interviewee for about 10 minutes, tryin to make them comfy as possible.
Then before the test, I've stressed very much that I am also a programmer, and I know how awkward it is to be coding while someone watching, and that'd alone would make me do silly mistakes, so I was expecting same from them and that was completely OK.
Afterwards, I've encouraged them to use Google in plenty, and feel free to stay silent or explain as they go.
So I got to assess their fluid intelligence, their ability to break the task down and progress efficiently, their English proficiency[2] (if they use Google in English), their usage of keywords, their choice among in stack overflow responses, etc.
At the end I've rejected a guy with 8 years of experience on his CV, and hired a junior. Looking back, that turned out to be an amazing decision, best I could have made, for work went good with the junior and I've had the (mis)fortune of working side by side with the 8yr guy several years later.
PS 1: Task was to write a recursive function to travelse a multidimensional array, and find out whether the first letter of every "value" was a capital letter.
PS 2: The work was to maintain and develop mid-scale SaaS project along with me.
Couple that with 45 five times over! I was flown down for a Facebook interview from the east coast. It was my first tech interview and it was with facebook. I was so exhausted by it, and then the next day I had an interview at Uber. I started that one already pretty tired.
I'm so glad this realization is the top comment on HN.
I think Fizzbuzz can be actually a reverse test: if an experienced programmer can fail fizzbuzz in an interview, then maybe it's the interviewer who screwed up.
Senior/Principal Engineers at FAANGs may not even code all that much in their day to day. The difficult decision that gets faced when interviewing those folks is whether they could walk into a fresh environment, and stack, and immediately earn the trust of their team.
In practice whiteboard interviews assess this pretty well. A new senior engineer must immediately be shipping optimal, tested, and well designed code. Otherwise you'll have Junior Engineers coaching Senior Engineers and the Junior Engineers will just get frustrated and leave. Worst case, you'll have an "architect" who never learns how to actually deliver independently in the environment.
Any significant system will take months to learn. Your statement is probably only true for some trivial system.
A senior engineer comes with years of experience designing and coding systems that took years to develop and run at scale. Given time to learn they should be just as good at solving day to day bugs but if you assign them that you are wasting your time and money on experience. If you don't value experience you are wasting your time and money on experience.
It's pretty normal to have more junior engineering staff help new senior staff learn the ropes. It's also common for those less experienced staff to learn from the more experienced people even in that situation. I'm not saying senior hires should be respected from the first second because they are over 30 or have a long resume.
Learning the ropes of a new company is fair and being slower at first is to be expected, but explaining that code reviews need to be tested to a senior hire isn't as fair. Similarly a new senior hire should be able to hop into a code review and catch problems/suggest improvements early in the process.
These two behaviors aren't as dependent on knowledge of the system as they are on core coding competencies. The whiteboard interview reasonably works to assess these, but it requires a lot of prep and is prone to false negatives.
You've been arrogant for years and you know what? So have I. I and many, many people in this industry have been doing this for years.
What's different from you and many other people is that you admit and face your bias rather than justify it.
I'm seriously curious what google interview board members have to say about this study. People like Gayle Laakmaan have been saying things to justify the whole process for years; but now that there's actual science, what do they have to say in the face of science?
I interviewed a lot of people while I was at Google, and sometimes the people were clearly, clearly too anxious to interview. Like one fresh-grad who's hands were shaking. I thought about how much pressure the kid must have been under.
You try to help them relax, but you don't have much time, and the whole thing is just so unnatural.
Other places I've since been at, and interviewed with, give you a laptop, a problem to solve, and some time. You can focus on the problem instead of the situation much better, and I think that really helps. As an interviewer, it also helps keep some distance from the interviewer. It's too easy to try to micromanage the interview and keep the candidate uneasy. But there's a natural "don't interrupt someone when working" instinct that keeps interviewers more distant when the candidate's programming.
Coderpads on video conference, I think, are pretty good when the candidate has a good space to work in. They're in comfortable territory and you can see them by their keystrokes. The interviewer feels like they're getting more accurate data about how the candidate really operates. I just wish it could do keyboard shortcuts better -- emacs users like me hate seeing Ctrl-N bring up a new window.
The issue of IDE/compiler/keyboard/keybindings is also there, but that's easier to work with.
Microsoft interview. I was just graduating college, and was super anxious walking in to my first interview. The interviewer realized that writing on the whiteboard was going to be awkward for the problem he was asking me to solve so he let me just sit down at his computer while he hung out to to the side and watched me work in notepad. I think that was the best way to do that interview, because the second I started writing code my brain flipped in to code-writing mode and everything relaxed. One of the better interviews in my career.
Nowadays that's "favoritism" in an interview and it wouldn't fly. Everyone must pass through the same meatgrinder (that, surprise, fails people other than white men more often).
She will just say that the process is designed to minimize false positives and the expense of more false negatives. A bad hire is more expensive than a no hire.
The problem is this isn't what gets you the smartest people on the planet. A small number might be interested in DS&A to that level, but most are not. They will learn enough to know how to google what they need and move on to what they are interested in. Smart people are constantly getting job offers from coworkers and bosses who move to other companies if they don't start their own.
The few smart people who do put in the effort necessary to go to the FAANG companies always leave after getting the golden sticker on their resume. They leave behind them a residue of SAT preppers who have PhD's in inversing binary trees.
From my experience it won't get you the smartest people on the planet, but it will get you people who _think_ they are the smartest people on the planet.
At Google I've worked with some very humble but also super intelligent awesome engineers, people much smarter than I. Definitely the majority of my interactions. But I've also worked with people who clearly took the "Google hires the smartest people" company line, and their own success at it, a little too personally.
I don't think it's a good message to send. Although to be fair I haven't heard internally in a while.
Also I don't know if it's really a gold sticker anymmore. Nor do I think it's true that the smart people leave. Some do, but honestly, there are some damn brilliant people at Google who have found their brilliant corner and produce brilliance there.
Or some people here who are just brilliant at playing Big Company. Unfortunately there's more of that all the time.
> She will just say that the process is designed to minimize false positives and the expense of more false negatives. A bad hire is more expensive than a no hire.
That's what they keep saying. However, it remains to be proven that regurgitating answers on a whiteboard in 45 minutes implies that the candidate is not a false negative, or even a true positive to begin with.
Someone who able to immediately understand and write out the solution to 80% of hard DS&A questions is at least able to write code. The question is if this is honestly better than generating your own FizzBuzz. I don't think so. I have heard rumors that Google has studied employee performance predictions based on interview score and found no correlation. I would bet this is an open secret and no one knows what to replace it with that isn't enormously expensive.
> no one knows what to replace it with that isn't enormously expensive
That's the crazy thing about this. FAANG has near-infinite resources, relatively speaking. Google could be A/B testing their interviewing process, experimenting new approaches, experimenting (as the OP study did) with eye-trackers and other tech to try to gain insights on candidate behavior. They have the "engineering for the sake of engineering" and "innovation for the sake of innovation" culture that allows for moonshots and boondoggles. I don't think they're exactly known for following "if it ain't broke, don't fix it", given how many times they kill off products only to recreate them later on, especially their chat apps.
Yet they stick to the traditional whiteboarding method because-- why?
Maybe Google has studied all of those things and found whiteboarding to be the best predictor. The company seems to be doing well, so something must be working with their hiring. They dropped GPA/college requirements so they're clearly open to non-traditional backgrounds.
> The company seems to be doing well, so something must be working with their hiring.
People use this argument often, but is technical competence of engineers- specifically at passing their interview process -the explanation for their economic success? And not market penetration, near-monopoly positions, great UX, etc.?
Not to mention, as we've seen in the high-profile Facebook SDK crashes this past week, share price doesn't necessarily mean product quality.
It means you're someone willing to spend two to three months on your life sucking a little dick for a chance to work at FAANG... the companies so good they've recommended 15 vacuum cleaners after purchasing one. How many hands do they think I actually have.
Boy, am I the only one who thinks working at FAANG is pretty great and worth the time practicing the interview?
The compensation is much better. The management is much better. The work life balance is much better.
I moved from SEA. Everything is worse there.
I read this and feel like people have unrealistic standard. FAANG isn't good enough? Working there for a year probably put your wealth of life at 0.1% of the world. That's not good enough?
We don't care about reality, merely perceptions. "Well I humiliated the person for an hour in front of a whiteboard" is the new "Nobody gets fired for buying IBM."
intelligence is multi dimensional. Even more so at a working environment. I can't believe algorithm intelligence is a relevant skillset for 99% of the projects. Including projects inside google.
Gayle Laakman specifically has a business empire dedicated to help people get through the process. Regardless of the actual merits of the process her opinion is going to be biased.
Quite a racket, being part of setting up this gauntlet and then selling books on how to get through it. Some might call that a conflict of interest, but there's no such thing as ethics for the nobility so to hell with what the peasants think.
I don't think it's relevant. Her business is designed to get you to game and pass the interview regardless of whether or not the interview is effective.
Therefore her business interests are orthogonal to her opinions on "interviews." She may still be biased, but I don't think business interests color her bias.
I'm not a googler (but I've interviewed there) and I disagree somewhat with the original premise. I think whiteboarding does filter out bad candidates but also filters out good candidates who fail the anxiety test. Google, with their massive pipeline, probably doesn't care about the false negatives enough to change; they still find enough people to hire. It's the companies with smaller pipelines that need to scoop up these good candidates.
I used to be reasonably good at algorithmic stuff in university 20 years ago. Since then I hardly ever practice. It's not that I am bad at them, now, but its just so hit and miss whether an answer will come to me in time, you might as well toss a coin. If I was doing algorithmic stuff every day I have no doubt I would be better. Usually what happens is I get out of an interview and a an answer (or a better answer than I gave) pops into my head. Like I say, might as well toss a coin.
In general, hiring practices probably optimize for filtering out bad (or at least somewhat speculative) hires over passing on a potentially really good candidate (especially if there are any concerns to go along with the overall positives).
Of course, if the pipeline is massive (whether it's jobs, schools, etc.) this tendency gets amped up even more and anyone who doesn't come across as pretty much perfect on all dimensions--whether they are or not--is going to get dinged.
I've trained for interviewing twice at Google. I don't volunteer to give the interviews, though, because the interview I was trained to give, I would never pass.
And sadly, this is common and recognized at Google, was mentioned in interview training, and is just said to be part of the fact that the bar gets higher, or something, mumble mumble.
Every year I've noticed the questions get harder too. A hard question in 2012 (off the top of my head, 2 pointer solution to linked list cycle detection) might be considered an easy question today.
From what I have read from Gayle Laakmaan, they know this throws away many good engineers, but it is acceptable to them because it doesn’t let bad ones through.
I've definitely repeated that line in the past to justify tech industry interviewing practices - but sometimes it goes beyond "setting a high bar" to something more toxic: an interviewer using the interview as a chance to show off how smart _they_ are, rather than assess the candidate; deliberately creating high-stakes "pressure cooker" environments because, you know, that's just how we work here and they should get used to it; and so on. These things then get rolled into the same justification: after all, we're an exclusive club of ultra-smart 10x ninja pirate Jedi, which of course means we should have the default assumption that others don't belong in this club.
I wonder now: for every "bad one" not let through by this sort of process, how many other "good ones" look at it and say "no thanks"? How many hear about this sort of hazing and are dissuaded from applying in the first place? How many experience it one too many times and just quit the industry altogether? How many of those "bad ones" are even objectively bad, and not just having a bad day or intimidated by a process rife with both intentional and unintentional hostility? In other words: are we actually assessing what we claim to assess with _any_ predictive accuracy, and is the collateral damage to company reputation and the pool of available candidates - a damage often hidden to the individual companies that impose this process - even worth it?
I think it goes far beyond that. I'd bet dollars to donuts that you will find both protected and unprotected groups over represented in the false negative pile. Do women have more test anxiety then men? If so, guess what your process is selecting for. Do minorities have more self doubt than majorities. Same thing. And so on.
And that doesn't address the file drawer effect. I'm 54. I can code a binary tree, hash table or what have you, if I have to, but I am not as practiced as somebody fresh out of school, because that heavy lifting is done by libraries, and I'm busy contributing original mathematical algorithms that no one ever asks about because they don't have the background to understand the explanation. So I don't go on FAANG type interviews. I know I will fail, and if I don't my offer will be predicated on that apparently lower performance. At best I will have an utterly miserable experience. So I self select myself out, mostly on age.
> I can code a binary tree, hash table or what have you, if I have to, but I am not as practiced as somebody fresh out of school, because that heavy lifting is done by libraries, and I'm busy contributing original mathematical algorithms that no one ever asks about because they don't have the background to understand the explanation.
This made me chuckle, because it's absolutely how it is.
Some other comments here talk about how nobody creates new algorithms these days unless they are a researcher. But it's not true at all! Like you I'm creating new algorithms and other tricky techniques all the time, but I'm not fresh on the stuff I learned at school... or so I thought.
I have been really surprised to find some screening interviews recently asked me shockingly trivial questions. So simple that I stumbled over myself trying to simplify the answers, thinking "I know too much about this topic and need to keep it simple", and "can these questions really distinguish candidates?".
I did the TripleByte online test recently and found the questions much simpler than I expected, including those in languages I've never seen before. That is not TripleByte's reputation. I see people writing about how difficult they found the questions. (Admission: I didn't score all 5s, but I can't figure out why unless it's timing as I think I answered them all correctly.)
So I'm thinking, perhaps it's not so bad, people just make it sound bad because there's a wide variation in people's knowledge, abilities and expectations.
If you are thinking you might apply for something like a FAANG or other hard-reputation company, and feeling put off by the horror stories, I would say, just take a look at the old stuff a bit for a refresh, then give it a try and you might be pleasantly surprised to find their reputation is because people less skilled than yourself found it hard. Their "hard" might not be hard for you.
You'll probably still fail the interview if it's a FAANG because they are so selective, but I would bet my dollars to donuts that it would be more refreshing than miserable if you're not attached to passing.
I know your point isn't about yourself, it's about bias in hiring, including bias due to perception by the candidates, but I wanted to address that side point about feeling there's no point applying. That sounds like anxiety to me. (I have it too, I'm trying to get over it.)
>And that doesn't address the file drawer effect. I'm 54. I can code a binary tree, hash table or what have you, if I have to, but I am not as practiced as somebody fresh out of school, because that heavy lifting is done by libraries, and I'm busy contributing original mathematical algorithms that no one ever asks about because they don't have the background to understand the explanation.
Ooohhhh this is frustrating. Nothing is more frustrating than telling an interviewer about something cool you've done, and realizing they don't understand, but also don't care.
Yes, I remember one guy asking what I would when a page was going slow. I explained how I would work backwards checking the network tab in the browser, make sure it was the endpoint that was the bottleneck, blah blah blah to the database. He just wanted me to say "use explain" - which i would have got to if he had bothered listening a couple more minutes. Then he wanted me to do a "challenge" that was expected to take around 10 hours. No thanks, you didn't convince me I want to work for you.
But they're not aware that the filter is just anxiety.
Google is basically using anxiety to filter good candidates and eliminate false positives which works in a sense but is still highly illogical.
Why not use a technical filter to filter for technical candidates? Anxiety seems like a pointless filter.... how does that even eliminate false positives?
it would seem it might be 'letting in' a whole load of people who don't have appropriate anxiety responses. There are some situations where anxiety is perfectly normal, and perhaps even useful, but if you screen out people who have normal anxiety, what impact does that have on your company operations and culture?
I'm a pretty anxious person, so much so that I've pretty much retired instead of doing anymore technical interviews. I had a discussion with a very anxious, but brilliant software developer a few years back and we came to the conclusion that as long as we don't let our anxiety get too far out of hand it's actually kind of a superpower in the job setting. That's because that niggling anxiety you get in the back of your head as you're coding generally makes you more careful about how you're designing and testing your code. You're more careful about security, safety and correctness. Whenever I get anxious while coding I stop and ask myself if maybe it's a message that I need to listen to - maybe it's trying to tell me to tread carefully because I'm entering an area that's potentially problematic.
People without that niggling sense of anxiety always in the background, the folks who are easily going to ace the programming interview because their anxiety levels are so low, those folks, I theorize, are potentially not going to be so careful. Now you could argue that that's a plus in many situations - a startup that needs to get code out the door right away, for example. But it really depends on the domain. For critical systems in applications like healthcare, avionics or robotic control I think the anxious coder is the one you want.
Companies that completely weed out the anxious programmers, as you imply, will have a different culture with perhaps too much emphasis on risky behavior.
> But they're not aware that the filter is just anxiety.
Could this be deliberate? As working with people with anxiety problems is often a pain in the ass as they won't report problems for fear of seeming stupid or ask for help.
Confident people are far easier to work on a team with.
This is essentially the point of a lot of hiring and performance evaluation theory.
E.g. it's acceptable (but not ideal) not to promote talented commanders. It's catastrophic to promote someone to General who isn't ready or is unsuited.
Software engineers are not commanders or generals and it is NOT difficult to fire someone in the United States of America. Everyone in this industry has a at-will contract.
However, high rates of employees getting fired is bad for culture. People are always anxious about their jobs - if they see a few colleagues fired for nonperformance, a number of them will be driven to anxiety. (I am one of these people, and being anxious about job security ironically makes me much worse at my job)
One of the things I've liked about freelancing/contracting is that I know I'm going to get "fired" (we call it "successful completion of the project" but its the same thing, just that they would like to have me work for them again). And since it happens on a regular basis, I also am practiced in finding a new job and I know about how long it will take and what types of prospects to look for. It's really reduced job anxiety for me, although I never had too much.
Although as a counterpoint, it's also easier to get work, because part of the value I'm providing is that you "fire" me whenever you want and no hard feelings. So nobody does "leetcode interviews", because it's not like we're getting married, like an employee. So there's much lower risk to hiring someone, because a bad hire doesn't infect anything, because you expected it to be temporary in the first place.
Right but I also do think that FAANG is over-estimating how many of and how bad these "bad apples" aka. slighly above average developers would be and how terribly difficult all the work they do is.
When you have your pick of the bunch, of course you're going to look at slightly above average as "bad".
I'm not sure how much engineer skill matters though. One slightly above engineer might be fine. But if half your org is made up of slightly above average engineers, I feel like there will be a knock-on effect.
I don't know maybe my view of slightly above average is inflated but I think of that bunch (which I might be a part of) as being easily pulled in by good technical leadership. If only to make their life ultimately easier / less painful. You don't need 60-70% amazing engineers to do that. But I guess they can do whatever they want. They have infinite money and unfortunately a good reputation (in many cases unwarranted). They've become like the Harvard or Yale of tech. Oh you went to Google, great come on in and join us.
And yet, time and time again we see that Generals promoted during peace-time rarely have the skills necessary to lead men under intense combat conditions. The best generals are identified on the battlefield.
Coming from a military background, I can confidently say that promotions at that level are more the end of a political process than a true assessment of skills...
Over the past few years in tech, I have interacted with and observed (as a fly on the wall) a wide variety of recruiters, hiring managers, etc. from companies as small as a dozen (usually around the time a founder/CEO brings in external "support").
There exists an incredible amount of arrogance and ego among them by-in-large. The best are humble and curious while maintaining clear and direct communication. To be fair, they are often tasked with making important judgement calls sans complete information, though that's the fault of the leadership/culture as to a poor approach.
At the start of my third year as a student, I biked through a record-worthy rainfall to my first interview with a small local games studio. Showed up completely soaked, but sat down relaxed and feeling reasonably confident. (I had done a handful of unity and opengl projects.)
The interview started with a bit of small talk, I chatted with one of the guys about Brood War and Starcraft 2. Later during the technical interview they asked me about the difference between private, protected, and public and said I was the only student they had interviewed who had answered correctly which was wild and honestly stunned me for a moment.
They liked how in one of my examples of scripting something I had written some fun dialog and I mentioned I did writing as a hobby which seemed like a plus. I talked about which classes I enjoyed the most and how they were challenging/interesting.
I did not get a job offer and learned from a friend that I had come off as "depressed and disinterested" (I don't think they realized he was going to relay that info to me...) All I can do is guess that it's a combination from showing up completely drenched and sharing how I enjoyed those "challenging" classes made it seem like I would get quickly bored of scripting...
After that, I bombed an interview for a QA role because I went into it completely unprepared for just how much it would differ from a programming interview.
Point being those first two experiences got in my head and I basically had anxiety over anything job-hunt related for the next 2.5 years.
When I did get a job, I was extremely happy with the interview process (even though I felt I did poorly on it). Here's more or less how it was structured (TL;DR):
- Office tour/chat with a programmer who had referred me
- quick 5 minute introduction to senior programmers in charge of the technical interview
- 1 hour alone in a meeting room with a laptop and 4 written questions (a generous amount of time)
- ~15 minutes reviewing my answers with the senior programmers
- importantly, they gave me a chance to talk about my answers and when I got something wrong they would simply state that it had an error and see if I could spot it
- 15 minutes on C++/memory/performance/behavior quirks (important stuff in AAA games)
- ~30 minutes talking about stuff I had worked on
- occasionally they would mention something related I hadn't heard about and explain it while gauging how well I could follow along
Basically, based on my own interview experience/anxiety if I had to choose an interview method, it would be very similar to what I just mentioned. Seeing a familiar face and the introduction followed by the alone time did a lot to ease my anxiety. The time where I felt most comfortable was talking about the stuff I had done because I was very familiar with all of it/it was easy to talk about.
The process seemed less focused on where I had gaps in my knowledge and more focused if I had a decent amount of knowledge in general and if I had the ability to recognize and correct the gaps in my knowledge.
Sorry if it's a bit of an info dump, but it's something I think back to a lot.
I am focusing on the algorithmic part, which is the only thing I can influence in my company's interviews. I wish we would evaluate candidates with some pair programming, but that's never going to happen.
> incredibly high stakes algorithm riddles
Yeah there you go. This is your problem (Google/Facebook I assume?). It's more than questionable to expect candidates to solves complex algorithmic problems on a whiteboard in 45 minutes. You should ask moderate algorithmic questions that offer many solution approaches and accept solutions that are "good enough". It is not about having a candidate gaining some magic insight that some MIT student had 30 years ago during his master thesis (and reproducing it in 20 minutes lol). It's about giving a reasonably complex problem, that is not too easy but also not too difficult and allows you to focus evaluation on:
* Does the candidate write proper code that isn't far from compiling? (If they can't, they don't have the experience. It's like not being able to write without a spell/grammar checker. Small mistakes are fine. Big mistakes indicate lack of understanding.)
* Does the candidate have a structured approach to problem solving (all the time I see people starting to write code immediately and getting completely lost in what the problem even was. This is a red flag to me and baffles me each time again.).
* Does the candidate debug his code and walk me through. Does he find obvious bugs while doing so and can he convince me that his code works? (If not, and that happens often, its another red flag)
* Can he rank the speed of his solution with other theoretical solutions? Let's say they found an N^2 algorithm. I usually ask if there is anything faster (even if there isn't). This shows if they have some decent fundamentals in CS and are able to think about the boundaries of an optimal solution and how far they are from it. This is something, people without a CS major usually can never do and unfortunately also not too many with a CS major. It's kinda relevant though, if you optimize for performance and have no clue what the theoretical limitations are, then you are grasping for straws.
There is one big secret for getting A LOT out of easy questions:
I start to modify the problem statement and see if they understand how this changes their solution and their algorithm. People who don't have a good grasp of CS will fail miserably at this task, because this isn't something you can memorize.
You've just slightly improved upon a very broken way of interviewing engineering candidates. I've been paid by clients happy with the results to do everything from cryptography to writing a compiler / interpreter. Very computer sciencey stuff at times. The devs I work with consistently rate me highly and I often get put in mentoring positions. I couldn't tell you what N^2 is. I learned it in uni. In decades of programming it has come up in many interviews and never once on the job. Maybe in your industry it's important. But if not, please consider re-imagining your interview process. Having hired many developers over my career including at one time for my own business I can assure you there are far better ways at getting high quality signal out of candidates than trying to improve upon what you experienced as a candidate.
"N^2 is bad, except when it isn't" is my take on it.
In uni I maybe had one week of doing O-evaluations, it never came up and I never got interested. I've seen 20,000-word debates on Reddit on whether something is n, n2 or logn... and to my knowledge nobody ever learned anything from those.
In the workplace I've several times ran into the problem of "This is taking way too long," developing tools and methods to measure and drill down, then figure out if it can be improved upon or should be left as-is for now.
Honestly discussions and articles like this leave me absolutely terrified of interviews. My first and only technical interview was my future boss leaving me alone for 30 minutes in a room with a laptop "Code something you like yourself."
> In the workplace I've several times ran into the problem of "This is taking way too long," developing tools and methods to measure and drill down, then figure out if it can be improved upon or should be left as-is for now.
To play devil's advocate, someone with the word Senior in their title would probably have command of recognizing Big O issues, and skip the "this is taking too long" step, and code it efficiently the first time. This is industry dependent, as well, obviously. I've been doing low-latency C++ for 15+ years, and it's not really something you can compromise on.
The mistake, I think, is skipping otherwise good candidates because they don't immediately see these issues. We should put more emphasis on identifying these weaknesses and finding the right mentors to teach them up. We should be hiring more junior and mid level engineers, with the assumption that they will learn and be up to speed in a few years. This has been my approach to interviewing, but I'm often vetoed by other team members.
The problem with Big O complexity is that it just isn't the reason most things are slow. Usually, your input is either so large that you physically can't run a polynomial solution, or your input is so small that it doesn't matter. Speed increases usually come from caching, avoiding indirection, removing slow API calls & properly making use of your hardware.
pivotal does pair programming interview. The morning with one person in a theoretical problem and afternoon with another in a real problem.
I liked the experience and the process, in the morning it was a blast. We really clicked.
In the afternoon I had to use the same setup as the guy I was collaborating with -- Vim with his bindings and Golang. I had never used Golang before and even tho I was proficient with vim his binding were quite opinionated.
I felt the guy from the afternoon had 0 interest in being programming with me. He wasn't collaborating and was looking at his cellphone the whole time.
The problem was to write a functional test for a CLI that would spawn a web service in Golang, parse some json and run some assertions.
Golang had some quirks with JSON and new line encoding that I spent my whole afternoon with. It is something he could have unblocked me as it a very specific problem, instead he kept telling me to read the official golang docs, not even to google the error.
As someone being interviewed I followed what he asked me to do and worked in his environment but it wasn't a representative of how I would have worked.
That said, I feel it could work! I just should be able to judge the person who pair programmed with me instead of just being judged :) the person in the morning would be a 10/10 and in the afternoon a 4/10.
I'm pretty angry with the person that you were. I'm frustrated with the people who look up a question, see the exact solution and walk through and then expect the candidate seeing the problem statement for the first time to walk through it in the same way.
There are questions being asked that full academic papers have been written on. What's even worse is typically I get interviewers that are not experienced+prepared enough to give hints or work with you.
I am happy that you've had an epiphany to see where you were wrong. If you ever are in a situation to build a quality team. Interview people for the skills to work with others+ teach others, look to optimize, experience, and that they're willing to learn. You'll have a fairly good group of candidates to pick from as that don't do well in these coding interrogations.
I bomb on technical interviews the same way because my mind just freezes and can't get through even simple problems.
I had the same question you mention. "This is a problem a paper was written on, are you expecting me to derive a whole paper from first principles in half an hour, or are you expecting me to have seen it before? If it's the latter, why not let me Google, as I normally would?"
I feel that saying "I would solve this problem with <algorithm>" should be good enough, and if I'm asked to implement it, I should be able to copy paste the Wikipedia pseudocode and start converting.
> I had the same question you mention. "This is a problem a paper was written on, are you expecting me to derive a whole paper from first principles in half an hour, or are you expecting me to have seen it before? If it's the latter, why not let me Google, as I normally would?"
> I feel that saying "I would solve this problem with <algorithm>" should be good enough, and if I'm asked to implement it, I should be able to copy paste the Wikipedia pseudocode and start converting.
Long ago, milo.com posted a challenge to codeeval.com (both websites now dead) which turned out to be an example of the "assignment problem". That is, given a set X, another set Y, and a payoff function f(x,y), assign each element y in Y to exactly one element x in X such that the total payoff from all assignments is maximized.
I tried to solve it, realized I didn't know how, and was eventually able to look up the solution in a couple of papers. There is an algorithm called "the Hungarian algorithm" which will do it.
So I wrote up the Hungarian algorithm and sent it in. This was good enough to get a phone interview with the company.
I tried to prepare for the screen by going over the Hungarian algorithm to the point that I could myself give the proof that it correctly solved the problem. This was fun. But in the phone interview, it was barely mentioned at all. According to the interviewer, I passed that step by just mentioning the name "Hungarian algorithm".
That's better than asking you to come up with the complete solution, but it's still a bit of a trivia question. On one hand, it's good to be able to recognize algorithms so you can look up their solutions instead of wasting your time on trying to reinvent them, on the other hand, how often are problems this algorithmic?
Maybe I've had a particularly depressing career, but 99% of it consisted of various ways of interacting with a database.
Yeah, in certain circles the hard part is figuring out the solution is the assignment problem. Then you’re supposed to just know that the most commonly used algorithm for it is the Hungarian algorithm, maybe its complexity and that’s it, because you can find code for it in most languages and so you just need to call the function with appropriate inputs.
Well, I found their challenge because they posted it to HN.
Everyone inside Milo seemed to think that their hiring process was hugely better than what happened elsewhere, including by producing better hires. But the approach -- set a difficult challenge instead of an easy challenge, such that, if someone passes the challenge, you will almost certainly hire them -- still seems to be vanishingly rare.
Oh, was that their logic? I don’t even recall if they actually interviewed me, despite passing their challenge. As I recall, I was rejected for not enough experience.
I find that hard to believe, since I was hired despite experience totalling less than a year of internship.
However, there might have been external reasons. I was hired after they were acquired by eBay, and apparently they had preexisting approval for a certain number of hires that year that they didn't end up meeting. If you were earlier or later (pretty likely!), maybe they might not have been so cavalier.
That could be. I could also be misremembering. In any case, I definitely wasn’t offered a job, despite solving the problem in exactly the same way you did.
I’m not complaining. Who knows? Getting rejected might have been for the best.
They're not looking for someone that is eager enough to show they're willing to work through it even though they have no idea to proceed.
They are looking for someone that got lucky enough to be ready to answer that particular problem, but won't stand up to them that the ask was a bit too much for a reasonable interview that is intended to see if you're a fit for the company and skilled enough to get the work done.
They aren't looking for someone that will dress them down to tell them that's incredibly insulting to ask something that took an academic a long time to come up with a solution for that. [Although a rough approach and fairly unfriendly. That shows a lack of desperation, confidence on the knowledge, and a good understanding of the difficulty of the task at hand (heck that's a good signal for won't underestimate points during planning)] (There's a singly linked list question that amazon used to ask that qualifies for this)
Exactly agreed, and because my inclination is to say "I've never used this in previous jobs and will never use it here", I don't think it's useful for me to be interviewing for jobs.
A friend of mine, when asked to implement an AVL tree, actually asked an interviewer when the last time he had to implement an AVL tree on the job was. He wasn't hired.
They must be willing to eat their own dog food or else the question won't make any sense for the interview.
The department should run the interviews through their own employees first. If some questions cause an "interview anti-loop" (a set of other employees S who would not hire E) it's time to revise those questions.
Revise and rehearse these practice interviews within the department, and do it blind, until all of S are willing to "hire" each other for their respective roles.
I don't think it's that complex. I think they're just cargo-culting. Technical interviews were designed to test certain things (brainpower, grasp of fundamental data structures) and not other things (workflow, ability to write out large volumes of trivial code). I think companies just copy that and don't realise what they're really testing for.
Don't forget, not only are they looking for someone who already just knows the answer, they frequently aren't hesitant to penalize someone who says "Fair warning: I've already seen this problem."
You're misunderstanding what they're trying to test. They're testing your raw ability to manipulate algorithmic structure in your head. It's important that you have to visualise the data structures and manipulate them without being able to look it up. It's an brainpower test, not a workflow test.
Which is fine if your job is really going to be centred around developing algorithmic software in a unique or high pressure way. The problem is most software engineering day jobs are nothing like this, so the test is not a good filter.
I always say that you aren't qualified to work on the hardest problems you can solve, so to check if someone can do the job you need to ask them to do something harder than you expect them to encounter. People who work at the edge of their ability makes tons of mistakes and are very slow, and it isn't very fun for them either.
It is the same as if you expect a warehouse worker to regularly lift 20kg then you don't test your hires if they can take 20kg at their best, because then you will get a lot of people who simply can't do the job for 8 hours a day. Instead you'd make sure they can lift 40 kg or so to ensure that lifting 20 kg regularly wont be a problem.
Yeah maybe, but these hardcore algorithmic "leetCode" things are like asking the warehouse worker to design a logistics system instead of seeing if they can carry 20kg boxes. The vast, vast majority of software jobs out there involve more soft skills (requirements, planning, scrum and other processes) and "plumbing" libraries/frameworks/data layers together than implementing algorithms from scratch. I'm not saying you'll never have to do it, but it's going to typically be a small proportion of your day job, so why test this stuff so much at interview stage?
The point is that designing an iteration that does something should be a trivial operation requiring no thought at all. If you can leetcode problems then we can be sure that you can do loops without effort, otherwise it might be that you struggle with them and as a programmer it will really hurt your productivity. Testing if you can do loops doesn't test if you can do loops easily, so we test something requiring a bit more thought.
Edit: And I think that you'll find that most leetcode problems are pretty easy if you are an expert at doing loops and recursions.
> I'm not saying you'll never have to do it, but it's going to typically be a small proportion of your day job, so why test this stuff so much at interview stage?
It isn't a small portion of your time spent at Google at least. Almost no engineers at Google ever talks to non technical people, product managers or engineering managers does that. Most engineers works to reduce cpu usage of backend servers or build scalable features for backend servers or similar stuff. Even high level decisions like deciding what service to use requires more engineering hours in time spent refactoring the thing than engineering hours spent deciding what to use, so there is much greater need for people who can code than people who can make decisions.
And if you argue that Google is bad at making new good products, that isn't the fault of their software engineers since they don't decide what to build. Instead it is the fault of VP's, and those don't do white boarding interviews.
So much agreement here. Who hasn't been rejected by some 25 year old who's been with the company all of 6 months and who asks silly questions straight from LeetCode? I have literally been asked questions that were PhD theses 30 years ago.
How is any of that supposed to be part of a good interview process? My only solace here is that those things are red flags on the company side, so I might have dodged some bullets.
Wouldn't it be a more balanced approach if the technical problem were presented on the spot to __all__ interview participants, not just the candidate?
It's a technical job, it must be a collaborative effort in finding solutions! Aren't we looking for a team member, not a lone-ninja.
Sure, the candidate has to be more vested in advancing the process, but the fact that there's no known-ahead answer would potentially surface not only technical skills, but also personal ones, and ... at both ends of the table.
Not to mention, the "house-experts" could find such sessions stimulating, to say the least... Imagine, you get back to your desk after interviewing a candidate, and your team mates ask you "so what was the tech problem this time?", "could you crack it?", "did he/she crack it?".
I like it. However, I have had an interviewer at a "top chicago tech place" (roll your eyes on that one) He basically did an inperson imitation of leetcode. Say problem, look at you. Don't give help when asked, stay emotionless.
I worked for a FANG and conducted over 100 interviews. Most of that time I thought I was asking really hard questions and wasn't sure I would have been able to answer them if I hadn't seen the question before.
I later interviewed at a different FANG, surprised myself and answered harder questions than I typically gave and answered them better than most of the people we hired.
That sounds like when you spend a lot of time exposed to and walking through certain types of problems led you to getting good at solving those types of problems.
Then I interviewed for another company and utterly bombed. It became suddenly clear to me that I had been an idiot. Of course nearly all of those candidates were perfectly good programmers. They had had shit interviewing days, probably mostly due to nerves, but they probably would have mostly been perfectly good employees. How frickin' arrogant I had been for concluding that people who couldn't solve incredibly high stakes algorithm riddles on a whiteboard in 45 minutes with enough speed and flair were somehow not qualified to be my coworkers.