The Matasano hiring process (to which this article refers) was designed in part because of this bad experience you've never had. I've seen it happen repeatedly over my career; people who'd pass interviews and then, over 15-20 minutes of face-to-face discussion a couple weeks on the job, not be able to comprehend the idea that a C pointer needs to point to valid memory.
That's what's so mind-blowing to me about our terrible process today. It aggressively tries to find and dispatch unqualified candidates, so much so that it forcefully repels talented people. And it fails miserably at screening out people who can't code! It is the worst case scenario of processes.
It's also really important to remember that the problem is People Who Can't Code, not "People Not Smart Enough To Code". Time and again I've "worked" with people who could very effectively critique my own code, who could participate in design meetings down to the bits-and-bytes layer, and, having taken responsibility for actually implementing some portion of those designs, could not commit usable code to save their life. I don't know what the deal with these people is, but they are out there: plenty smart, totally fluent, and completely ineffective.
I was talking to Erin this weekend about the phenomenon and realized that these people not only exist, but are actually favored by the way the market works. A smart person can last many months in a dev role before anyone realizes they're ineffective. Then, it takes many, many more months to manage them out of the team. By the time the story plays itself out, the bad team member has more than a year on the job, and when they parlay that resume line item to another prominent job elsewhere, their previous teammates aren't outraged but instead relieved. They end up with gold-plated resumes and tons of credibility.
The software industry is broken for older engineers. Look at the typical algo/CS heavy interview today. It favors and selects for these characteristics:
1) Young, fresh graduates who still remember well the CS material. Or those who have a lot of free time and can study for these interviews (for example look at this Quora link http://www.quora.com/How-much-time-did-you-spend-preparing-f... the first reply mentions spending 28.5 hours per week for 5 weeks, which is 142.5 hours total) From a senior software engineers perspective that's a ridiculous commitment for something where an arrogant asshole interviewer can shutdown a whole onsite interview process because they're having a bad day or have some favorite question they think is the One True Question for determining if someone is a software engineer.
2) The software industry tends to overvalue youth. It has a very high salary at start, but also a very quick salary cap. And most companies have no technical ladder, and even those that do it's a joke compared to the manager ladder. All of this is not surprising. Since the industry is so heavily biased towards hiring and rewarding youth, older experienced engineers are not valued.
3) Senior software engineers have heavy pressure to leave the software field. I'm feeling this now. I'm a senior at my current place and my salary is basically capped. There would be little to no benefit going to another company. And even if I wanted to, I have a family now and don't have
the free time to study for software interviews. My options are basically go into management, consulting, or start a business. I have no desire to start a business or go into consulting because I simply don't like doing those business related things. I want to be an engineer not a business person. But if I just stay an engineer, I will probably get laid off someday due to ageism. And if I'm an old engineer who is laid off, job mobility is basically gone due to ageism and the CS heavy interviews that I don't have the time or motivation to study for. So the only safe path is management. A lot of senior engineers face this decision. Which again reduces the number of mentors and experienced engineers to teach the new younger engineers.
4) And doesn't anyone notice just how wrong whole concept of studying for an interview is? There's a whole book publishing industry dedicated to technical interviews such as Cracking the Coding Interview. The technical interview has become like the SAT. Many could just study to pass it, but be horrible engineers.
Interestingly, the author of that book also wrote the top comment on the TechCrunch story, disagreeing pretty strongly with me about hiring processes. I got to talk to her a bit about it on Twitter:
It seems to me that Google's perceived hiring issue is "we can't tell the candidates apart, because they all studied our interview questions they found online." (and not "our process is fundamentally broken to the core" like it probably should be).
In that context, "give take-home tests" and "ask better questions" sound like the exact opposite of good answers.
I think this is a problem that's unique to very large corporations. Not only do thousands of people interview there, meaning questions and answers are likely to be traded verbatim online, but there's also actually an incentive to cheat your way in: you can hide out for 1-2 years and acquire a horde of cash and resume fodder.
Google have actually made this problem worse for themselves, too. By overvaluing their interview process, they've made cheating even more lucrative. Cheat your way into even an offer there and you have something valuable to put on your resume!
I suspect you haven't seen a lot of cheaters on take-home tests before because at a company like Matasano, there's little incentive to cheat your way in. You certainly can't hide out in a corner doing no work there.
Bummer. Her career seems to be predicated on a correlation between algorithm tests and performance. Can't expect her to seriously entertain disagreement as long as the current model is profitable for her.
Google is no more a proof of the value of algorithm tests than Github proves the value of manager-free organizations.
I was a bit horrified at the acquisition coaching comment. I did not realize that happened but of course it makes sense. Can't manage an acquihire if your devs can't pass the algorithm tests at the big acquirers. Shame.
It is actually more profitable for me if I say that these algorithm tests are BS and completely study-able, and even the worst engineer can pass them. That easily translates into "here! buy my book and even YOU can get a job at Google!"
It's actually a much harder sell when, really, it's more like studying helps someone perform better, but only to an extent.
Oh, and the acquirers encourage studying and prep, just as most of the top tech companies do for their normal dev (and non-dev) candidates.
To be clear: my experience is that the algorithm interviews, when done properly, work reasonably well to identify intelligence/problem solving and coding skills. Those are more important at some places than others. They can work well for certain places -- but not at all. I do not think that all companies should use this process. It's not right for all places, but it's reasonably good for some.
As for Google being proof of the value of algorithm test: I think you're pulling that from a twitter conversation very much out of context.
Context: Some people were arguing that the algorithm interviews offer zero value - that you'd be at least as well off, if not better off, hiring at random.
If they truly believe that (and they appeared to be sticking to this argument), then it's absolutely relevant that not just Google, but basically all the top tech companies, use this hiring process. Could a randomly selected group of engineers build the technology at companies like Google, Facebook, Amazon, etc? I suppose that's what they must believe, and I find that pretty absurd.
If they had argued that these interviews sort of work, but aren't nearly as good as some other interview processes, then you're right that the company about Google and other companies wouldn't be relevant. And I also wouldn't have brought it up.
"To be clear: my experience is that the algorithm interviews, when done properly, work reasonably well to identify intelligence/problem solving and coding skills."
Given that there is no correlation between job performance and interview score, are you saying this based on your gut instinct, or do you have objective data somewhere?
My best estimation is that the Google interview process is just a glorified fizzbuzz presentation, and that the hiring decisions are more or less made by arbitrary factors, including the nebulous "culture fit".
While it may be amusing to work through the details of doing a merge sort on a linked list in 30 minutes, it's certainly not germaine to the quotidian experience of an SWE at Google.
>> Given that there is no correlation between job performance and interview score, are you saying this based on your gut instinct, or do you have objective data somewhere?
That's not a correct assumption. I suspect that you are referring to the (infamous) Google studies on this that were alleged to show this. The articles about this got many details wrong, and the study was fundamentally flawed. Remember: they only studied the people who did very well. That's like finding no link between exercise and health when you only study people who run regular marathons.
Merge sort on a linked list is a bad interview question, so I agree with you there. Good questions are problem that actually make you design a new algorithm.
It may be that the studies were flawed, and that the reporting was bad, but that flavor of study has been replicated in other venues besides Google. Given that, it essentially comes down to culture being the deciding factor in these interviews. Given the stark cultural skew at Google, it's possible that the pseudo-analytic SWE interview is actually masking systematic bias in the hiring process.
The question for Google is whether Page, Brin, Bak, Buchheit, and Dean passed an algorithm interview before building Google's core technologies. Hiring doctrine after the fact does not ensure a billion dollar company.
I disagree with your assertion that lambasting Google's hiring practices is an easy sell in the market. The last time I tried I was dismissed out of hand by a talent consultant.
> I disagree with your assertion that lambasting Google's hiring practices is an easy sell in the market. The last time I tried I was dismissed out of hand by a talent consultant.
>And doesn't anyone notice just how wrong whole concept of studying for an interview is?
This has been going on for ages. A lot of coding bootcamps and similar things are just interview prep programs. A lot, I mean a lot, of people I have worked with can nail interviews but can't do real work for shit.
Unfortunately to land a job, you have to go through this process. Developing interview skills is parallel to being able to do actual work.
If your engineering org is 50-weeks-a-year structured around delivering stories and fire-fast and doesn't build in time for ramp-up, it has a good chance of spitting out ineffective hires. But most organizations --- and we worked with lots of them as a high-end engineering consultancy --- aren't structured like this. More importantly: it's disproportionately the larger organizations with bigger brand equity that tend not to be run entirely out of Pivotal Tracker, which feeds the resume effect.
At many large organizations, there's even political pressures that keep employees from being let go early in their employment. If a hiring manager says yes to a candidate and they go through training/onboarding, there's a certain amount of clout to be lost if you jettison that employee early on. It is, instead, better for the manager to make the employee appear to be productive to make it seem like the manager did a good job in hiring.
Team accomplishments can be embellished, stronger team members can cover for the weaknesses of weaker team members, but admitting that a decision you made was incorrect gets punished. It's obviously better to hire someone competent in the first place, but the next best alternative, for the manager, is to live with their bad hiring decision.
I used to work at a company with thousands of staff, and the HR policy was pretty clear:
As engineering managers, we were expected to "manage out" about 10% of the lowest performing staff on a yearly basis. The way that worked was basically that the 10% under-performers were to be put on strict personal improvement plans, and were under no circumstances given raises at all, and it should be made clear that no raises would ever come their way unless they improved.
That was it. Sounded harsh if you were used to 10% a year raises, as many of the good performers got. Not so harsh for those who were used to live in fear of getting "exposed" and fired.
And as a manager, if I were to designate a lot of my team as under-performing come review time, it'd reflect badly on me, so managers also had an added reason to push for good reviews for their reports: You'd end up getting paid more by getting good reviews yourself if your reports were helping create a good impression through rankings that you as a manager could trivially fudge (we did "360 degree reviews" were the manager and report could both designate some co-workers to review the person, which was of course trivial to game for those who wanted).
There was also an implicit expectation of a bell curve. If you had an under-performing team, that'd be awesome news for at least some of the weaker members on your team (and conversely, I several times had reports who got short-changed because I was told too many of them were designated "above expectations" even though my manager and other people who had worked with them agreed, and so several of them were ranked lower to get the "correct" spread - it was absolute lunacy).
Unsurprisingly, come hiring time, I felt like we were under siege by an army of ineffective developers hoping to find refuge somewhere safe and hard-to-get-fired.
There are scarily many places like this for ineffective people to hide. Sometimes for many years.
"comprehend the idea that a C pointer needs to point to valid memory"
I ran into a C programmer that did not get the relation between pointers and parameter passing. It was some scary awful code. Damn prep for code review almost got me in a car wreck.
Is there a process in place to handle such people? Do they get asked to leave (shortly after being hired), or are they moved to a role which doesn't require programming?
That's what's so mind-blowing to me about our terrible process today. It aggressively tries to find and dispatch unqualified candidates, so much so that it forcefully repels talented people. And it fails miserably at screening out people who can't code! It is the worst case scenario of processes.
It's also really important to remember that the problem is People Who Can't Code, not "People Not Smart Enough To Code". Time and again I've "worked" with people who could very effectively critique my own code, who could participate in design meetings down to the bits-and-bytes layer, and, having taken responsibility for actually implementing some portion of those designs, could not commit usable code to save their life. I don't know what the deal with these people is, but they are out there: plenty smart, totally fluent, and completely ineffective.
I was talking to Erin this weekend about the phenomenon and realized that these people not only exist, but are actually favored by the way the market works. A smart person can last many months in a dev role before anyone realizes they're ineffective. Then, it takes many, many more months to manage them out of the team. By the time the story plays itself out, the bad team member has more than a year on the job, and when they parlay that resume line item to another prominent job elsewhere, their previous teammates aren't outraged but instead relieved. They end up with gold-plated resumes and tons of credibility.