I don't agree with this. I think the issue here is that your interviewers aren't going deep enough in the conversation. If you ask someone to tell you about some interesting technical project they worked on and they give a good high level answer, push a level deeper. Still get a good answer? Push another level deeper. Repeat. At some point one of two things will happen: The candidate is capable of talking about the project intelligently at a lower level than the interviewer or the reverse. If the first happens, re-think whom is doing the technical interview, if the later happens, you've found their level of competence.
In practice, though, what really seems to happen is that technical interviews are split between whiteboard-grade programming puzzles and "friendly deep discussions" about technical problems that both (a) can be prepped for with rote memorization and (b) are extremely game-able by people with strong interviewing skills (for instance: the skill to respond to questions with questions, or to redirect the course of a line of questioning back to safe ground for the candidate).
It's not like we were asking candidates "hey, how did you enjoy your last software security job?". We were doing something closer to "here's a piece of code with a heap overflow in it; talk us through how you'd write an exploit". It didn't work: there were people who could answer those questions indistinguishably from an exploit developer who nevertheless couldn't write a 1995-era stack overflow exploit if their life depended on it.
I'm a little confused by this statement. Is your issue that they aren't _capable_ of writing the code or that their job performance isn't there once hired? i.e. Are the technical chops not there or is the work ethic not there? It seems like you are describing the later to me, or we are talking about different levels of conversation.
More specifically, you make so many small executive decisions when coding. Naming, how functions and classes are designed, which file/module this goes in, is this already tested, oh I found an unrelated bug, do I make a ticket, a new PR, or include it in this one, etc etc etc. The only thing I've seen that can answer those questions is homework--maybe previous work, but homework is better.
But I think you're remiss if you're not asking your senior engineering candidates how they make those decisions, not because the answers to relatively trivial questions are important, but because you should know--and they should be able to articulate--how they make those decisions, and they should know the pros and cons of their approach and other approaches.
Something about this example is not registering for me. I can't imagine how this situation is possible. Going from a detailed, step-by-step description to a code implementation is a trivial rote task in my mind. I must be missing something.
There's definitely tons of senior devs out there that care way more about linting and unit testing than they do about writing the code that implements the spec.
This guy couldn't deliver a single project and would spend days debugging issues before he asked for help and I or someone else on the team would solve it in a couple of hours or less. I have no idea why this happens, but I've seen it twice.
Do you think the problem was that they couldn't write that particular kind of code, or was it the more general problem of not being able to code well in general?
There are also the opposite. Folks who are not great at verbalizing programming principles, but do deliver solid code in reality.
In summary, hiring is hell...
The people I talk about aren't even deliberately fake. They really think they're awesome. They can say all the right things. They're charismatic and sound real smart. They have great anecdotes, dripping with wisdom.
It's just that their work is mediocre at best.
I guess you don't have to be good at fooling people when you believe it yourself.
How do you hire people who are more technical in a subject area than your current staff, if your test bottoms out at the depth of the current interviewer in the room?
I'm not sure I get this question - I'm not talking about a white-boarding exercise. I'm just talking to the candidate about a technical project they have worked on in the past, the pain points, how they resolved them, etc. If that results in the candidate being flustered it's not a good sign on numerous axes.
> How do you hire people who are more technical in a subject area than your current staff, if your test bottoms out at the depth of the current interviewer in the room?
That is a _very_ different hiring situation. If you have no one in the company that is equipped to properly technically interview the candidate they obviously you can not properly technically interview the candidate. You'd likely have to leverage the connections of the management team to find the right person to lead up unknown territory.
I generally think software developers (I count myself among them) know far, far less about psychological assessment than they think they do.
Are you hiring sales reps or engineers? Why are you asking for an engineer to sell you on their past project(s)?
Interviews by their nature are about judgment, and knowing you're being judged by a complete stranger who holds the power over this potential path of your future is enough to get anybody flustered, especially a brainiac that just wants to work in relative peace behind their desk.
I don't ask them to sell me, I ask them to dive into the technical issues they faced and how they solved them. It tells me a lot more about how they will handle similar issues in the future than scribbling DFS on a white-board would.
I don't understand what you're calibrating for...
With a deep enough interview process you’ll find people who know what they are doing. But it won’t tell you anything about their practical coding skills; it won’t matter if you are hiring an architect, but if it’s for a main developer you’ll want a decent level of productivity, proficiency with the tools, application of best practices and reviewing skills.
Those aspects are a lot easier to understand with actual code produced.