You have no idea how the "challenge" was completed. A candidate could have rent-a-coder-ed the project, refactored it a bit, and slapped their name on it.
Sure, if you ask no follow-up questions after the challenge is completed. However, with a few well-placed questions about the implementation and some inspection of the code, it'll be quite clear whether you're talking to the author or not.
We once gave a (non-web-dev) programming "challenge" (I thought it was a joke/insult given the candidate's claimed experience) to a senior dev candidate. Mind you, this was on-site. I even tested it by putting our newly hired grad to the test and he aced it within 20 minutes. We gave the candidate an hour to complete it (and added a 30 minute extension when he was struggling).
The challenge involved developing something against an SDK (interacting with a camera) and we gave him internet access and the SDK examples to work off of. In the end, he produced something not quite what we were asking, but he was sure to claim it as his work. Although it wasn't ideal, I was prepared to say that he "passed" the assignment, because he was close and I was sympathetic to factors of stress and nervousness.
As he was toured around the facility, I inspected the code and saw that he was completely passing off one of the SDK examples as his own work (diffs showed only two variable names had been changed). The reason his code didn't quite do what we wanted, is that he wasn't even able to tweak the SDK example to do what we asked. He was a complete fraud and never mind technical ability, we couldn't have dishonesty like that on the team.
Before that interview I thought those kind of challenges were a joke, but after this I saw their value. Of course, their value depends highly on how their are implemented and targeted, but dismiss their usefulness at your own peril.
This is a lopsided view. You don't indicate how long your "newly hired grad" was working there, whether the "SDK" was an in-house SDK or a third-party one, what the candidate's resume skills were in comparison to the kind of programming around the SDK your test was designed for, etc.
These are all relevant details, because except for the most trivial of SDKs and their appropriate domains, a thoughtful, skilled developer might spend an entire hour simply reading through the SDK and/or its documentation to get a handle on its capabilities.
So I don't get that your story is an anecdote in support of the potential usefulness of code challenges. Instead, what I gather from your story is that your company prefers candidates who practice cowboy, shoot-from-the-hip programming instead of thoughtfully approaching a new, unfamiliar SDK and pondering how it might be used to solve problems related to the domain it was built for, then applying this thoughtful process to provide a solution to a problem. In fact, if I'm given a non-trivial SDK and told "write a program that does X using this SDK; you have one hour", my response might be something very similar to the above, directed at the person giving the "assignment" out, followed by a polite but firm indication that the company doesn't meet my requirements.
>You don't indicate how long your "newly hired grad" was working there, whether the "SDK" was an in-house SDK or a third-party one, what the candidate's resume skills were in comparison to the kind of programming around the SDK your test was designed for, etc.
I wanted to avoid writing a novel, so I left out some of those details you're interested in. Trust me, when this "test" was first suggested, I was wary of it myself and did everything in my power to make it very achievable. That's why I considered it a joke, it was not nearly difficult enough to assess the level of seniority this candidate was claiming. This was someone with 10+ years of development experience, in the same field, experience with the language and environment we used. On paper and even on the phone, he made himself sound like the perfect candidate. In fact, before the on-site interview, I thought he had it in the bag and was excited to get some help on the team.
The SDKs in question were for a camera (to get images from the camera) and OpenCV, which is well known in imaging circles. The test was to receive a video stream from the camera (there was an example showing how to do this and I made it clear I didn't mind if he made that example his starting point) and to subtract consecutive images to create an image representing the "difference" between the two images. Not a meaningful result and one could barely call it image processing, but it would show us the candidates abilities to interact with multiple well-documented SDKs to accomplish a very simple task. If someone who claims to have years of experience with image processing can't accomplish subtracting two consecutive images and drawing the result to the screen, given all the tools to do this, there's a big problem.
Our new grad was mostly a tester and an occasional developer. He had less than a year of experience. He completed in 20 minutes without supervision.
>Instead, what I gather from your story is that your company prefers candidates who practice cowboy, shoot-from-the-hip programming instead of thoughtfully approaching a new, unfamiliar SDK and pondering how it might be used to solve problems related to the domain it was built for, then applying this thoughtful process to provide a solution to a problem. In fact, if I'm given a non-trivial SDK and told "write a program that does X using this SDK; you have one hour", my response might be something very similar to the above, directed at the person giving the "assignment" out, followed by a polite but firm indication that the company doesn't meet my requirements.
While I am fully aware that there are severe limitations to the usefulness of a test like this, in this situation, it saved us from a terrible hire (this test wasn't the only indication that he wasn't who he claimed to be).
In all honesty, you sound like a fairly thorny/difficult person to work with and in your hypothetical scenario where the test caused you spurn the position, it would have inadvertently done its job.
I'm often surprised by what people will take umbrage at. It's one thing to decide that it's a seller's market, and that therefore you'll interview somewhere where they'll buy you lunch and offer you a six figure salary and generous share allocation after nothing more than a cozy chat. But it's quite another to become actively offended by the effrontery of a company demanding that you demonstrate the very skills they're hoping to hire you to apply on a daily basis!
What are people worried about? I have to say I can't help but assume the worst. Because what have they been doing for their career as a professional programmer, if when asked to actually write a program, they can't actually do it? Even with an example right in front of them?
Don't forget that in this particular example we have a supposed senior programmer! Somebody applying for this sort of job should, I feel, be able to actually program. Even if they've never seen the library before. In fact good candidates should be barely fazed by an unknown language! What is the "senior" bit for, if not this? At the very least, perhaps the candidate could admit that they seem to have found themselves interviewing for the wrong sort of role, and one that makes better use of their other skills would be more suitable? That might be a better tactic than storming off in a huff.
But I've never stormed out of an interview in a huff - so what do I know.
There's a reasonable expectation to have to demonstrate one's skills in an interview. Whether any given test or set of questions actually is valid for doing that is another matter entirely.
Put another way: it's just as unreasonable for a company to expect a developer's skills to be measured by contrived tests or to demand they spend any unpaid time at all doing "work" (e.g. projects), as it is for a candidate to expect a company to effectively kiss his butt and lay out red carpet just because he decided to apply.
I agree that there certainly are problems that would make a good "code challenge" question, but I'd argue that there are better tools for weeding out poor candidates. I also think that there's a big difference between on-site challenges and "here's what we'd like to have you do, call us back when you've got something" challenges.
Sure, if you ask no follow-up questions after the challenge is completed. However, with a few well-placed questions about the implementation and some inspection of the code, it'll be quite clear whether you're talking to the author or not.
We once gave a (non-web-dev) programming "challenge" (I thought it was a joke/insult given the candidate's claimed experience) to a senior dev candidate. Mind you, this was on-site. I even tested it by putting our newly hired grad to the test and he aced it within 20 minutes. We gave the candidate an hour to complete it (and added a 30 minute extension when he was struggling).
The challenge involved developing something against an SDK (interacting with a camera) and we gave him internet access and the SDK examples to work off of. In the end, he produced something not quite what we were asking, but he was sure to claim it as his work. Although it wasn't ideal, I was prepared to say that he "passed" the assignment, because he was close and I was sympathetic to factors of stress and nervousness.
As he was toured around the facility, I inspected the code and saw that he was completely passing off one of the SDK examples as his own work (diffs showed only two variable names had been changed). The reason his code didn't quite do what we wanted, is that he wasn't even able to tweak the SDK example to do what we asked. He was a complete fraud and never mind technical ability, we couldn't have dishonesty like that on the team.
Before that interview I thought those kind of challenges were a joke, but after this I saw their value. Of course, their value depends highly on how their are implemented and targeted, but dismiss their usefulness at your own peril.