If your point is that these sorts of tests don't have any statistical power, I would whole-heartedly agree. Certainly you have to be careful with the types of conclusions one draws from the results of such tests. But they are not useless. The probability that a candidate will work out after having solved the problem with the best and most efficient solution is certainly much higher than if she only gave a middle-of-the-road solution. They are useful as positive filters.
And just to be a unnecessarily technical: if you accept the premise that the combination of appropriate CS knowledge and intelligence increases the likelihood that a candidate will successfully answer the question, then the fact that they answered the question successfully increases proportionally the probability that the candidate has sufficient CS knowledge and IQ (good ol' Bayes' theorem). This then increases the probability that the candidate will be successful on the job (citation in ancestor comment). That was a long winded way of justifying these types of questions as a positive filter.
I didn't say they're not a positive filter; just that they're a black and white positive filter.
Usually, hiring isn't about "passing or failing" candidates; it's about ranking candidates relative to one-another, and hiring the "best" ones (under whatever utility-function you think will help your business succeed.)
And putting people into "really impressed me in solving a puzzle" and "did not really impress me in solving a puzzle" does not much help in ranking people. It just gives you two buckets, one full of people who were both intelligent and lucky enough to get that particular trick that day; and one full of people who might or might not be intelligent--more intelligent than the people in the other bucket even--but who didn't get the trick that day. Personally, I'd rather not allow "luck" into my hiring process if at all possible.[1]
And this isn't so hard! I don't see why there is such a strong digging-in against the notion of changing away from "one or two long puzzles, that each give a single bit of informational entropy about the candidate's capabilities" to "hundreds of trivial, short puzzles, that together give a strong, statistically-sound measure of the candidate's capabilities relative to the other candidates measured."
I imagine the answer is that "the long-form questions give me a chance to get to know the candidate better, and see their [cognitive, and social] approach to problem-solving." But the OP (this guy: http://news.ycombinator.com/item?id=5264735) never mentioned knowing someone's approach to problem-solving as a dependent variable for their workplace success. IQ tests--relatively ranking someone's general problem-solving ability--is a dependent variable for workplace success. Work samples--showing, technically, that you can actually do the job as asked--is a dependent variable for workplace success. That's it. Rely on those, not your gut.
---
But, if I may go on a bit of a rant here...
This implies that convincing your interviewer that you'll be a nice guy who stays on-task, doesn't complain about problems, works long hours because they're "dedicated to the company", will talk down their successes (that is, be "humble") and won't demand fair compensation for their abilities, and is otherwise the "Agreeable, Conscientious"[2] kind of person that schools are expected to produce[3] and HR departments want to consume... is not a dependent variable for workplace success. There's no correlation. People can be great workers and produce their best work with their boss hating them and their coworkers thinking they're an arrogant jerk. And people will not just put up with them--but even approve of their behavior--if they get the job done better than someone who isn't like this. (Anyone who's ever watched House will probably grasp the concept intuitively.)
In fact, in software development at least, I suspect that this "not submissive to workplace domination" attitude can even be positively correlated with success--it's what gives people a sense of ownership and responsibility over project components, makes them passionate about making things right instead of just working, getting coworkers to refactor their own APIs so it doesn't hurt to consume them, etc. And I think that at least some software shops know this--it's, as far as I can tell, the original meaning (before it got diluted to meaninglessness) of looking for a "rockstar" developer. A clearer term might be developer primadonna.
But these types of people--unless they're lying--interview very badly, since, implicitly, an interview is an hour-long test of meaningless workplace submission. This probably isn't a bad thing--if you have the type of organization heavily infected with stress addiction[4], then this type of person--who will avoid the gametalk[5] that powers reciprocal dopamine transactions--won't fit in well anyway.
But if you're looking to build an organization full of people who care more about winning[6] than about "being a company", my advice is: skip the interviews. Just hand them the IQ+work sample test, send them into a room, and get them to do it. Maybe we could even turn this process into some sort of widely-recognized certification [a developer's journeyman certificate, basically] so the interviewed would only have to put up with it once. That would reduce the hiring process, simply, to "let's go out for a pint." And that sounds, in a healthy industry, like how things should be.
---
[1] For the same reason, I don't want to know what school you went to; luck [of parentage] significantly determines what schools you'll naturally end up at, with or without trying, but when people know what school you went to, a peculiar kind of nepotism/cronyism appears, with Harvard interviewers hiring Harvard graduates even if the Ohio State University candidate is otherwise equivalent, etc.
I appreciate your comment. I didn't realize you were actually advocating giving a bunch of small programming questions in place of one or two involved programming questions. I totally agree that this is a far superior way to test programming ability. Actually developing such questions would be rather hard I think, it seems it would be very easy to degenerate to trivia. Perhaps a set of questions similiar to that Quixey one minute programming challenge that was posted here a while ago (http://challenge.quixey.com/)?
I actually have a very ambitious idea to completely disrupt tech hiring. You have echoed much of my rationale. I don't want to go into detail, but the premise is basically to remove ego from the hiring process and make it far more statistically sound. Of course, this is also the very reason would very likely fail. Everyone loves to employ their pet "interviewing hack" to find world-class developers (who happen to be a mirror image of themselves). Most people will not give that up lightly.
And just to be a unnecessarily technical: if you accept the premise that the combination of appropriate CS knowledge and intelligence increases the likelihood that a candidate will successfully answer the question, then the fact that they answered the question successfully increases proportionally the probability that the candidate has sufficient CS knowledge and IQ (good ol' Bayes' theorem). This then increases the probability that the candidate will be successful on the job (citation in ancestor comment). That was a long winded way of justifying these types of questions as a positive filter.