1. It is a problem representative of the work the person will normally do in the job (we use an extract of our actual data for it)
2. We provide clear guidelines and criteria for success.
3. They are allowed to implement a solution in any language they choose, using any tools they choose (we ask everyone to bring a laptop with their preferred dev environment, but we do provide one with some common tools for them if they can't).
4. They are allowed to use Google and open-source libraries to help them with the solution (we want to see how they really solve a problem)
5. We are there to interact with them and answer domain specific questions, as well as to offer advice if they are stumbling on something to help them move forward.
I've really, really gotten a lot of good mileage out of this. The range of skills that I've encountered over the years has really surprised me. The problem is pretty straight-forward, and usually involves parsing some JSON, looking at the data, and matching disparate sets of the data together. The basic O(n^2) solution can be done in about 20 lines of code and there is some nuance to the datasets that require them to think through the problem and not what they have memorized in their head. I've had programmers with years of experience from places like LinkedIn take 20 minutes just trying to read a file in from the file system, and I've had people who are self-taught developers crank through a really good solution in the same time. I like being able to see how people are writing code, how quickly they are able to lookup information they might need, and listen to the types of questions they have when trying to solve the problem.