One problem would be that if the challenges were easy, you'd get tons of silly entries, like the bottles of beer challenge. If they were hard, you might not get enough entries to be representative. Solutions:
1) Find a sweet spot of difficulty for each problem. This would be hard.
2) Offer prizes, bragging rights, publicity, etc for excellent solutions. This wouldn't be hard, just take resources.
You'd also want to differentiate types of challenges and solutions by whether they were using base language, base lang + libraries, or anything available to language. I think its different saying I can solve a challenge with a little hacking in a language, and saying I can solve a challenge importing a nifty library then use a few DSL commands. On the other hand, if we're just trying to gauge productivity using a language, the ease of incorporating open source solutions (even written in other languages) should be important.
1) Find a sweet spot of difficulty for each problem. This would be hard.
2) Offer prizes, bragging rights, publicity, etc for excellent solutions. This wouldn't be hard, just take resources.