However, I doubt they have the bandwidth to manually grade thousands of assignments. Given the scale of the class, the grading is forces to be automated. It's hard to come up with such assignments and even harder to make them challenging, given an automated grading script.
1) They provide some set of data and establish rules for the competition.
2) They implement their own solution to the challenge, and that is the benchmark.
3) A "passing grade" is obtained by getting any working system.
4) The actual grading is then given on a curve, compared against their benchmark.
That said, learning what the algos are and how they work is one thing; learning how to actually apply them to real life situations is another thing. I think the class leans quite heavily towards the former, but I really love the few glimpses of the latter.
Personally, as someone who is new to the field (didn't do maths at college) & is barely fitting the classes & exercises around a fulltime workload & other things, I am glad that the programming exercises are "easy". Some of them are ridiculously easy, agreed (where 1/2 the solution is given basically verbatim in the pdf notes, and the other 1/2 in the code comments) - but for most of them I think it's enough to wrap my head around what's actually happening, especially in terms of the multiclass neural network assignment. That gives me enough foundation to try to apply them to real-world situations on my own time.
And he knew all the tricks. I don't think anyone ever beat him. And he didn't show anyone any of the solutions until after the final.
I felt like I learned more from the few minutes I spent reading those solutions than I did during the rest of the course.