The ideal purpose of grades is teaching. It's a way of informing you of the parts of the curriculum that you do not yet fully understand. This sounds like an entirely appropriate situation to lose grade points on.
Unfortunately, grades now also serve as some poor measure of intelligence or competency in many people's minds.
Marking work is a way of teaching, grades are used as a measure of understanding. I had a friend in college that, at the start of a semester, was typically a mediocre to borderline failing student if you looked at the marks on his assignments. By the end of the semester his understanding would be on par with mine (typically As in CS/math courses with the occasional B). However, his grade reflected his poor early start with the material and he'd end up with Cs. Now, that's good enough to pass to the next class, but it still helped to screw him over when he started applying to jobs after college.
A grade is also how they decide if you get your undergraduate degree. It would be a strange shame if that decision had nothing to do with your intelligence or competency as you imply.
Unfortunately, grades now also serve as some poor measure of intelligence or competency in many people's minds.