It's also useful to have a notice such as, "This user's commenting privileges were suspended for 100 hours for this comment." By making the punishment public, the commenter is (hopefully) shamed into behaving better in the future. And like you said, it may even prevent others from behaving similarly.
With displayed score you end up with digg/reddit where people just post short memes and try to get a high score displayed in the easiest way imaginable.
Nope. That doesn't show you the difference between a barely-passable comment and an excellent one.
I learned a lot during my early HN days, when comment scores were visible, simply by looking at what sort of comments tended to score 5 points vs 1 point, or 15 points vs 5 points, or 40 points vs 15 points. Here on HN, "just post short memes" tends to be met with negative signals; thoughtful, helpful, and accurate responses are met with positive signals. I found it helped to be able to see that, and I think many newer HN commenters have the same misconception as you do because they haven't had that same training.