When comment scores were visible, it was obvious that many people would just vote with the herd, downvoting comments with lots of downvotes, and upvoting comments with lots of upvotes.
This is still a problem, since HN still tends to put highly rated comments near the top, and low rated comments near the bottom. But it's not nearly as much of a problem as it was when comment scores were visible.
I think the quality of comments has increased with the new system, and I find myself reading more of the comments now that the scores aren't visible.
I also find myself voting less, and voting only on comments I personally feel are exceptionally good or exceptionally bad.
One change I would like to see HN experiment with is making the score of a given comment visible only after you've voted on that comment.
This will both encourage voting and also satisfy people's curiosity, while still discouraging voting with the herd.
Hmm. First of all you had no way of knowing that people were voting because of the high (or low) scores. Your theory seems to be "Hmm, the score on this comment is un-naturally high, therefore people must have just been voting on it because it was high". Secondly, now that we can't see the scores, you've no way of knowing whether the points awarded are just as 'un-naturally' high as they were before scores were shown. For example, one of my comments in this thread has got 11 points in the last ten minutes or so - that's without scores being shown.
In the past, you may have attributed that score to herd-voting. But clearly that can't be the case, as no-one can see the score. ( except the few who've installed the points plugin!)
However, it's not true that there weren't signs that herd voting was going on.
I've spent quite a bit of time on HN, and have monitored many popular threads and have seen how comments fared when they've gotten lots of up or down votes.
It was my feeling (though just a feeling, without hard data to support it) that comments would tend to get upvoted substatially more when they already had a relatively high rating, and downvoted when they had a relatively low rating.
Also, I noticed many comments that I considered to be of high quality get passed over for low quality comments with higher scores, and what I considered to be kneejerk downvoting on valuable comments with lower scores.
Now that comment scores are no longer visible, it's harder to gauge these trends, but not impossible.
HN still tends to place higher rated comments near the top, and lower rated comments near the bottom, so you can get a feel for how people are voting on a given comment by making note of how the comment moves up and down the page.
The scores of your own comments are also visible. So you can draw some conclusions from monitoring voting on them.
I would certainly love to see public release of anonymized HN voting data, and some good analysis of these sorts of trends (also on voting patterns in relation to how long a given user has been on HN, and how active that user is, etc..).
Until then, all we have are our own subjective interpretation of what we've seen happen on the site.
That would be an acceptable compromise to me. It fulfills my own use-cases for having scores, but retains the benefits of not having scores.