I think it's actually better now than it was. There are a number of contrarian voices in each contentious thread arguing firmly that there's more to an issue than perhaps simple analysis would give credit for. There used to be more of an Ayn Randian/"supermen" feel, more people patting each other on the backs for being superior creatures cos entrepreneurs are the end point of evolution. The site seems a fair deal saner, from my point of view.
The nature of systematic conversation ensures there will always be groupthink. Systems are designed to function in certain ways and deny other approaches to get their desired result. The result is that people who want this system flock to it, and people who want to somehow act counter to this system are going to be a minority that's possibly chased away by the system itself. Options like downvoting and "ranking" comments" and flagging inappropriate topics basically say that there is a right way to behave or talk, there are topics that are better than others. Then either people are attracted to the sites that reinforce their worldview or they're conditioned by these rules to decide there're right and wrong ways to think or to be – I'm not sure which it is. Probably a little of both.
That's why I don't think down votes should be used to express disagreement (although pg doesn't share this opinion). I personally frequently up vote comments I disagree with if I feel they are thoughtful and bring an interesting perspective. The variety of ideas is important and should be encouraged.
Up/down is the wrong direction for the arrows I think. Left <--> right, meaning on-topic/off-topic ? One of the problems I think is that there are so many dimensions represented by two simple little arrows that it is difficult for the system (the web software itself) to discern the intent, so it turns into a popularity (I agree,I disagree) button.
But if you had a button for each piece of information it would look like
[] well written
[] well thought out
[] agree
[] disagree
[] poorly written
[] funny
[] off topic
For bonus points they would be in a random order in the UI so people wouldn't develop muscle memory voting on comments.
I have thought about the On/Off-Topic button as well for some time now. I think it's a good concept to try out, mainly because it is not judgmental - Sometimes I like following an Off-Topic discussion and I've had a couple of good ones on HN. But most of the time, it can be very distracting - separating it from the main discussion is worth a shot.
As for more detailed buttons - I think that would make it too complex. Slashdot has established the same concept, probably based on the same reasoning (mostly to distinguish between "just" funny and actually insightful comments). Can't speak that much about how successful it is, maybe somebody else has more insight here.
Yeah, the buttons get out of hand, but the information they would represent is greatly needed. And how do you encourage people to rate what they read and interacted with? Based on the stats people post about the amount of traffic driven by an HN or Reddit submission ... the vast majority of users don't interact via any sort of voting mechanism at all.
I am working on a site now that is heavily based on passive activity voting, users have no idea how their actions are being used by the site to rank itself. Their inputs have much less weight than purposeful actions where both parties are aware of the event.
It is almost like you need an agent that votes in certain public ways (sentiment analysis,summarization,etc) to decide if comments snarky, funny, off topic, etc.
I guess that would fall down once you get to mixed comments - something can be snarky, funny, off- and on-topic at the same time. Human conversational interaction is just bloody hard to quantify. The first few steps are always simple and then it gets hard fast.
Passive voting sounds interesting, though - would that work via Javascript? How would you determine what the user is focused on?
Finally - I think it's important not to get ahead of ourselves - because I think that's what happened at Slashdot. Talking about new classifications is all nice and well, but it's useless when you don't have a clear plan what it should result in and how it would engage people. You need a clear use case.
For instance - my proposal about an off-topic discussion button is pretty straight forward. People would see the immediate benefit: Hey, this can help us make conversation less wasteful and save me scanning time as it separates the cream from the crop. I think it wouldn't have a problem enticing engagement, too - nothing drives engagement in nerds like being annoyed, so they'd click that button.
On Slashdot, I can theoretically reduce a conversation to exclude all the funny-only comments so I'm left with just the cream. But seriously - how often does anybody use that? Once you give people elaborate personal filters, you actually end up having them worry whether they're missing something. Which is why they can only be consistent and transparent. If I reduce a comment thread on Slashdot to exclude the funny business and see that one of the nodes has a HUGE conversation, I wonder whether I should check it out and maybe end up annoyed because it's just a chain of memes after all. If there was a way in HN to mark things as off-topic and have that a clear process that is always applied, I think I would be more confident in skipping. Mainly because if it was broken, the community would complain until it is fixed.
It's a social problem unlikely to be solved by a technical solution. When it's an open flat democracy and votes from hordes of new users have the same weight as elder ones it is pointless. They'd just start flagging for spam or offtopic to hide dissenting comments (like it's done in youtube).
On the converse, if the older people can amass power and karma new people can't move it.
I don't have a solution. I am not sure there is one or that it could be "fixed"
Maybe semi-intelligent Eliza like agents that constantly clean the place up.
Stackoverflow has done a pretty decent job I think. Wikipedia could use some work.
----
I think the base set of rules need to be codified by the system. The users can guide each other and the system, but the system itself should be stable.
The nature of systematic conversation ensures there will always be groupthink. Systems are designed to function in certain ways and deny other approaches to get their desired result. The result is that people who want this system flock to it, and people who want to somehow act counter to this system are going to be a minority that's possibly chased away by the system itself. Options like downvoting and "ranking" comments" and flagging inappropriate topics basically say that there is a right way to behave or talk, there are topics that are better than others. Then either people are attracted to the sites that reinforce their worldview or they're conditioned by these rules to decide there're right and wrong ways to think or to be – I'm not sure which it is. Probably a little of both.