Note that down votes are a key part of what drives a number of automatic scripts that clean up things.
* People who waste too much of the communities time (asking poor questions that get closed and down voted ) get question banned - driven strongly impart by down votes.
* Closed questions that have a negative score and no answers with a positive score get deleted after a bit over a week. (If people complain about how much crud there is, one should look at how much crud gets automatically deleted).
* A down voted question helps signal to people reading to not waste their time reading it because its not useful. When you've got 8000 questions/day, someone saying "don't bother with this" can save you a bit of time reading it.
For smaller communities where a person who wastes your time can get addressed through individual moderator attention. Where there are 8k questions per day asked by a similar number of people, you can't have every moderator address each poor question and time wasting troll. It is for this that simple down votes can help drive the automated system when there is enough signal that the person is wasting everyone's time without involving moderators.
Update:
So, because of how the system was designed, it isn't something that one can easily go back to and change (and remove down votes). Its part of the software and the culture of the group and the two are inseparable - the technological and social part of the community is tightly interwoven. Further reading on this at http://www.shirky.com/writings/group_enemy.html
In such cases downvoting needs have a crowd effect.
say 3 or 5 downvotes are required to start flagging inappropriate content.
surely if more than 3 people flag something, then its likely there is a case to answer. takes away power from pissed off people who can't stomach a diff viewpoint.
Downvotes serve to help signal to the person that there is something wrong - and it does it noticeably. Flags are things that go to various levels of moderation (close flags go to the close vote queue, moderator flags go to the elected moderators, low quality flags (only doable on negative score posts) go to the low quality review queue, and not an answer flags go to elected moderators).
Downvotes mean that its not helpful, or that its not correct, or that it insufficiently answers the question. Flags and downvotes are often orthogonal to each other - one doesn't mean the other is needed.
With a good question, viewpoints shouldn't be an issue at all - there's a question, there's an answer.
An individual who systematically down vote another individual will have the votes reversed by the system.
Upvotes by themselves should be enough to bring the best posts forward. IMO this is true for HN too.