I believe allowing them would open a floodgate and start a race for reputation.
And I kinda disagree with your assertion that this site lacks intellectuals. There seem to be intellectuals from very diverse backgrounds (doctors, lawyers, even farmers). Sure, the discussion may not be as intellectual as, say, r/AskHistorians, but its still more intellectual than most other forums.
Sure there's groupthink in HN as in any other forum, but HN is more open to different ideas than any other forum that I have known since the days of Usenet in late 1980s.
I just wish someone could come up with a new Usenet (<rant>instead of a dozen of open-source copies of damn Twitter</rant>). Something that doesn't depend on a company or a couple of individuals. In similar vein to what happens for the Twitter copies I ranted about, but for long, threaded discussions in Reddit/HN/Usenet style. Oh, and Reddit lacks proper hierarchies, so the discovery of subreddits is pretty bad. So, yes, more like Usenet.
But then see these, which seem to acknowledge downvote to disagree is a problem:
There is no resistance to humour. Funny comments get upvotes.
There is a resistance to unfunny comments, and most attempts at humour are just bad.
If you stomp out humour you can pretend it's exactly that.