I've got a little theory. It seems to me that the provocative thing about the essays is their aesthetic. They're governed by a particular style. One principle in it is minimalism: compress the writing until everything extraneous is gone. Another is vividness: whatever is being said, seek the phrase or image that throws the point into the sharpest possible relief.
The dominant quality of the essays is that they pursue this aesthetic with utter ruthlessness. Anything that would use a few extra words to reassure the reader is thrown out. Anything that would tone down an idea a little bit to make it more palatable is thrown out. There isn't any room for these things because the author is optimizing for something else - say, meaning per word count. In fact, an entire dimension of language, the phatic dimension, is thrown out.
So, Paul Graham's writing is radically aphatic. That's disorienting. People are used to writing that includes, among its various threads, one whose purpose is to reassure you that the author is a nice guy, that he might be wrong, you can still get along even if you disagree, and so on. This is not only absent from the essays, it's been deliberately excised. On top of that, what is there has been distilled for maximum impact and often touches subjects that people have strong emotions about, such as programming languages and what the hell we're doing with our lives :). Not surprisingly, some readers feel punched in the gut. For them, an obvious explanation is ready at hand: Paul Graham's writing is like this because he is like this. He must be someone who doesn't care how others feel and wants only to magnify his own grandiose ideas. In short, an arrogant asshole.
I think this explains why people project so much emotion into what they read in those essays. "Oh... you haven't founded a company? You suck." But the essays never say anything like that. People don't read them this way because they say such things. They read them this way because they lack the kinds of things writers are expected to put in to stave off provocation. They lack these things not because the author is an asshole but because he cares about a certain style of writing. Enough, in fact, to pursue it ruthlessly... in his writing. To naively map that back to the personality of the writer is an obvious error, a kind of reverse ad hominem. But it's an understandable error. There aren't many people who care that much about an aesthetic. (I mean "aesthetic" in a broad sense, by the way. As much a way of thinking as a cosmetic thing.)
No doubt there is a connection between an author's personality and his style, but it's hardly an isomorphism. I don't know Paul Graham, but I know he doesn't talk the way he writes. For one thing, one can point to examples (like the interview in Founders At Work). For another, nobody talks like that.
I'm always surprised by how offended people get by things I write. It seems totally unpredictable. I didn't expect people to be so offended by this one. In fact, I thought I was saying something rather smarmily ingratiating, if anything: that the famous startup founders you keep reading about in the press are not that different from you, but that they just have, in effect, a healthier work environment. See the last paragraph.
And yet somehow that message has gotten completely twisted around. It's as if people wanted to misunderstand this essay.
I've been mulling over why this happens, and one reason is certainly the one you suggest. I try to cut every unnecessary word, and I don't say things unless I'm pretty sure of them. The result sounds arrogant, because it doesn't have any of the hedging people usually surround ideas with to make them palatable.
But there's no alternative. People won't read essays if they're too long. If you want to get a lot of ideas into an essay short enough to read, you have to be so curt you sound arrogant.
On another note, I also get the impression that you write to a certain type of person. Possibly a type that is similar to you. People like Jeff who take issue with your essays are simply not the target audience (whatever that means). As someone who is currently caged in a 9-5 job with a big company, I have to say that your essay resonated deeply with me...and I loved it. I appreciate the way your essays frequently force me to take a good hard look at what I'm doing, where I want to go, and what I need to do to get there.
Keep in mind that different situations demand different writing styles. For instance, if you are trying to resolve a bitter conflict, "writing things that cannot be disputed" is just the way to go. In that situation, your #1 objective would be to find things that both parties agree on, and build up from that. (This also might be a good way to win an argument.)
There's a really dense book called "Difficult Conversations" (Stone, Patton, Heen, Fisher) that covers, among other things, how people frame discussions in terms of their identity. I think if I had adsorbed that book better I would have predicted that this essay would deeply offend some people.
One other reason is that you have a touchy subject. Tell anyone they're doing the wrong thing (implicitly or unintentionally) and you'll get the same response. People are very sensitive about their own choices (stubborn), especially ones that are hard to defend rationally (religion, major, job, etc).
I picked up this habit in the Harry Potter fandom, which aside from being a community of writers also happens to be 99% female. Women's writing tends to include many more phatic expressions, because (particularly in something as socially-constructed as fandom) its purpose is often relationship-building. I adapted to fit in; before then, my writing style often tended to be brusque, mechanical, and to-the-point.
I still use my old style in technical posts, where I figure the reader can deal with any unintentional rudeness. For example, compare the last paragraph of this: http://news.ycombinator.com/item?id=144001 to this comment: http://arclanguage.org/item?id=3103
I agree, but I don't think it's because they're hard to defend rationally. Instead, I think it's because they're an essential part of their identity. If I criticize you're choice of colors, that's not something that you can defend rationally, but you won't be too concerned if you're not much of a design person. On the other hand, if you're an academic who built a career on some theory, criticism to your theory will be very personal. The issue here is one of cognitive dissonance. It's very difficult to differentiate between critcism for someone's choice and general mean-spirited feeling for that person.
One of my peeves is the use of "in my opinion ..." since pretty much anything anyone says is, by definition, their opinion.
Yet your mention of the phatic (thanks, too, from teaching me a new term) is interesting. Those little qualifiers, even when not strictly needed, do seem to keep the wolves at bay.
But that doesn't at all mean that it should play the same role in writing.
You say phatic language is about maintaining the connection. If I understand this correctly then there's a role for it also with people one does know well. If you prefer referential conversation and meet someone who's completely restricted to that part of language, you may initially enjoy the conversations a lot. But, after some time (this could be months or years), despite their referential content they may appear increasingly "empty": you realise you're still completely replaceable as a conversation partner by a random new person.
The bit about using phatic language mostly with people one doesn't know well is not from Jakobson. I made it up, and it's probably wrong. I didn't go over that sentence 20-50 times :).
But as there are four other components according to Jakobson, I should be careful. Added Jakobson to my reading list.
What does that mean? No online dictionary I regularly use lists "aphatic". Google turns up a geology term here with a different spelling.
 - http://www.greengonzo.com/dictionary/Aphantec.html
When you read, say, Mencken, you know it's an op-ed and to take everything with a grain of salt. He would even contradict himself from one piece to the next. You just chew on the points and smirk to yourself when you see the flaws. No, Mencken definitely didn't write like he was trying to be your likable friend.
Graham writes like he's building a legal case or trying to get published in a scientific journal, even though the pieces are really casual observational op-eds. The style invites you to poke holes and take each point too literally.
People are used to writing styles adapted to the format for good reason. The reactions like this to Graham's pieces make a good case for learning and using rhetoric.
If what you mean by casual is that I haven't thought
much about what I'm saying, I disagree. In a typical
essay I probably go over every sentence 20-50 times.
When I'm unsure of something I'm saying, I try to
be precise about the uncertainty. So while I
may often be wrong, it's not out of carelessness.
I believed you because I know what you're talking about. However, for a skeptic, it leaves the door wide open for them to say that you're just seeing what you want to see. Also, you come across as someone who thinks his five-minute subjective impression gives him the right to dismiss other people's life choices. They feel justified in writing you off as arrogant.
So, a couple of suggestions:
1) If you're going to present an argument from subjective observation, the example has to be very vivid, almost like a novel. The reader has to be compelled to your conclusion by feeling like they are observing things right along with you. Specific details are important. These corporate programmers -- did they wait glumly and patiently in line, or were they eager with anticipation to get back to what they were doing? Did they mumble to the barista while looking away, or speak forthrightly, making eye contact?
2) Rhetoric becomes great when you point out the invisible monsters in the room, and tame them. When you talk about startups to people who are not in startups but could be, these monsters are envy and self-doubt. You have to deal with the fact that your audience sees other people growing rich or at least having fun, and is plagued with self-doubt about whether they are chickenshits. Given a chance they'll project their self-doubt onto you, the person who raises the question. There are various strategies to prevent this from happening, but i-could-be-wrong is not one of them. The main one is to get people to identify with you before you move into more challenging territory. It does require more space to accomplish this, but it's far from filler, it's essential.
No. It's hard to dig out somebody's argument when it's surrounded by self-deprecating rhetorical armor. Presumably pg wants the argument front-and-center. That way if he convinces you, it's definitely the argument and not the rhetoric that's doing it. If you disagree then you know exactly what you're disagreeing with. Ignore the ad-hominem people; they'd just find something else that's wrong with you.