I don't see the point in denigrating people's passions, Bill. Paul Graham likes programming and making money. Not everybody does. Other people like to write music. Others, buying attractive clothing and learning how to look good. These things aren't inherently better than one another unless you want to buy into a specific social idea, and then we aren't required to go along with you.
The one thing I do detest are people with persecution complexes. Nerds are frequently unpopular because they insist so passionately how unpopular they are. That's unpleasant and leads to people not liking them. As I've discovered, if you talk to people about designing web sites and programming innovations, they'll still like you. You've got to understand going in what other people will be interested in and what nobody gives a fuck about, and that some people will never care, but if you know that going in, you can make delightful friends even with non-nerds.
The other problem is that some nerds know nothing but one specific thing, and they look down on everything else, so nobody likes them. There are people who know nothing but fashion, too, though they're rare and they're just as unpopular as nerds.
I was tempted to argue that providing "value to society" is always good but what you really mean to say is not that passion leads to "value" (there's no guarantee of that) but that passion leads to the motivation to act and the desire to have an effect somehow. Whether that will be good or bad is unknown, often irrespective of intentions, and can only be evaluated through value-judgement.
If that's your belief, currently, I can't fault it.