Knowing what I know about the bill rates for the kind of work we do versus the kind of work you likely do, I'm going to humbly suggest that we're in more demand than you are (not a value judgement, just an observation about scarcity).
We absolutely do respond to RFPs, when they're for projects we want to work on or clients we really want a relationship with. And we always spend serious otherwise-billable time on proposals, even though that's the same speculative work that this article argues against.
It's important to us to be easy to work with, because we want good projects. We don't want to confine ourselves to whatever lands in the margins of "unusually liberal clients"; those clients don't necessarily have the best projects.
I read the other post referenced on this thread, and so I'll add that being able to smoke out the bogus RFPs you shouldn't waste time responding to (because they indicate inept purchasers, or, for instance, because they probably mean the client is in the tank for another vendor) is just one of those things you need to be able to do to succeed as a consultant.
I don't know, and that's a good question. Most security research proposals I've seen are crap, too. But then, in competitive proposal situations, we tend to win. So I'm an advocate of competitive proposal situations, be they formal RFPs or otherwise. =)
Any pointers on how to identify clients already in tank with another vendor? Being overly specific in the RFP and demanding the exact features a vendor has is a clear giveaway but I am wondering if there are any other signs.
We get on the phone before we respond to expensive RFPs, and there are questions you can ask (incumbent vendors, past experience with vendors, is there a formal requirement for the RFP, or do they genuinely want real proposals, etc).
On "real" RFPs, especially if you have a good shot, you'll find the purchaser you talk to is accomodating; they'll set aside time for calls to answer questions, and they'll do a good job with your Q&A submission (as a general rule, real RFPs have Q&A phases, though note that you never know whether your Q's are going to other vendors too).
There is an old negotiating trick a bizdev guy taught me once, which is, if you want to figure out whether a deal is real or whether the other side is just wanking, figure out a reason to schedule a meeting on a Saturday. If the deal is big and the other side is serious, they'll do it. Similar idea here, although less aggressive.
If your firm's way of handling the RFP process is to simply pick up the forms off some website, mail them in, and hope, then yes, I can see how the RFP process would upset you.
If anything I think you're probably underestimating the extent to which GSA-style RFPs are rigged. I tag-teamed a bunch of times with our FedGov sales guy at the last company I was at, and we closed deals at the Pentagon and elsewhere, and (a) the stories you hear are ridiculous (like the companies that pay, say, an Inuit to be on the board so they can get some obscure diversity credit), and (b) even our company, which had multiple tens of millions of dollars in both funding and topline revenue at the time, had to work through a subcontracting arrangement.