Take a standalone product like Camtasia. It works really well and its worth the money to buy it. But if it was FOSS and I could just donwload the sources, build and it use it, and only had to pay for support -- well I probably would never buy it. I've never needed support with it. It's so easy to use, I've never had a need for support (in fact, that's partially why I'm willing to pay for it!).
Figure out what your business model is first. And then do what makes sense. Open sourcing often does, but not always.
Not only this, but anyone could build and maintain a free version and make it so the average user doesn't need to even compile it.
Open sourcing an app doesn't really make sense for a business. You will be making it easier for competitors to put you out of business.
EDIT: And to be clear, that's largely what Google's Chrome/Docs strategy is. The total revenue from it isn't very large, even if it is a success. The real win is chopping down the core revenue of someone who wants to chop down their core revenue (search revenue). But today Google has to build this all. If MS did FOSS, MS would be handing Google the keys.
To play devils advocate to your point though what about mysql, that's a downloadable product that makes it's money from commercial support or to a lesser extend tools like Phusion passenger. Most people don't pay for them for the thousands of Rails websites passenger hosts. However, for those larger enterprises it's much better to employ the experts for support.
The key is looking to see if you business fits exactly as you say.
With that said, when I look at the hot enterprise software I see a lot of closed source software. Is this wishful thinking or something that is actually beginning to happen?
No, SAS, SAP, Oracle Apps, Lotus Notes, etc., won't be going away tomorrow. But I think there are clear signs that acceptance of F/OSS in the enterprise is growing.
For my own part, I'm working on an Open Source startup, playing in a space similar to Jive or Yammer... so - for what it's worth - I've bought into the idea. (Then again, I'm a F/OSS ideologue, so I'm not exactly unbiased).
That does seem like an interesting middle ground. It seems like your goal would be to develop a community of developers around it, but with the exception that you are the only person distributing the software (for a fee) to non-developers.
The main issue I see is that in order to work it really requires copyright assignment of all code to the company, and in many cases that's a raw deal for your open source contributors.
Licensing something under the GPL makes financial gain from your copyright practically impossible. People make much more money if they retain their legal right via copyright to be the sole source of (commercial) distribution. Hence, very few corporations are interested in using the GPL for software that they intend to package and sell as its own product.
The ideal license in my mind would include a clause that source and/or binaries should only be given to licensed users. I understand this greatly restricts freedom of distribution compared to the GPL, especially since a user can't know if a user is licensed or not (a practical application of this may be a copyright notice bundled with every patch, telling unlicensed persons they are not allowed to use it), but we know that GPL is too lax on this front from a commercial perspective, making it inviable to sell GPL software (the software itself, not attached services) as a major product. In some cases, a merely non-commercial license might be acceptable. I am always in favor of the most freedom, but we know that the amount of freedom given by the GPL is too much to retain profitability.
I would still encourage GPL use where possible, but a license like the one I've described would make it possible for consumer-friendly developers (Valve's a good candidate imo) to open up their code, allow the community to hack on it and share their hacks amongst themselves, and still protect their revenue stream. This could also be extended to game engines like Source; a non-commercial license would be ideal for that so hobbyists, academics, etc., could still learn from it and use it, and Valve could still make their money licensing it out to other game developers.
The point is to get source into the hands of as many persons as possible. I don't think it's good to restrict this because of divergent economic or political realities; if you're using a computer, you should have source to your programs, and be able to modify them as you see fit, and developers should still be able to make money off of their software.
I wonder if the FSF would agree to write such a license. It doesn't quite fit with their goals, I think, but they also preach "pragmatic idealism". And they provide the LGPL, so it's not without precedent.
You might consider something like the AGPL instead, which still qualifies as Open Source, but which provides added protections for a hosted service.
(I'd also think AGPL would be less OSS-like than that suggestion, since it limits what changes you're allowed to make instead of just limiting how you're allowed to distribute it.)
As another commenter pointed out, AGPL is actually the opposite direction, as it merely applies the copyleft constraints of the GPL to hosted software.
I have non-mainstream economic and political views too that involve the principles you're talking about, but is Free Software a movement about software or reshaping politics? It's nice to conflate the two and spread philosophy through the GPL to the extent that you can, but it's also nice to know what you're computer is doing and to be free to change it. One goal at a time.