If something takes off I'll buy a dedicated server for it, but if you're starting a company, get a Dreamhost account for $10/month and they throw the domain in for free. You can buy yourself two years of hosting runway for $200. If you max your dreamhost account you're well on your way.
I don't even bother paying for github, just throw git on my dreamhost account. (They have dedicated SVN services as well)
Also, I just got about a dozen notifications today that dreamhost had upgraded all my wordpress sites to 3.0.4. I don't do anything they take care of it. Someone breached an account? I get an email.
The only time I've ever run into trouble with dreamhost is an old old e-commerce site that needed MySQL4. I could get the ancient version of PHP it used working but couldn't get MySQL4.
It might not be the fastest, but you can't beat the value, especially if someone you know can give you a discount code (30OFFER is mine. I think I get some money off my bill if anyone uses it.. )