Unless you have money to burn, I would definitely recommend against hosting at a colo.
At the last startup I worked at we had no end of fiascoes regarding our colos. The startup bought over a million dollars of servers and then was paying high-priced employees and consultants to set them up at a colo. But then once the servers were all set up, the colo discovered that they didn't have enough power for our servers. Then we had trouble getting out of the big colo contract that we had signed and had to move all the servers and get them set up at yet another colo. Turns out the next colo also didn't have enough power for our setup even after many reassurances at the beginning that they did. So all in all, this waste of resources put us months behind schedule. Then, ironically we got acquired and no longer even needed our setup because we could then just use the bigger company's server setup.
This is another case of 'premature optimization is the root of all evil'. Seriously, we could have done the whole startup with a couple of Dreamhost accounts and launched earlier with a better product and with WAY (millions) more money in the bank. They were optimizing our server setup for HUGE traffic when they should have just been focused on the product.
I'm glad I was able to see those mistakes close up. It gave me confidence to do my own startup self-funded since I knew how low I could keep my burn-rate.
So now, I'm using EC2 for my servers and it's an amazing experience. I set up a custom server image and then can run multiple instances of it for only 10 cents per server per hour. I can kill instances that I'm not using and bring up a new server within 2 minutes. My server bill is less than $50 a month during development and will be about $100 a month once I launch. I'll also have the ability to scale up VERY quickly and cheaply if I need to.
Anyway, that's my take on it from my limited experience. I'm sure there are some guys that are obsessed with doing their own custom server setup at a colo. Good for them. But you have to realize that you'll have much higher capital costs up front and will be exposing yourself to much greater risk. Focus on your product and find a server solution that is cheap, low-risk, and flexible. Remember you can always scale later - but you have to have a reason to scale first! Have the faith to focus on product and don't fall into the trap of an 'intriguing' 'interesting' academic experiment in server setups!
If you want a dedicated server, aplus.net has "value servers" that are basically unmanaged dedicated servers using last-gen hardware. I pay $50/mo for a 2.5 GHz P4 with 512 MB RAM.
The problem with VPSes at least is if anyone on the physical box manages to crash the machine, it goes out for everyone. Xen has historically been buggy -- just starting SBCL would crash it(!). Count on Murphy's law -- I had just finished some work and was waiting for a client to take a look at it, and of course my webhost decided to reboot the VPS right before the client checked. Since I hadn't put a restart in rc.local yet, it was just down.
Even with all your other ducks in a row, you don't want a third party rebooting your machine while someone is in the middle of using your site, especially if there's anything important going on, like customers purchasing with a CC or investors taking a look.
Jey's suggestion of cheap dedicated servers for $50 sounds better than EC2 if you're CPU-bound (2.5 GHz vs. 1.7) and is cheaper ($72 per node for 24-7 EC2 usage) and won't wipe everything on restart like EC2 does.
Note that mileage varies widely with webhosts. The majority of them do not understand even the basics of computer security, e.g. the importance of publishing their SSH key fingerprints (via https).
I'm running at home off a DSL circuit. I host using a mixture of virtual FreeBSD servers under Parallels on an Intel Mac Mini and two real servers. It works pretty well for low traffic web sites, but the VM's need to have a real hardware NTP server to keep their clocks syncronized. I also run NFS on real hardware. It works surprisingly well. Watch how you allocate VM disk I/O across your physical disk spindles. I have three physical drives plugged into the mini.
Classic "that depends" question there. I'd suggesting going to http://www.webhostingtalk.com/ and doing some searching. You'll see some wide ranging opinions on Dedicated vs. Colo vs. VPS.
Go there for an hour or so, ask around. It's mostly a hangout for infrastructure dudes but you'll still learn quite a bit.
I'm ok with running my own machine, so I have a dedicated box from Layered Tech that's not too expensive, and has a gig of memory and hardware raid. One thing you can do with a box like this is, while you're ramping up and don't use the full capacity, share it with a friend.
Related: What's a provider that can give me lots of CPU time when I can no longer host CPU intensive processes from my home computers? What about for cost less than a dedicated server?
After a lot of testing, we went the colocated route. VPS was the second best option but the memory issues (too little) that we've experienced with two hosts were unacceptable.
I am going with Media Temple (GS) I can't give you feedback about them yet, but they cost $20 a month and their grid server can grow with your needs...
I recommend doing a Technorati search on that. Lots of people have received less than reliable service on the Grid. They do great marketing, but their tech is overrated from I've been reading.
I've launched two Rails apps on MT(GS) and my experience has been very good. I believe they did have some issues initially, but I can say that I have not experienced any of the problems others have had.
At the last startup I worked at we had no end of fiascoes regarding our colos. The startup bought over a million dollars of servers and then was paying high-priced employees and consultants to set them up at a colo. But then once the servers were all set up, the colo discovered that they didn't have enough power for our servers. Then we had trouble getting out of the big colo contract that we had signed and had to move all the servers and get them set up at yet another colo. Turns out the next colo also didn't have enough power for our setup even after many reassurances at the beginning that they did. So all in all, this waste of resources put us months behind schedule. Then, ironically we got acquired and no longer even needed our setup because we could then just use the bigger company's server setup.
This is another case of 'premature optimization is the root of all evil'. Seriously, we could have done the whole startup with a couple of Dreamhost accounts and launched earlier with a better product and with WAY (millions) more money in the bank. They were optimizing our server setup for HUGE traffic when they should have just been focused on the product.
I'm glad I was able to see those mistakes close up. It gave me confidence to do my own startup self-funded since I knew how low I could keep my burn-rate.
So now, I'm using EC2 for my servers and it's an amazing experience. I set up a custom server image and then can run multiple instances of it for only 10 cents per server per hour. I can kill instances that I'm not using and bring up a new server within 2 minutes. My server bill is less than $50 a month during development and will be about $100 a month once I launch. I'll also have the ability to scale up VERY quickly and cheaply if I need to.
Anyway, that's my take on it from my limited experience. I'm sure there are some guys that are obsessed with doing their own custom server setup at a colo. Good for them. But you have to realize that you'll have much higher capital costs up front and will be exposing yourself to much greater risk. Focus on your product and find a server solution that is cheap, low-risk, and flexible. Remember you can always scale later - but you have to have a reason to scale first! Have the faith to focus on product and don't fall into the trap of an 'intriguing' 'interesting' academic experiment in server setups!