I wanted to get some thoughts on what sort of hardware configuration I would need to get kicked off (1U colo). Do I need 1 web+1 db server day 1? firewall box?
We're basically starting with the cheapest dedicated server we could find (single $79/month Celeron from GoDaddy). No, it won't stand up to any significant load. However, there's really no reason to pay a lot of money upfront when you have no idea whether your idea will even be popular at all. Might as well save the cash up front, and expand once you know what sort of load you'll be facing.
We keep all our config files in Subversion and try to automate as much of the setup process as is possible. Hopefully that'll let us expand quickly if we need to upgrade to a bigger box or split over multiple machines.
The less pieces you have the lower the complexity. The lower the complexity the easier it is to fix when stuff breaks. The easier it is to fix bugs the faster you can get a real product shipped. You would be amazed how much load a single box can handle. I wouldn't split until I had to.
My current setup is a single FreeBSD box sitting in a colo downtown that putters along without a hitch.
Start with one machine but always communicate with services over TCP/IP and not local IPC mechanisms (unix domain sockets). This way everything is easy to split up, just change the IP from "127.0.0.1" to your new database server, etc.
Also stuff like not storing sessions locally (put them in DB) makes it easy to scale out to multiple web servers.
With some decent memory (1GB+) even a single processor server is plenty to run a sizable site. It's really amazingly what can be done from a single machine these days, given their power and the efficiency of the available tools.
