I've set up a few mailservers mostly through manual changes to the configurations and if you just need something simple and secured with SSL/TLS, it's not too bad. From my experiences, the only major headache is how intricate you want your user and password configurations to be.
Even if you have DKM/SPF setup, someone can decide to start spoofing email from your domain to send spam, especially if your domain has been around a while and is relatively clean. Since you don't send a lot of real mail, all of a sudden 99% of the email marked as coming from your domain is spam.
It's not always even obvious that you've been blacklisted; mails may simply be significantly delayed (AOL does this, I guess so they can check for similar messages to other users), etc.
This is an issue for email providers as well, but they have people on staff paid to deal with it.
Even if you only have to deal with this once every 2-3 years, that's still generally going to be worse than simply paying someone $5 a month for a managed email service, not to mention to you have to deal with backups, etc.
What kind of spoofing are you talking about? Attempting to spoof the headers is common, but no blackhole list looks at that - they blacklist based on the IP of the machine the spam is coming from.
If fact, every email server setup I've seen explicitly rejects emails simply due to absent or inconsistent rDNS records.
First and foremost, static IPs are a must. Second, make sure those static IPs haven't been misclassified as dynamic or dialup. Then get reverse DNS setup properly (some providers won't do this). Then setup SPF. On top of all this, make absolutely 100% sure you don't run an open proxy. Then keep an eye on blacklists and your logs, and be ready to call and deal with obstinate tech support for hours on end. DKIM, and having some way to reject emails at the envelope stage (to save bandwidth, avoid double bounces, and punish the guilty rather than bouncing to innocent parties being Joe-jobbed) are also good ideas.
It's not horrible, IMHO, but it's not necessarily a cake walk either. You might need to have an understanding employer so that you can take off for half a day to deal with things.
It was difficult to find it. Dreamhost was great because I can create a lot of e-mail accounts for my company but in the last year there were a lot of outage that complicated my business. On the other spectrum Google/Yahoo/Rackspace could be great but they are expensive when there is a linear relationship between e-mail accounts and price, and we create new e-mail accounts all the time not only for employees but for testing specific pieces of software.
Leaving Gmail might be uncomfortable, but as long as I can backup my e-mail via IMAP and can move my address at a moments notice, it'd just be a nuisance.
I don't see the point in avoiding Google for the sake of it. But I do see the point in making sure the the effort involved in switching a dependency away from them is reasonably contained.
It is Ruby on Rails application with Ruby scripts to autamate installation and configuration of E-mail server. Post-install configuration is done througth web application.