Re "zero point of failure mail system": I also use Dovecot, and I learnt about DSync recently, although haven't played with it yet. It provides two way synchronisation of mailboxes between two Dovecot servers, so you can store the same mail on two boxes, in completely different locations.
The clever thing is, it can recover from a split brain scenario completely, safely, and without any losses. If your two servers can't see each other for a few hours and you make conflicting changes on both of them, then it apparently is able to recover completely and entirely automatically when the connection comes back up.
You could have one remote server in a DC, and the other locally in your office. Point your mail clients at the one in the DC, but then configure your office router to intercept connections to the DC server and re-route them to the local office server. When you're in the office, you hit the local office server. When you're outside of the office, you hit the DC server.
[edit] Any chance your can elaborate on your S3 test setup? I've considered something similar. I would be interested if there was any mail server software that already does this. The alternative would be to use an S3 based filesystem as your store, but this doesn't seem very efficient. You'd definitely want local caching of messages in this setup.
So, for S3, I cheat and have my own Riak CS cluster so storage is fairly quick and cheap. I should probably use Riak unless the message is too big and then put the body in CS but I was thinking that straight-up S3 is a good lowest common denominator for storing things.
Basically when I started out learning Go I saw a presentation by bradfitz where he made a POP3 server that used twitter for the backend and thought hey, access S3 instead. He had a smtp library on github so I hacked up something that accepts all mail and stores it in a S3 bucket too. I haven't hacked in things like user auth so it's all very proof of concept/useless at this point.
I'll have to take another look at DSync. The SSH batch job per user scared me off last time. I might be able to use it for HA/Failover.
The clever thing is, it can recover from a split brain scenario completely, safely, and without any losses. If your two servers can't see each other for a few hours and you make conflicting changes on both of them, then it apparently is able to recover completely and entirely automatically when the connection comes back up.
http://wiki2.dovecot.org/Tools/Dsync
You could have one remote server in a DC, and the other locally in your office. Point your mail clients at the one in the DC, but then configure your office router to intercept connections to the DC server and re-route them to the local office server. When you're in the office, you hit the local office server. When you're outside of the office, you hit the DC server.
[edit] Any chance your can elaborate on your S3 test setup? I've considered something similar. I would be interested if there was any mail server software that already does this. The alternative would be to use an S3 based filesystem as your store, but this doesn't seem very efficient. You'd definitely want local caching of messages in this setup.