So I am not that very surprised to hear a news article like this. It makes perfectly sense with the current mentality. It also adds another nail in the coffin about idea that "only a machine is reading my emails. Why should I care about that?".
Hopefully this will encourage some people enough to run their own mail servers. It's far less problematic than most people think it is. Maybe it was hard to install and configure this in 1990s, but this is 2013. You buy a domain name, install a Debian machine, and do:
aptitude install postfix
aptitude install postgrey
It always surprises me that people can use a software library with complicated, half insane API's, but can't follow a single page of simple single steps installation. It's not hard and you get the bonus of actually owning your own emails again. If you are a company, this should not even be a question. If the options are to give away all your emails and customers emails to a third-party and thus lose all ownership to them, or asking a sysadmin/programmer to spend 5-10-30m tops to do an email installation, the answer should be obvious.
As for a few examples:
1. Mailservers around the world have different degrees of strictness in what they accept. Some require the sender’s mailserver to have a valid and matching PTR and A/AAAA (!) DNS records. Others don’t care. Some check black lists, some don’t. Some even resolve the MX record of your sender domain, connect to it and try to start delivery of an email to ensure your address is valid.
2. By default, postfix doesn’t warn you about undeliverable messages for quite some time (a week is the default, I think). So if there is any error in your config (e.g. I changed my DNS resolver config, then didn’t restart postfix), your mails will be stuck without any notice, for a number of days.
3. If there are SSL certificates invoked, they should be valid. Some mail servers will not use TLS at all, some will fall back to plain text, others will cancel delivery if your certificate is invalid.
These are just a few examples I have encountered recently, but every time they happen I am incredibly frustrated that emails either did not reach me (usually I detect that quickly) or are not delivered (detected only after a few days).
apt-get install gmail-server
Still, shouldn't be that hard to do as long as it is your own server and you trust it; I imagine a lot of the difficulty of gmail is scaling, and even a single user with 20-50GB of mail isn't in the same league.
If you were to outsource it to a service provider:
I'd love a way to do privacy-protecting search on my mailbox, either by building/maintaining a local index, or even more amazingly, some kind of cryptographic/data structure magic: do processing once either pre-encryption or on the local device, then add it to an index, with a configurable slider for data leakage vs. search quality. Search could execute locally and remotely.
(This is to allow you to use webmail with no local persistent storage, or a new phone, to search your email on a server, without trusting the server).
Mutt is more like "everything I can come up with from the command line." Sure, I won't make my grandma use it, but I've never needed any searching capability beyond grep and co.
I download my email to the Mac mail app. That has let me find some emails that were unfindable on the web interface (even when I searched by sender email!)
Still, I do think there exist some very talented grafic designers out there that would enjoy playing around with a webmail interface. Hopefully, articles like this will increase the demand and interest in run-it-your-self webmail packages, so one would do that and not spend all focus on building more CMSes.
Then there's spam.. what a headache. Suddenly one day I realised that I have better things to do with my time and now everything goes through google servers.
As for hotmail, I strongly remember doing a telnet to hotmail servers and succeeded in sending emails while writing commands by hand on my private computer. Thus without any special headers or other magic, I could send email without any issues. It would had been interesting to hear details on why hotmail rejected your emails. Default postfix installation?
I wonder if there is a "Mail Admin Best Practices" doc floating around the web yet?
This is pretty good. :)
But to address some of those concerns.
While errors in the config can cause big problems, it's not very common for people to change it once it's installed. It's like doing changes to the kernel, Apache, or Xorg settings. Sure, things can easily be broken there but for the common case, the defaults are good once one has passed the installation wizard/guide.
In case of SSL issues between mail servers, I am rather sure that postfix only does a best-effort. Thus if your certification expires, they will fall back to plain text. However most CA's will warn you several times once it gets close to the one year mark, so it's not a very big workload to handle. If you know of an email server that will cancel delivery, please let me know as I would be very interested to hear it.
PTR is a real issue, but if you have static IP, PTR records should be rather easy to get. It is also a one time cost for the domain name. For a company network, a static IP and support is commonly included in the price. You email/call them, and they add a PTR. For private users, a VPN solution might be needed in some cases, but I'm unsure if it's that much of an requirement. Gmail seems fine in sending emails to locations without PTR, or receiving emails from servers without one. However I have not tested this fully or with other services like to Apple or Microsoft.
Reviewing the history of the mailop list  shows how frustrating it can be when a behemoth (AOL, MS, etc.) stops accepting your mail. It's bad enough when you're a sizeable mail operation; as a tiny standalone mailserver, you are not at all a priority, and it's possible you'll lose the ability to communicate with a significant fraction of the Internet's email users (regardless of how complete/correct your configuration) and with little recourse.
Buying a mail server in the cloud is the most practical option. I doubt EC2 would peek inside your VM to censor mail as Apple is doing here.
Running a mail server at home can get complicated. Other than having to manage the hardware yourself, there are limitations on residential internet connections. Not only do residential ISP often provide dynamic IP addresses, they tend to block port 25 (at least outgoing) so as to counter spambots. Some spam filter may also treat mail delivered from your home server more harshly.
Though, if you really can't trust anyone (cloud providers, SMTP relays, or the network in general) you're be better of using PGP.
But Apple isn't in their right mind. They have never understood the cloud, they still don't, and every single Internet service Apple has ever produced, from eWorld to .Mac to Mobile Me to iCloud has utterly sucked gigantic balls.
Which makes relying on Apple to handle your email a lot crazier than relying on say, Google, Yahoo, Rackspace Mail, Tuffmail, etc.
As a developer / sysadmin who used to host his own mail and uses complicated half insane APIs every day, I know that even things that are reliable and easy to set up (such as debian servers) require maintenance and all the work and planning that goes into making something secure, reliable, and performant.
I don't see how anyone with experience could contend with a straight face that running a network service like email is a set-and-forget proposition. I remember having to brush up on my mail stack every time I needed to troubleshoot a problem, and over time it became clear that I had over-prioritized hosting my own email.
Edit: It's a fine hobby project though, as long as you're honest with yourself about what you're undertaking.
(2) set up mail server, create account(s)
(3) reset public email account with domain registrar + hosting facility to newly set-up email addresses
(4) kill public email account
You only need one for a very short while to bootstrap your own email server, you don't need it forever.
But I'll bite, if your mail server goes down - and it will - you now have no email. You can't email your hosting support. You try to log in to your host control panel to do a reboot or raise a ticket but forgot your password. You can't get the reset email. Somehow you manage to login and raise a ticket but won't get any email notifications when the technician responds asking for your server id number...
Thanks, but no thanks. I just went in the opposite direction a couple of months ago, switching to a paid Google Apps account so they can handle email for us. I couldn't be happier. Our Postfix / Cyrus / Squirrelmail setup worked fine, but then the server got compromised and taken over by spammers, out IP got blacklisted by most everybody, and right in the middle of trying to fix all that, I got sick and wound up in the hospital.
When I got home, I sat down to start trying to fix this mess and had an epiphany "WTF am I doing this? Hosting email servers is not a core competency for us, there is no competitive advantage to be gained here, and Google Apps is fairly cheap."
I switched over the same night and haven't looked back. After fighting with Postfix and Friends on and off multiple times over the past decade, I'm out. I want no part of hosting email, now, or probably ever again. I have much, much, much better uses for my time than fixing broken email servers.
It always surprises me that people can use a software library with complicated, half insane API's, but can't follow a single page of simple single steps installation.
I have yet to find an email setup that can be described as "following a single page of simple single steps installation". And even if I could, I still don't want to, because there just isn't any real point to it. Email delivery is a commodity, not a core competence. Outsource the hell out of that shit, IMO.
1) in some industries (like in finance) the regulatory requirements make it near impossible to use external email services. For example, there needs to be a record of every email sent out of accounts in the company that needs to be presented to regulators when they demand it.
2) People within the same company may slip up and send emails to coworkers that shouldn't be read outside the company. When you run the email servers, no other party reads them. It stays in the family, so to speak. When you use google or whatever service, they get a peek at the emails.
3) AFAICT Gmail is not HIPAA compliant, making it inappropriate for health care businesses.
Or to put it another way... we're a 2 person, self-funded, bootstrapped startup. Our most precious resource right now is the time of myself and my cofounder. Every minute we spend tweaking and tuning an email server, is a minute we're not talking to customers, doing market research, working on our product, developing strategy, doing competitive intelligence research, etc. For us, that's not a tradeoff that makes sense.
That and then there are periods throughout my day (especially at the $DAYJOB, like right now) when there isn't anything productive I can really do on the startup, and there aren't any pressing assignments, so reading HN is about as productive as anything else. :-)
Even if you think deliverability is good for 95% of mail servers there's always some ISP that won't cater for you. So you're constantly having to be monitoring the logs to ensure your emails are being delivered.
Sometimes people get hacked, either because they pick a abc123 password (nowdays blocked by policy), or because a customer webserver CMS theme has not been updated in the last 2 years. When that happen, a few mail servers (mostly yahoo) has temp banned mails for a few hours. Mails still got delivered however.
So lesson to learn there is to not use passwords like abc123, and to actually update that 2 year old CMS theme. Or you don't combine webserver with webmail.
Also: how do you know you've never had deliverability issues? If the server on the receiving end is blocking your mail it will happen silently.
In many years of running my own server, this was never a problem. I checked blacklists on a somewhat regular basis but was never in them. And people I sent mail to usually replied, which means they are either psychic or my mail got delivered. (The problem comes when you start sending spam, of course, which is what many people do get blacklisted for.)
The problems with running your own email server are spam and the time it takes to keep things running. All the other issues are trivially solved.
Even on Ubuntu Server it's very very easy to setup a mail server. Linode has guides, and there are others on the interwebs as well.
Just make sure to implement a regular backup regime.
In my experience IMAP to my own mail server is WAY faster than dealing with Gmail (even web interface).
The most conservative approach would be a co-located machine that you own (i.e. your physical property, so they probably need a warrant) or on a machine in your home.
edit: I just tried again, for more than a couple of hours. First I had to guess that I had to install MDB2_Driver_mysql. Set up a DNS to match my /var/lib/roundcube and now I'm struggling to understand how I could sign-in since he didn't create any user for me.
As for users, roundcube uses imap and thus do not create new users.
Anyway, GL, and if you want to have some fun, try out PostgreSQL.
You seriously think this is simple for non IT people?
This is as expected, as most people require some expertise to do technical work. I would have no idea how to fix the electrical system in a car, but I do expect that the mechanic knows how to figure it out if it one broke. Same, I expect a sysadmin or a programmer has the expertise to follow single steps installation guides.
In Germany, email is actually protected by the Fernmeldegeheimnis , even while the email is saved on the provider's server. The reasoning behind that  seems to be that the mail server's storage is part of the communication process, no matter how long the data sits there.
But "There is no postal secrecy law, no rule against reading." Seems like something of an exaggeration.
I used to do greylisting with spamd and ended up silently losing quite a lot of email. (Many mail hosts do not re-send from the same IP, meaning messages essentially get stuck forever.) Doing spam checks at DATA time and rejecting obviously bad emails outright seemed much more effective and less dangerous. I never managed to get Spamassassin to do this, but auto-rejecting languages you don't read also cuts down on spam a lot.
(I missed out on a consulting opportunity because the client's host of choice seemed to be a known spammer and my mail server's filtering was too aggressive about trusting blacklists. I chose to silently-reject those types of messages, so nobody got a bounce. Fortunately, someone was nice enough to ask me about it out-of-band so at least I was able to turn off greylisting and blacklisting before losing much more mail.)
I never found an imapd that scaled to having a lot of messages in a folder, so I ran a cron-job to move mail offline after 2 weeks (for mailing lists) and 1 year (for INBOX). Similarly, I never found a good client to use; Gnus had a very cryptic configuration that I could never believe worked, and mutt was not Emacs-y enough. Reading my email mostly consisted of waiting for Gnus and deleting spam. (I never figured out a good way to get Gnus to move messages marked as spam somewhere so that I could run a cron job to automatically train Spamassassin on the known-bad emails.
For search, I used HyperEstraier: http://sourceforge.net/projects/hyperestraier/
You set it up to run the ingestion program as a cron job every 10 minutes, and then you get a CGI that will show you matching emails for your query. Of course, you can't actually click the links and go anywhere unless you set up some sort of web-based email viewer. I never found anything I liked so I lived without webmail. (There are lots of options. All difficult to configure and probably riddled with security holes.)
Once your server is up and running, you need a secondary MX and a backup plan for your email. (I used Dyn.com's secondary MX hosting service. A lot of spam comes in through the secondary MX, so you can't just implicitly trust it. This involves more configuration.)
Finally, spam filtering uses a lot of CPU and RAM, so you have to pay for a rather expensive virtual machine. Linode's $40/month plan seemed mostly adequate.
I don't really like composing email in the web browser, but I've gotten used to it and $5/month for Gmail and $0/month of my time screwing around with spam filters seemed like a good tradeoff.
Then it goes through amavisd that does the spamassasin checks, as well as verify DKIM and the like, and then it gets delivered to dovecot.
Now, Dovecot as an IMAP server is fantastic. Along with dovecot-pigeonhole I can sort messages into different folders server side.
Dovecot currently handles one mailbox for me that I archive a mailing list into ... 150k messages and counting, and no issues. Uses Maildir on the backend. Although, I do think at those sizes it comes down to good file system caches, and a good file system that doesn't have a problem caching the entire directory.
It works well for me, so well in fact that I am moving all my stuff from Google Apps for Domains back to in-house. For two reasons, 1. more control, and 2. I have had issues with Google Apps in the past, and even with a paid account the support has been lackluster. I'd like to know more about my email, be able to check logs if need be to see what is going on.
Using it effectively, and knowing the dark corners of scalpeling, is quite another thing. It would be easy to get into deep trouble with a scalpel.
Note that I do NOT know much about email servers and that is precisely the point. I eat iptables rules for breakfast, I can set up a Debian server from scratch and I can certainly configure a mail server. But there's more to it than meets the eye. Been there, done that. And we got burn.
I don't know "why", but all I know is that we had problems!
tl;dr: 30 team person startup and we tried to use our own mail server. Too many problems. We moved to Google Apps for domain and all our problems were fixed...
As for using a previous owned spammer domain, I strongly question that logic. Black lists tend to be both conservative and done by copy-pasting. A blocked domain abcd will not block abc or abcde. If the block lits do not see mails from abcd within a month or so, they also tend to remove them to clear up space. This time frame is also way below common domain name expire dates.
Regarding reverse lookup, I assume that 30 team startup bought a company network contract from the ISP. Such deals should include support and static IP, which mean reverse lookup is a single email away from being done. Co-location contract might be different, but I wouldn't know.