Hacker News new | past | comments | ask | show | jobs | submit login
A Hacker's Replacement for Gmail (dbpmail.net)
464 points by dbpatterson on June 30, 2013 | hide | past | web | favorite | 215 comments

While I don't necessarily trust an external company with all my emails, I also don't trust myself to maintain the myriad daemons involved in this setup without doing something subtly wrong that results in my server not sending/receiving all the mail it should -- or, worse, being used for spam.

What would be useful is a pre-assembled virtual machine image or other form of appliance that allows you to deploy and test a mail server within about an hour or so, without having to duct-tape any of this together yourself.

I've been hosting my own mail since 1996. It's actually one of the easier services to self-host:

1) SMTP was developed for unreliable environments. If you have problems with uptime, your incoming email will bounce around for 5 days before it gets dropped. So assuming you can get your SMTP server running one day out of five, you shouldn't be in danger of losing anything.

2) Contemporary daemons like postfix and dovecot have sane defaults, so even a naive default install should be mostly secure. They're also extremely low velocity, so once you set it up there's not a lot of ongoing maintenance.

Hats off to ya. I used to view setting up your own email services as a rite of passage for any UNIX admin worth their salt. SMTP was one of the first services I ever got working on my home Linux box, and I had to do it with Sendmail and the goddamn Bat book. Nowadays it's a bit easier.

It's easier now, in that we have good well-documented software, but the external environment has changed. While other servers used to just accept the email you sent, spam countermeasures have gotten complex enough that if you just follow the postfix installation guide you're going to have a lot of your outbound smtp filtered.

Grab a free account at mailgun.com and configure it as your outgoing SMTP relay.

You'll get an IP address for your outbound traffic which is "clean", monitored and registered with a ton of ESPs. You can also use Mailgun as a proxy for your incoming mails as well, for spam filtering or custom routing purposes.

Full-Disclosure: The comment above was made by the co-founder / CEO of mailgun.

Is mailgun a US-based company that would comply with a national security letter if faced with one?

What about Digital Ocean or another VPS provider? What is to stop them from just handing the NSA a copy of my server image complete with all email history, address book, and authorized PGP keys? I'd have even tagged and indexed all the mail for them!

Yes, that's the point. If you are worried about Google cooperating with the NSA, and you decide to roll-your-own mail solution, but are using US-based services like MailGun, you are doing it wrong. :)

Moving to a self-hosted solution (even a US one) offers you more privacy protection options than Gmail/Hotmail, that's for sure. But since physical access is everything, using a US-based VPS provider means there is only a small speedbump between the government and your mail. Using a US-based service like Mailgun, while extremely cool, removes even this speedbump, since they will presumably be forced to cooperate in the same way that Google or Yahoo do.

The best option would be to host your own mail with a VPS with a very strong privacy record, explicit statements about not cooperating with US inquiries, based and hosted in a country with strong privacy protections.

Out of interest, how would you go about doing this, especially interested on the inbound?

I have not experienced the troubles you mention with postfix defaults (actually debian's postfix defaults). Can you think of some examples to be on the look out for?

I was the same way, although I preferred NetBSD, but I eventually determined that it wasn't worth the effort to host my own email. That said, I still learned configure it for local only use. I grabbed incoming mail using Fetchmail and sorted it with Procmail.

By the way, I still feel that Sendmail has to have the worst configuration file I have ever seen.

Sure, it takes 30 seconds to install postfix, but in practice, email is stupidly hard to admin because of spam. You may be able to send email easily, but a large percentage will be blocked as spam unless you use a well know relay even if you do everything right. Similarly, you will also be fighting a flood of spam in the other direction.

Inbound spam : greylisting as stated elsewhere.

Outbound : Use a smarthost.

I host my own mail. I use OpenBSD as my Firewall/Router so it was a simple decision to use the spamd daemon on it for greylisting. The SMTP destination sits on a CentOS machine behind the OpenBSD gateway. I see a spam message maybe once every week or two. I have also used MailScanner to do SpamAssassin and virus scanning of received mail at places I have worked. The spam count went down so far I had people ask me if the mail server was down. A little bit of verification showed we weren't loosing legitimate mail, users were just getting so much spam they thought that was normal.

A smarthost solves your outgoing problem. I can't send direct from my home connection because it is a residential IP and that is going to be blacklisted all over the place. I could use my IPS's SMTP server, hell you could use GMail if you wanted. I however, use a VPS I pay for. My SMTP server logs in (password) over a TLS connection (preshared certificates). It only relays for my mail server because of that.

You don't need to use a well known relay, you just don't relay from a blacklisted IP. Don't leave an open relay and don't send directly from a residential IP block and you probably don't have to worry, unless you start sending spam.

http://www.mxtoolbox.com/blacklists.aspx has various tools to give you a view into how other mail servers will view your hosts.

> but a large percentage will be blocked as spam unless you use a well know relay even if you do everything right

That's hardly a rule. You clearly will end up on a blacklist if you host from home or if your IP is already on a list due to previous spam activity, but of you run off a static IP in a decent colo and have a proper PTR record, it is more likely than not to be all OK.

The 90% of inbound spam is still very effectively trimmed off by greylisting, i.e. postgrey if you are in postfix. I've been running my own mail server for close to 10 years now and it's really not a big deal.

What do you use for spam filtering?

Another problem is you will have to use a third party smtp server, otherwise your mail will be rejected by a lot of email providers.

SpamAssassin and friends still do a reasonable job for me, and have for years. I delete a handful every day, but like the parent I've had my email address since 1996, and I haven't been too careful about using it. If I had to do it over again today, I'd look at rspamd. Getting people to accept your mail can be a pain in the ass, yes, but it's not impossible.

* Get a "clean" statically allocated IP address.

* Control your forward and reverse DNS and make them match.

* Set up SPF/DKIM.

* Fill out the Yahoo Bulk Email sender form

That last one probably isn't really necessary, depending on how much email you send to Yahoo.com addresses.

I've been running my own mail server since 2006 and have never had a problem with being rejected. I've run a fairly tight ship:

1. Only allow email locally (i.e., via webmail and ssh) - which many would regard as a real pain, but I and the few people who use my server have been OK with. IMAP is very useful with this policy, for batch move/send from outbox folders.

2. Set up TLS to only deliver via SSL

3. Advertise both these facts via SPF (I haven't bothered with DKIM)

Do you have any tips on testing for deliverability? I have a hunch that if I try to send something, say to my gmail or one of my close friends' email, that the spam filter will allow it because it's a familiar name/domain, but I would expect that if I email a stranger's gmail, it might get flagged...

You can use http://www.mxtoolbox.com/ to see if your domain or IP have been blacklisted by any ofthr public lists. Also, if you get blacklisted you should receive a bounceback email telling you who blacklisted you and where to go to get yourself removed.m

That's bollocks. I have my own email server on some random Hetzner IP address and have never had email rejected; and do note my domain TLD is .ro which tends to get higher spam scores.

Try OpenSMTPD its now ready for production use http://www.opensmtpd.org

I've been meaning to finally get around to doing this myself, so I'll shoot you a question: Do you have a fallback MX? I'm thinking of getting a super-cheap VPS running only Postfix as a mail fallback in case my primary host goes down for an extended period. How is accessing mail on the fallback host (when primary is down) usually handled? An IMAP daemon running there as well? Should the fallback just wait for the main one to come back and then send all the waiting mail there?

Usually, the backup MX just gathers email (so that the email does not bounce to the sender) and delivers it to the primary MX when it gets online and then the primary delivers emails to the server that is going to expose it via IMAP or POP3.

But this is just for receiving emails (from the outside world), if your IMAP/POP3 server is on the same server as your primary MX you will not have access to emails on the server while the primary is down. You have to find a way to sync your received mail (maildir/mailbox) to two or more servers.

If you decide to implement a backup MX, try to sync your allowed recipients list from the primary because spammers often try to send emails to an MX with a lower priority, and if it accepts all emails for the domain without checking if that mailbox actually exists you could became a source of back-scatter.


Perhaps you'd like something more integrated? There are plenty of projects in this space that are entirely or mostly open source.

http://www.kolab.org/ http://www.sogo.nu/english.html http://www.zarafa.com/ http://www.open-xchange.com/home.html https://www.zimbra.com/

Citadel is another integrated solution that is really easy to setup. Just one application, all written in C. The web interface is the only thing that I think could be better.


"Citadel offers versatile email services with very low administration needed. It provides its own implementations of these server protocols: IMAP, POP3, SMTP, ManageSieve, XMPP, Citadel."

Wow, I'm impressed!

Call me behind the times, but I like the fact you can DL a virtual appliance, and have it working in 10 mins flat.

I didn't mention Citadel because there hasn't been a release since November 2011; however on further investigation it looks like development is still active. http://code.citadel.org/?p=citadel.git;a=shortlog

I've been looking at Kolab. They have a nice page listing their dependencies:


All of these look fine - I already deploy all the server items - except Round Cube: does anyone know anything about them?

I've used roundcube before, both on a shared hoster and on my own server later on. It's okay, I wasn't impressed with the user interface at the time though that seems to have changed judging from their website. The software is a bunch of php and shell scripts and uses a MySQL, PostgreSQL or SQLite database.

Thanks. It doesn't sound hugely convincing. I guess people do harden PHP+shell scripts, but it sounds like work to me. eitland thinks Kolab can be run without it, at the expense of not having a webmail interface, which would be OK.

Have got it recommended from a consultant who seems to know what he is doing.

Also I think in kolab it is only used for optional webmail.

E-mail and the extensions and extras that make it go and make it nice /are/ duct-tape. I wouldn't advocate for /anyone/ running their own mailserver unless they're totally aware of what they're doing and on top of spam issues.

When considering if running your own mailserver is practical for you, consider the total cost of ownership; you'll most likely be paying for a dedicated server (or vps, VM, or whatever the cool kids are using these days) which will require staying on top of all the "normal" things; monitoring, backups (and you'll probably want to test them), updates, what blacklists to use, ensuring that /your/ server doesn't end up blacklisted, etc. Unless you've been a sysadmin before and know what this all entails, I wouldn't really recommend it. If you're not part of some bigger org (a hosting co or bigger company), it'll also be harder to get your one-off no-name server removed from blacklists, since this looks just like something Joe Linkfarm would do. I could show you this one really simple trick that a mother found to whiten your server reputation, but...

As far as creating a VM image that sets all this up for you, it's kinda the point that it doesn't exist; mailservers don't work well when run by people who don't really care about them. It's quite far away from a "set and forget" deal. This is why many hackers / sysadmins still use GMail; it's not the best at everything, but it's OK enough at most things that it's the least hassle to use.

That all said, I do agree that it'd be great to have a service which offers both more control for power users (I'd love a mailfilter-style config w/regex support for GMail), better privacy (easy PGP integration, etc), tag support, threading, and normal IMAP support, but I'm not quite holding my breath. This seems like a decent problem for some start-up to solve :) It's not a "sexy" problem, but it's a real one.

> What would be useful is a pre-assembled virtual machine image or other form of appliance that allows you to deploy and test a mail server within about an hour or so, without having to duct-tape any of this together yourself.

There at least a couple Linode stack scripts that will do this, but I haven't given any of them a try since last time I checked they were mostly/all Ubuntu based. Dovecot + postfix isn't all that difficult to set up, and there are literally tons of guides all over the place (Arch wiki, Linode, Dovecot's site, etc.) for dovecot, postfix, courier, and anything else you might want. Perhaps the most dumbfounding thing for a beginner is the certificate step...

Bitnami offers a ready-made mail solution with roundcube.[1]

[1]: http://bitnami.com/stack/roundcube

I was about to say, doesn't this scream for a container? i.e. docker.io

Agreed. Setting up the mail server is such a nightmare. I'll gladly pay for a for-dummy-solution.

To play the devil's advocate, what exactly is the practical use of all this if most of your family and friends are on Gmail (and couldn't be arsed to figure out pgp)? From what I can see, your emails will now be sent in the clear over the internet, instead of staying within google's servers. Either way, the government's going to get your data, but at least you're protected against... /more/ unscrupulous people snooping on your stuff?

That's a great point. Reminds of the time I taught my friend to use PGP and sent him an encrypted email. Every single time, he would reply in plain-text, thus exposing my older conversation. When asked why, he told me it's too much of a pain to do it. So my being careful about my privacy doesn't help if other people don't play along.

I've been experimenting with encrypting using bitcoin addresses, publishing keys with gravatar and sending encrypted messages as links as a way of trying to make all this stuff easier and more accessible. If you're interested in my proof of concept, it's http://kybernetikos.github.io/VisualSecrecy/

Linking bitcoin address to email address doesn't sound like a step forward in privacy...

Depends what you want to use it for. Anyway, you can have multiple bitcoin addresses for different purposes.

On top of that, while I wouldn't rely on this cryptographically, the relationship is one-way. Gravatar links email addresses to gravatar images, but it's not intended to link in the other direction, so the same is true for any information stored in the gravatar.

You can use my system to reply with encrypted text to a comment on a blog post without knowing the email address of the person you're replying to, only their gravatar image.

The OTR plugin for Pidgin does a better job of userfriendliness and gives more useful privacy guarantees (like plausible deniability).

FYI, last I remember, Pidgin stored your passwords in plaintext in an ASCII file on disk, unless you jumped through some hoops to integrate it with your desktop environment's keyring. They even have a article up on their site explaining why it's necessary to do this.

You can disable storing passwords. If you don’t want to have to enter a ‘master password’ when Pidgin starts up, there is no way they can store the passwords more securely than plaintext.

Get full disk encryption and be happy.

That's simply not true. For example, on Windows, they can use CryptProtectData. Mac OS has a keychain function too. Pidgin devs are being disingenuous by suggesting that accessible to current user is identical to storing plaintext on disk.

Full disk encryption and per-user encryption are good steps. But an accidental backup of Pidgin will still reveal passwords that would be safe if they bothered to use platform specific APIs for such storage.

Pidgin is predominantly developed for Linux, where they would have to support Gnome, KDE and probably at least one other mechanism. Sure, that could be done, but it is a lot of work to do that sensibly on all platforms, and, more importantly, of questionable sense: I would classify the logs of my conversation as much more relevant to a potential attacker than the mere password to my XMPP account.

All of this is true. But when the Pidgin devs state things like "there's no way" and "it's just as secure" (as they do on the wiki), that's just incorrect. An intellectually honest description would note that many platforms offer protection, but it's not standardized across Linux (I'm assuming).

The logs of the conversation can be protected in the same way, so I'm not sure what that has to do with anything. (Although you might wish to keep logs as plaintext, to facilitate backups, if you're not backing up the user's keychain info.)

The wiki explicitly says (under “Is that the final word?”):

> No. The Pidgin developers are generally open to, and would encourage integration with keyrings (KeyringSupport).

and then goes on to state that this is difficult to do, since Pidgin runs on so many different platforms, then stating again that they will happily accept such patches.

However, I find it understandable, that the devs don’t go out of their way to support use-cases they don’t feel necessary to support. On their systems, they trust the filesystem and are happy with that, if others don’t have that level of trust in their computer, that’s fine, but not necessarily their problem.

My point about the logfiles was that to store these in the keyring (rather than a key to the encrypted files) would probably annoy the keyring somewhat (at least the poorer implementations thereof), given that it is intended for use with few-byte passwords and not multi-megabyte logfiles.

The wiki does mention some info, yes. But it also makes a false comparison, noting how you can extract passwords on a system, and implying that the level of security is identical. That's 100% false, and to suggest so is being very misleading.

As far as logs, do what everyone does when you can only store a bit of material: Store your bulk encryption key there.

This is the exact reason I created this: https://privatemsg.matthew-dove.com/

Where can I grab source code?

Can such kind of reply put your secret key in the risk? For example, it gives attacker enough information to deduce the key within reasonable time?

I don't believe so. I think the parent comment was more concerned about the message itself being sent in the clear.

The shuttering of Reader and Gmail's new and awful Compose have convinced me - more than years of Stallman rants and NSA snooping - that relying on closed-source SaaS is a terrible idea. It is a question of if, not when, your SaaS provider does something you intensely dislike, and which you have no power to change.

With SaaS, you don't even have the power to say - Damn your improved version, blast your hip new design, I will stick with the old one, thankyouverymuch. You are always on Version Now.

The more intertwined your relationship and dependence on integration with related products, the messier the ensuing divorce.

This, to my mind, represents a far more inevitable reason to figure a way out of GMail.

I see encrypted connections to/from gmail all the time. Here's an example from a test I just ran:

Trusted TLS connection established to gmail-smtp-in.l.google.com[]:25: TLSv1 with cipher RC4-SHA

Anonymous TLS connection established from mail-yh0-f48.google.com[]: TLSv1 with cipher RC4-SHA

Hmmm, no Perfect Forward Secrecy on RC4-SHA...

Would it be considered paranoid to read anything into the fact that they offer ECDHE-RC4-SHA for https sessions, but only RC4-SHA for SMTP connections?

It could be an artifact of my postfix configuration. Looking over a few logfiles, here are the ciphers I've seen recently (all servers, not just gmail):

ADH-AES256-SHA (256/256 bits) AES128-SHA (128/128 bits) AES256-SHA (256/256 bits) DHE-RSA-AES256-SHA (256/256 bits) EDH-RSA-DES-CBC3-SHA (168/168 bits) RC4-SHA (128/128 bits)

Curiously, I'm seeing a bunch of TLSv1:DHE-RSA-AES256-SHA:256 between one of my machines and google...

I wonder why/when the connections end up RC4-SHA instead?

Google serves up static content uses DHE. We've noticed only Google sites using DSA certs with ECDHE. A report on what we found http://trisul.org/blog/dsa-xdrill/post.html

> Hmmm, no Perfect Forward Secrecy on RC4-SHA...

Actually dont see how you can have PFS on DHE either if one of the endpoints doesnt co-operate. You can simply dump the master keys and provide those to the decrypting app.

Or just dump the plaintext data.

If I tell you a secret, and you tell someone else ... there's not a lot I can do a about that. If you don't want a third party to be able to hand over your plaintext (or store it) -- don't give them your plaintext (or a means to access your plaintext).

Similarly, if I send you a PGP encrypted email, I can't know if you decrypt that and hand it over to someone else (willingly or unwittingly).

I can though, still assume that if I send you a GPG encrypted email to your gmail account - I've only got to worry about _you_ leaking the contents, not Google.


hQIMA13LrXtLThhwAQ/+LmYMzmaQ3Ui0AF5yRKzCVL/rXzUO3h+cKZVnA2AL/SAR PHcVjgGkm4BT3C8pokeTl+UQPqsBj/i3gteC0zi5QTMyXYxnkCC6915yVGON86BS E5i+pEpXIubnWiKZh81Ik+YARYnTqi+Ea5zW0OAzKmd48FX9m21MK0fKHcdjoYZk 56JaMbTgcSTcW2RIztwQr9EeTnf/XIHsIrhQuOGmZd9kTmbxn9mA+W2AKzgPmv7s Z+RUgEMrbyjNK+s2V/ibPE0CDpBKR6cleWRmAgEknu2Z8QaBIgiv+a64mKMbtL6I H8ZCcM1djgBmXvjfHRwJEvEKEIfJKVQ5Q1SMyskAkWt23CQIbd1toLzx/2e0F0O3 Zjppm+qnBhM6JUOnuc5L42uvZK1+0L3aT99UX5L2xOV8OdqgVto1u+d/Q35LUhNl jjslEKidDxhxFWVHJvVhY/4ogQZIq4WrEpDMoYjRzniECMi779MTl6UnX0vRjVuw 3dbXppozqhB40P7q9Om+ORXGfMrzpIRwABltY6NI5PPjeFgHeNZ/gAFxfWn6INYa mielp57irCYBAVaVIodds2EZNSJ3o8m8A/p4HKbuS8W1qDkU2QY4k+Ns27LY3EQM 0fXx6Ug5INql6vHQpj02W4q4S8A0FipS70WZIH4eWm/aLDWV4PT/0hMoGAhYPgvS lgFoQeoAPeaPJ+Tlb1WX5V7cePBH3EZte+0WcBwlZBBejCBNVAjpyFUG4jMcOv/B IPPa+7IFWjE/1kf7n6e+/OsqDjXWem2j5wJd8R0SJlJk97/VjGDvYAn2mdNCqQ43 /sLRy5oTgEE+kljtFriL5Qfdhkei5UR8RZxV3Yv3J+ARohj7JJovSi0psR9hVI5J BGL2emenig== =Eqf+ -----END PGP MESSAGE-----

Oh, absolutely. I just wanted to point out that if you're communicating with google, you're communicating with google -- so even if they enable perfect forward secrecy over smtp/tls -- that's not a 100% fix.

It is still better than them not enabling it -- because if we can assume they do not log the data by default, on their own (aka: we can trust google) -- old data won't be accessible once a (theoretical) new warrant arrives.

None of it is impermeable, even your own server. It's just another layer of misdirection and/or difficulty.

I'm not sure why you can't do those things on FastMail.

(disclaimer: I work for FastMail)

Sure we have folders rather than tags, which means you can't add multiple of them to the same message. Probably the biggest lack is that you can't manage IMAP flags via the web interface. Otherwise, our search is now very powerful (since about March this year) and allows you to build filters that show messages from multiple folders in a single view.

OP here. Fastmail had 4 out of my required 5 features. I used it for a while. I'm still a paying customer (I paid for a bunch of years). Tags were a deal breaker. It's just not sufficient for how I want to organize things.

I'm going to raise this in our next meeting (Tuesday) and see if there's a way we can have IMAP flags exposed somehow. Along with fast cross-folder searching on flags, we could quite easily implement virtual folders per flag - which I think fills your use-case perfectly.

The difficult parts are:

1) UI 2) limits. We have a hard limit of 128 "user flags" because it's in 4 x 32 bit fields in a fixed-width data format. Subtract a few for our internal tooling, and you probably only have 120 you can use for yourself.

Would 120 be enough?

One thing that many older clients did was had $Label1 => $Label5. That was often enough for people... so I suspect 120 is probably fine.

I think if you'd had that before, I never would have left (and I think many other people would be happy with that).

Just another happy FastMail user here. I don't need tags/labels, so you have everything I need.

I have an Enhanced account that is linked to my domain, and an Ad Free account where important mail gets forwarded to. The latter is accessible from my phone, but the former isn't, so if anyone steals my phone they can only see the last few messages I exchanged, and I can just disable an alternative login (similar to revoking an API key) to lock my phone out permanently. I also set up my default personality in the Ad Free account to send all mail through my Enhanced account, so every mail I send from my phone is automatically saved in my Enhanced account and even has the correct DKIM signature.

One question, though, and I'm sure a lot of people are curious about it. You are an Australian subsidiary of a Norwegian company, but your servers are in the US. What happens when an American three-letter agency wants to see the contents of your servers?

Jeremy Howard, the founder of FastMail, responded to a similar question on Slashdot in October 2009. http://tech.slashdot.org/comments.pl?sid=1391605&cid=2963235...

  We've had a number of US-based law enforcement bodies over
  the year try to get hold of our data without going via the
  appropriate Australian bodies, and it doesn't work out for
  them. In the end, they have always ended up submitting a 
  request for cooperation via the Australian Federal Police,
  as they are required to do, and we respond to that request
  in line with Australian law.
Even so, as an Australian I decided to use a hosting company whose servers are physically hosted in Australia.

That was before the Opera acquisition and the subsequent changes to their infrastructure. I wonder if their servers are still encrypted?

Fair enough. Hosting in Australia is something we consider occasionally, but the bandwidth pricing is just so much higher. It's always worth looking at demand of course - are there enough people who would be willing to pay a premium to have their email in Australia to justify running up a full instance here.

I'd pay extra for Australian located email hosting too. Or even a European option.

Also, since we're talking feature requests: 1) contacts syncing via CardDav and 2) Calendar.

Contact syncing is vital though. It's really a killer at this point.

I currently do pay a premium and I'd prefer to be paying it to FastMail.

Recent revelations mean there may be a few more Australians willing to pay a premium to host mail in Australia.

Also, thanks for your work on Cyrus.

I've just registered FastMail for free trial. Its speed is amazing, much faster than Gmail. And its UI looks great, too. Thank you for your work.

It would be better if you have a more desirable .com domain name. I will probably buy an Enhanced account to use my own domains after free trial.

We have a bunch of .com names as well. But I agree that your own domain is the best approach, both for the ultimate in portability, and the ability to choose whatever localpart you like.

I signed up for Fastmail enhanced, really like it so far, only things I don't like are the read only LDAP contacts and the security settings. Specifically on the security settings, I want to see support for a limited alternative login which can access IMAP desktop/mobile clients (such as K9) but not have full unrestricted web access. Tags, I can without those if they can't be implemented well. Thanks for a great product.

I love FastMail. Thank you for your work!

Cheers :)

I've been using FastMail for a while and I like it a lot. Actually I love the fact that it uses folders instead of tags.

The only problem I have with FastMail is that I've been receiving an increased amount of spam in my inbox.

Did you set up the advanced spam filtering and then train it on all of your folders? I also set the super-advanced options and changed the thresholds to around 3.0 (instead of the default, which was I think 5.0), which really helped. It's better than when I hosted on gmail, but I'd say 5-10 per day still get through and have to be reported, after about a month on fastmail. That's only a couple of percent, though (scanning my spam folder, I get something like 10-12 per hour, but this is an e-mail address I've had and not hidden at all since 1997).

I contacted support and they did setup the super-advanced options for me. It seems to be much better now.

Tags would be useful, but the UI aesthetics could be better too: now it's too plain and minimalistic. Look at Yandex Mail for example: it's just less boring. I guess you could attract much greater audience by adding theme support and proper marketing in light of this outrage.

That said, I like fastmail very much: keyboard navigation, search and overall speed are great!

Happy paying FastMail customer here. Would love to see CalDav support (public and private), so I can ditch Google once and for all.

Seconded, especially with Calendar sharing with specific other people.

Another happy Fastmail user!

I just wish it had an address book and calendar that I could sync.

Those are the things I miss the most from Gmail.

Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it.

That's still a lot better than Gmail.

Setting up your own mail server is not a terrible woe-inducing undertaking if you have a working recipe to follow and are comfortable with the Unix command line (e.g. http://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204...).

Organization and categorization are the sticking point features, given what I've seen of most open source webmail applications. But worth looking around. If you have a basic mail server image, you can keep trying out applications on top of it to see what works for you.

Going beyond that to something with a whole lot more encryption and less of an ability for hosting providers to read your data would really require a product dedicated to that end: that is hard to get right.

> Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it.

How exactly is that any different than it was 6 months ago?

The difference today is that not only should I assume that Google sysadmins could read my mail, but there's now pretty convincing arguments that Google are sending all my (as a non-US person) email to the NSA - who're storing it forever "just in case it turns out useful".

I'm somewhat less concerned about rogue individual sysadmins curiously snooping on my mail, than I am about the NSA's comprehensive perpetual archive.

What? Have you been following this topic at all?

Exactly where are these convincing arguments that google is sending all e-mail to the NSA? Links? Data? Do you have anything that supports this at all?

There is the telephone stuff that was leaked, which has been well known since 2006 [0]. But, where is this miraculous evidence that google is handing everything over to the NSA?

[0] http://yahoo.usatoday.com/news/washington/2006-05-10-nsa_x.h...

Yeah, I've been following, with the point of view of someone explicitly not protected by your 4th amendment rights. (I'm assuming, possibly incorrectly, that you're a US citizen?)

As a "non-US person", I read all the "We disclose user data to government in accordance with the law" explanations from Google/Apple/Yahoo/Facebook et al - as saying "Oh, you want our data from that guy in Australia? Sure, that's 'in accordance with the law' - here's all his email… Anything else you'd like? His social graph? His contact list/address book? His Adwords clickthroughs? His Google Analytics enabled website visits? Oh look, it seems he accidentally put his GPG private key into his GoogleDrive folder - have that too!"

Do you really find it surprising that people outside the US see Snowdon's revelations and the US government's reactions to strongly suggest this is the only sensible interpretation? _All_ the denials/re-assurances we're seeing are of the form "in accordance with the law", then going on to talk about presumed protection against _domestic_ spying, and the 4th amendment rights of US citizens. None of which is reassuring from where I sit.

Though I'd agree that non-US persons have reasons to be concerned, you moved the goal posts. It may not be particularly difficult for the NSA to get your data from Google services if they decide to target you specifically, but Google has explicitly denied broadly sending over data from some significant fraction of users, as you suggested in your previous post:


Maybe - but I'm reading it with the suspicion of someone who's just having been served a wake-up call on the exact nature of a friendship, and finding it's a lot less mutually respectful than expected.

I still see that "Second, we provide user data to governments only in accordance with the law." glaring out at me from the middle of that denial.

My suspicious mind now reads phrases like " … and frequently pushes back when requests are overly broad or don’t follow the correct process" and "Press reports that suggest that Google is providing open-ended access to our users’ data are false, period", as saying "so long as 'correct process' includes a FISA rubber stamp, and that '(claims of) open-ended access to our users' can only be true if it includes all domestic US users" - then sure, we can say that while still handing over _all_ data on non-US users -it's still _technically_ true.

This is for me, kinda like when someone you thought was a close friend gets married and doesn't invite you to the wedding - nothing's actually changed, but you now view everything differently. Right now I've got that "Oh, _that's_ how it is huh? I guess that explains a few things, I wish I'd known earlier…" feeling - complete with that sense of foolishness for ever having thought things were different.

It might be paranoid - or it might just be pragmatic. Either way, things _are_ different now. I just didn't get invited to The US's wedding, while Apple, Google, Yahoo, Amazon, Skype, TeamViewer, PayPal, eBay, Digital Ocean, Linode, Rackspace, Dropbox, Evernote, Trello, Freshbooks, and a bunch of others all did. Now I have to re-evaluate all those relationships too.

How paranoid is it to wonder if I can trust 1Password any more?

The issue is not what we know, but what we don't. Keep in mind, all we have is what was leaked. We only have a tiny glimpse into the massive amounts of post-9/11 surveillance. We can ascertain that the government is doing extensive amounts of surveillance on large amounts of people, and that the NSA has either refused to tell Congress the scope of the programs or outright lied about it (take a look at this (http://www.wired.com/dangerroom/2012/06/nsa-spied/) article from a year ago , and look at Clapper's "Not Wittingly" statement).

To ask for "Links? Data? Do you have anything that supports this at all?" is asking the wrong question, since the amount that is public is incredibly small (just a tiny fraction would be my guess, but all we can do is make estimates). The question is, Is it reasonable to assume that there is more surveillance? The answer is yes, and to assume that there is detailed email surveillance, either at the ISP level, through backdoors, or through direct cooperation is not unreasonable.

"The answer is yes, and to assume that there is detailed email surveillance, either at the ISP level, through backdoors, or through direct cooperation is not unreasonable."

I'd suggest _not_ assuming that might be considered negligent.

It's not just the NSA. All big countries have intelligence services and anyone with anything to hide should assume that they will try to get their hand on as much internet traffic as they can.

So e.g. if you're a US company that has a competitor in China, and you don't want the Chinese company to read your email, it's a sensible precaution to encrypt it.

The point about trusting hosting providers is an interesting one. Indeed, when renting a Virtual Private Server from a service provider, you have no choice but to trust them to keep your data safe.

This made we wonder: would it be possible to actually secure the server in such a manner that the hosting party won't have access to your stuff without your say so ?

I think you can (sort of) do this already with having something like an encrypted virtual server running on the rented VPS. Of course, I don't think this is bullet-proof and you also do have the downside of an additional layer of overhead that comes from further virtualization of your actual server(s).

A good compromise is to just run the actual machine in your house and rent a VPS for the sole purpose of installing a VPN server to provide your machine at home with access to a static IP address that doesn't have outgoing SMTP blocked. That way all the VPS is doing is forwarding data to and from your machine at home and it doesn't have access to anything your ISP wouldn't, and TLS/PGP/etc. goes a long way to help with that as well.

That should keep anyone from reading your email (assuming you and other senders are using TLS), but the next problem is that an ISP-level observer can always tell who you're corresponding with unless you use something like mix nets or Tor. And allowing that requires some cryptographic authentication method to distinguish legitimate anonymized senders from spammers without leaking the sender's identity to an intermediary. Having the sender sign the message and then encrypt the message, the signature and anything else that identifies the sender with the recipient's public key ought to do it but I'm not sure if there is any existing software that will actually do that. It might be possible using a combination of PGP to authenticate the sender and SMTP TLS over Tor to actually transmit the message, but the missing piece is for the sender authentication to be integrated with the spam filter.

No -- unless you have exclusive physical control of the machine you'll always be vulnerable.

A malicious VPS provider has access to the machine's RAM, which means they can do almost anything. For example, they could extract your SSH private key and silently decrypt all your traffic.

Is such an attack easy? No. But I could compile sshd with debug symbols and it would be. Even without them it's still possible, just very difficult. Though nothing a skilled employee of a nation-state couldn't do.

As far as I know, SSH v2 offers perfect forward secrecy, so the attacker would have to read the session key from ram, not the private key (the private key would allow the attacker to do MITM, though).

I have a dedicated server with encrypted partitions and admin backdoors turned off at ovh. So theoretically they shouldn't be able to access the running system, and if they take it down to access the partitions directly, they're encrypted so that won't work either.

That's why they take memory snapshot first, which is trivial with VPS and then pick encryption keys from it to access encrypted volumes. This is well known method and works with pure hardware machines too with physical access. It's great question when you get to server, to shut it down or leave on. If on, it could destroy data, if turned off encryption keys are gone. I think it would require some individual case analysis before deciding which one is better approach.

Isn't the pure hardware method for memory snapshots some ridiculously complex mechanism with freezing the memory and quickly transferring it to a reading device?

If you're going to pull this, you need to know in advance that it's necessary, that's not some minor thing that everyone is going to just do automatically, it's an extra, complicated step it's easy to screw up.

That said, I acknowledge the possibility of compromise in the aforementioned scenario, but once again I don't understand people who always jump to the "I am vulnerable to this narrow threat model, ergo, I should not bother to protect myself against any threat models". Especially when the measures you can take to protect yourself most likely will address the actual threat models you are liable to encounter.

"ridiculously complex mechanism" ... Not really. Anyone with the ability to stick a USB stick in a USB port and hit a power reset button can pull off this attack:


Hmm, that requires USB booting though, if you disable that from the BIOS and password protect it, they can't really use this method. If they pull the battery the machine needs to be turned off and so the RAM will clear.

You don't get BIOS access to low cost dedicated servers on OVH, even though they're dedicated, you can't KVM them.

mid to high range ones you can though so that might be a workable solution there.

Also, they can still rip the ram out and nitrogen it.

Interesting stuff. TRESOR might not be overkill after all.

> Isn't the pure hardware method for memory snapshots some ridiculously complex mechanism with freezing the memory and quickly transferring it to a reading device?

With Virtual Private Servers, it's trivial to do the virtualized version of this through the hypervisor.

My original statement said "dedicated server". The response said; "virtual server attack, also works with pure hardware machines". The only version of this attack that is relevant to the original statement is one that works on a dedicated server, so re-stating that it's trivial to do this against a virtual private server isn't really adding anything to the conversation.

How do you do that with a pure hardware machine with physical access without rebooting the machine? Even if you freeze the memory with liquid nitrogen and extract the data from it, that process results in the server needing rebooted afaik. Obviously I'm assuming there's no 0-day you can just enter in via the serial console or similar.

You might chill the RAM, reboot, read out the encryption keys, mirror the disks, verify that you can decrypt it -- and then leave the server off -- mail the customer a notice saying the server rebooted due to a power glitch.

At this point you have pretty much everything on hand to help you fool the customer into believing everything is ok (eg: replace the whole physical machine with a vm...).

Come to think of it -- if you know the hardware in the machine, you probably could replace the machine with a vm anyway -- just mirror the disks and go.

The right answer. With a design that focuses on simplicity ( reducing the entry points ) you can be pretty well assured that if someone attempts to gain access to your data you will likely know.

For me that is an important factor. I rather dislike the fact that in scenarios like Gmail and other providers of their ilk -- access is provided transparently to a third party.

Running your own server can give you a reasonable expectation of privacy. Even in a hosted/colocated environment if you take the time and effort.

That said, even with a dedicated server, I still use gmail, because https://github.com/etherael/phoneme

Possibly some too-cynical questions…

Do you think ovh is any less beholden to GCHQ than Google et al are to the NSA?

Do you think your encrypted partitions and turned-off admin backdoors protect you much against people with physical access to the hardware?

> Do you think ovh is any less beholden to GCHQ than Google et al are to the NSA?

OVH maybe a little more, because at least they're not under the direct auspice of the most megalomaniacal state in the world and will act according to their economic incentives in being a reliable and trustworthy service provider, which in a purely free market would align their interests with my own. However being as they are subject to various hostile state entities that have the ability to exert deleterious effects on that self interest, to the extent that they are forced to, they will cooperate with those entities.

I take action to protect myself from those threats to the greatest extent that I am able in either scenario whilst acknowledging that every theoretical threat model can never be entirely negated, coupled with the fact that at the end of the day, bothering to break through all of my measures and taking that trouble, they would find basically nothing that they were interested in.

I despise the state, sure, I'd like to see it dissolve into something like the system outlined in the machinery of freedom, sure. However in my view, taking any violent measures to expedite this process would both make me as bad as the enemy, and compromise the goal of a better world even if it actually contributed to the downfall of the state.

> Do you think your encrypted partitions and turned-off admin backdoors protect you much against people with physical access to the hardware?

"Much" more than non encrypted partitions and not turned off admin backdoors? Absolutely. Completely and utterly impenetrable to any attack by any means at all? Not likely, I can think of one off the top of my head that would work; freeze the memory, get an image, power off the system and extract the encryption keys from the image to re-mount the partitions. That's 1) not automatic 2) not easy 3) easy to screw up 4) not undetected

Great response - thanks.

Another question - have you done the thinking or got some advice about whether marginally trusting a potentially subvert-able hosting company in a jurisdiction you don't particularly trust (US, UK, and unfortunately for me, Australia) is likely to be a better or worse bet than trying to source a server/vps in a more trusted jurisdiction? (perhaps Iceland? Or am I fooling myself assuming there's anywhere "trustworthy"?)

I used to be Australian, too.

I think of it like trusting gangs in the thieves guild, there might be some gangs that are ethically more admirable than others, but at the end of the day if it comes down to it, the administrative board of the thieves guild is able to compel any of it's member organisations to act in accordance with its edicts.

So if you're going to take any action that might cause the administrative guild of that thieves board to attack you, it should be a foregone conclusion that the trust of intermediary parties beholden to that central authority is irrelevant.

Arrange your affairs accordingly. If you must deal with gangs of thieves, deal with the better ones, and if you can avoid it, don't. And if, heaven help you, you decide to take action that will paint crosshairs on your back from the central thieves guild committee, your opsec should take into account the full powers that committee is able to bring to bear upon you.

As a former employee of OVH, I can guarantee that law enforcement compliance does happen, I don't know to what extent, but having been there for the better part of two years I have seen requests from local government be held up and data handed over to authorities. Though in those cases it was child pornography accusations.

Which could mean that the only thing an unscrupulous government official needs to do in order to access one's data is hinting at implications on child pornography.

How do you handle reboots? If all of your partitions are encrypted I would guess you use a serial console to enter in the encryption password to decrypt and mount the actual OS drive?

small boot partition (basically enough to boot, get network, start ssh), all the actual data and /tmp on encrypted stores.

But do you validate that boot partition each time you reboot the system? How do you know that your computed hashsum (or similar) is actually the true one? ...

got a hash of the boot partition, validate when it comes back up.

How do you know that you're not logging in to a vm, with all the data mirrored from your physical server?

This is reasonably clever, but how do you get a full disk mirror from a server you can't get into without powering off, if you do power off, the amount of time the system is down is a tipoff to the target as to what's going on.

Assuming however it could somehow be pulled off; I guess potential ways to mitigate would be to keep a copy of the exact hardware characteristics of the system you originally have, kernel log on bootup, precise size of disks, chipsets of all the various controllers and compare when the system reboots. It would however be possible though extremely hard to get an exact duplicate of all of these at the virtualisation layer level.

You could call the virtualisation layer in the cpu requesting access to virtualise a subsystem, clock the data bus speeds... There should be some overhead from virtualisation that wasn't there when dedicated...

It's an interesting problem. I'll think more about it.

Please note that as soon as you've typed in the key needed to unlock the disk in the vm, your data is compromised. So any verification you attempt, has to be made from the unencrypted boot image...

As for time needed, assume this: The attacker sets up a physical server that can accept the same physical disks as your server (with hotswap), with an additional disk (or ssd) for booting into a hypervisor, sets parameters for this hypervisor to closely mirror that of your server. The attackers server would typically be faster than yours -- it is/can be bought some time after your server was, so this should almost be universally possible.

The attacker boots the vm server, sets everything up; then kills power to your machine, moves the disks over. Your downtime: the time it takes to move the disks (literally).

Good luck differentiating between this and legitimate downtime.

I still think reading the encryption keys from RAM after cooling them with some co2 would be the more likely attack, but it's a fun thought experiment...

Not sure if anyone will see this, this late, but someone else recently posted this presentation on Rakasha, a malware based on top of coreboot open bios and friends -- so if you know the type of server, you might just pop up the cover and switch out the BIOS...:


> Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it.

Indeed. If you want to be secure, you need to keep your email server at home, or in some other location you trust, and make sure all communication on the net is encrypted. That way, if an adversary wants your secrets, they will have to burgle your house.

> That's still a lot better than Gmail.

True, because it takes effort for GCHQ/NSA/whoever to look at your server, which they probably won't do.

> Going beyond that to something with a whole lot more encryption and less of an ability for hosting providers to read your data would really require a product dedicated to that end: that is hard to get right

That's what I'm working on.

There is a very good chance that an email server running on a "home" IP address block of any of the major ISPs will be blacklisted.

True. If your ISP won't remove your IP from the blacklist, you could probably use a commercial service (a la Amazon's SES) for outbound SMTP while running your MX and mailstore at home. That would have somewhat different privacy tradeoffs, but it's not that different if you expect most of your email recipients to be hosted elsewhere... I may try this.

Assuming you don't have any sort of console access configured and its a physical server how could they do that without letting you know via at least a reboot?

First, they could always ask your ISP to route a copy of each smtp to their servers.

But this is a bit irrelevant until you solve the social problem: I'm going to guess that >90% of your contacts use gmail/yahoo/hotmail; so the NSA already has 90% of your mail in their databases anyways- until you convince your contacts to install their own email server.

When I saw the link's title, I immediately thought that it would be another webmail client hosted by someone, especially given the domain name. Because almost everytime I see a "hacker's X" or "X for hacker" title on HN I'm just afflicted by the content, so I got used to it.

But here, what a pleasant surprise. The post is actually describing a real hacker's replacement for Gmail (which coincidentally is almost my setting, except I use mu [1] instead of notmuch). I'll keep it as a reference to send people asking for alternative email hosting.

[1] http://www.djcbsoftware.nl/code/mu/

I did this for a long time, but it's really annoying:

1. If your provider goes down, you lose mail.

2. If you are conversing with people who are using an insecure mailer, such as gmail, Yahoo, etc (which is probably > 99.9% of all e-mail users), your e-mail is still accessible to the NSA, or to some Fortune 100 advertising company.

3. It's only a matter of time before the "big dogs" in email abuse the position and decide who is and isn't allowed to send/receive email outside of their little oligarchy, either on their own or at the behest of governments.

Like so much else that has been corrupted, we need to scratch the current architecture as too insecure, and build something truly secure for the future. This isn't in the interests of the Googles of the world, and it's actively in the worst interests of the NSA/FBI/CIA, so it's probably the right thing to do.

> It's only a matter of time before the "big dogs" in email abuse the position and decide who is and isn't allowed to send/receive email outside of their little oligarchy, either on their own or at the behest of governments.

Uh, what? We have the tools today. TLS to your mail server, PGP for your email content.

SMTP is already a highly distributed system that is controlled by no single entity. If the big email providers decide to block all the little players, making your own message passing system (with blackjack and hookers) and convincing everyone to come use it is many steps more work beyond just staying on SMTP and convincing everyone to stop using the big email providers. Then at least you don't have to develop new protocols, you don't have to develop new clients, and you don't have to convince people to use them (but still, good luck with that first step).

How about instead we just make email encryption a whole lot easier to use and promote email providers that aren't beholden to court orders (aka probable cause must be demonstrated at a minimum)?

Meanwhile we can work on reforming the security apparatus in the US because 'reasonable expectation of privacy" most definitely includes all the geographical locations I access my email account from, amongst many other things, so should require a warrant.

"Like so much else that has been corrupted, we need to scratch the current architecture as too insecure, and build something truly secure for the future."

This is really the core technological issue that has enabled much of the recent mass spying: the Internet really was not designed with security, privacy, or anonymity in mind.

Remember that it started as a research network that was used primarily among academics. Academia is generally a very open and trusting environment. The last thing the inventors of the Internet though of was trying to protect their computers, data, or privacy. We are living with the consequences of that mistake.

The second major cause of this mess is the computer illiteracy and historical ignorance of much of the world's population. They don't understand how they are making themselves vulnerable by sharing so much information about themselves and trusting corporations that provide "free" web services to them. This is slowly changing for the better, as people become more technologically savvy and stories like the current spying scandals and various security breaches hit the news.

As for the technological and security minded among us, particularly those who are just now starting to think about running their own mail servers, what took you all so long? None of the recent revelations should have been unexpected.

The Internet needs a major privacy, security, and anonymity overhaul. If it's not rebuilt with those concerns at the core, they will all remain mostly illusory to the overwhelming majority of Internet users.

I see a couple of problems here:

1. It's likely he's storing emails on the VPS. This puts us back at square one. A third party has a copy of your emails. And we know email does not garner the same privacy protections as postal mail.

2. You need a domain name. That system (DNS), as it is currently implemented (i.e., everyone setting their root zone to servers they do not control), is highly centralized -- few people maintain their own root zone, despite being easy to do. Domain names are susceptible to false allegations copyright and trademark infringement by private parties, not to mention easy censorship by the US gov't. When you lose your domain you lose email. (Though you shouldn't have to: email works fine with IP addresses in brackets.)

So what's the solution:

1. Get a reachable IP (e.g., through ISP) or get a VPS. But if you get a VPS only use it to pierce NAT (how is left as exercise for reader - hint: supernode), not run a mail server. Don't store sensitive data like email on a VPS, or route sensitive data through it.

2. Use IP addresses not domain names. Alternatively, set up your own DNS that is available as a peer-to-peer service, or have your email contacts use a DNS server and root zone you collectively maintain: free domain names that you control. No one can censor your DNS (phonebook), except you.

While k9mail is a must, I suggest linking to the repo, which is usually lightyears ahead of public releases on the Play store.


On a side note, they seem to have just hit v4 two days ago.

Second side note, if you decide to use k9, be sure to turn off the signature under composition settings for each account you add.. it's turned on by default.

>handing an advertising company most of my personal and professional correspondance seems like a bad idea

That's your main complaint? Google is an advertising company. People buying ads on Adsense don't have access to your personal information. This is simply not true.

Mining user profiles for targeted AdSense ads is just the tiniest part of the problem.

Massive government spying revelations based on tapping into centralized infrastructure at cooperative organizations that maintain broad cross-web and cross-mobile profiles of all users, along with their personal communications and data, and you don't see how big this problem really is?

Many of us did similar in the 90s. I might go this route again but would use Postfix and Dovecot. I'd do this for my wife and kids as well - but if I get hit by a bus, email eventually not working is not something I should burden my wife with.

Just prepare for that eventuality. Make an upgrade path in the event of such a thing, and include it in your will.

Tip: don't use dovecot-lda. Let postfix handle it. Dovecot LDA is chroot hell.

I worry about the same things which is why I'm actually migrating my email to outlook.con slowly (in spite of the NSA etc). She knows my passwords already.

I always thought a big part of the reason people used gmail was for the snazzy web-based UI that was one of the first popular AJAX-based web applications.

I eagerly read the article to see what alternative to this feature the author was suggesting, so I was surprised to see he's reading the emails with a standalone client...in fact, it's an emacs plugin!

Exactly. It's hardly a solution. Viewing in-body images and attachments turns into a nightmare.

Normal people definitely don't want to manage a mail server though. Life is too short to waste figuring out why you're banned on Spamhaus for the 93th time.

GMail sucks, but a home-made contraption is not the alternative.

To be fair to the author, I don't think an article that starts with "A Hacker's..." and/or involves Emacs has a pretension to being a solution for normal people.

Yeah, I know. I'm just pointing out that GMail alternatives should exist that don't involve a huge PITA.

I know how to configure mail servers just fine, but I still wouldn't do it for myself. That's what I mean by "normal".

Why must this be pointed out every time? There's enough of us weirdos out there.

> Normal people definitely don't want to... Life is too short to...


All you are saying is that if you aren't looking for a home made contraption, then a home made contraption isn't what you are looking for.

I am not normal.

We could push a step further: "EMail Server as a service for common people". Somehow like http://instantserver.io/ .

Or like Heroku: you create your "managed" mail server with a click.


I had a similar setup a couple of years ago. The main problem I had was the maintenance required. If you have any machine publicly accessible you have to be on top of security updates and proper system hardening. I gave up after my exim4 Debian system got 0-day rooted.

If doing it again I would avoid a Debian based distro. I'd probably use openbsd. And the less ports open the better.

Nice stringing together of unixy tools to get this working. I had not heard of notmuch and its related ecosystem (afew, alot, etc.), so that's a useful discovery on my part.

It probably is not much popular.

I've thought about doing this, but email is important enough I don't trust myself to provide as much uptime as a commercial email provider.

You probably should add SPF records too, if you don't want your outgoing mail marked as spam.

Reliability doesn't matter that much. SMTP is store and forward so if you Bork something, it'll still get to you eventually.

Reliability for incoming mail, sure. But there's making sure outgoing mail is delivered and not spam filtered, making sure your ip isn't on any blacklists, managing and testing backups, keeping up to date on security issues, etc. All of which are doable, but that's not really how I want to spend my time.

I ran a set up similar to this for many years. It's not that hard, for those with a little unix experience. As moxie mentions, email is very forgiving—you have to break it badly and leave it broken for a long time before you start to lose messages.

What eventually drove me to GMail was spam. I tried a bunch of different filters, and never found one with good-enough accuracy. Finally I decided that the independence and privacy wasn't worth the time I spent fiddling with filters and dealing with misclassified messages. As far as I can tell, Gmail is 100% accurate. Problem solved.

> As far as I can tell, Gmail is 100% accurate.

My experience is I've had a few false negatives and false positives. Gmail is still very good though, and any replacement, if it is to be widely used, needs to solve the spam problem.

If each message had to be separately encrypted for each receiver, would this add much to a spammer's costs? I'm guessing it would, but not by enough to make spam uneconomic. A better solution might be to require that the first email someone sends someone else, unless they've been OK'd by a third party, contains bitcoins to the value of $0.01.

Fun fact: Bitcoin's proof-of-work algorithm is based on Hashcash, which was an anti-spam measure along those lines: a message contains a partial hash brute-force that takes the sender about a second of CPU time to compute (and depends on the recipient). But it didn't take off, and is problematic when spammers have vast amounts of CPU resources in the form of botnets.

> As far as I can tell, Gmail is 100% accurate.

I can confirm that that is not correct. Google regularly flags internal ticket email as spam for one of my larger customers (mail that is sent within GApps). I also see about a 10-20% false positive rate on our various mailings (account setups, billing notices, etc).

It would be incredibly useful if there was a mail service that received email over SMTP, encrypted it straight away with a public key, then just dumped the encrypted email into a general-purpose online storage solution (e.g. an S3 bucket).

That would IMO provide a good base for encrypted client-side apps to build on top of. Open source would better be able address the problem of writing a client once the money needed for hosting and storage is taken out of the equation.

Hushmail and Lavabit sort of do this. They encrypt emails before storing them. However, as far as I can tell they handle the public/private keypair on their own, so the security only goes as far as you trust them.

Countermail does this: https://countermail.com/

I was imagining an even lower level service. I guess that CounterMail must store unencrypted email headers in order to serve IMAP, right?

I was thinking something that just manages the problem of being online 24/7 to receive email. This (and possibly sending email) is the only thing that can't be done completely client side. A service like I was thinking of would just accept messages, immediately encrypt them (headers and all) with an RSA key and then dump them into some third party online storage system (maybe sending a notification over XMPP too).

The rest of the email pipeline could then be run completely client-side. I'm talking about doing everything client side: spam filtering, sorting into folders or tagging, running email rules, indexing, and of course viewing and reading email. (A clever filesystem/database would need to be layered on top of the online storage system to manage the state of the pipeline and provide fast indexes.)

For maximum ease of use the client-side app could even be a rich Gmail-like JavaScript app that stores private keys locally and but stores most data remotely (e.g. uses S3 directly via CORS).

This would all allow hackers like us to build interesting (but still encrypted) email services without having to worry about infrastructure. That's something I don't want to manage anyway!

Also the barrier to entry of writing a Gmail clone would be much lower because you could do at almost zero cost. Since you don't have to receive email or manage storage your service would mostly be static JavaScript containing the client-side logic. Most - even all - of the service could just be served on a CDN.

I think more of us need to run mail servers. For ourselves, for our families, and possibly for others who are willing to pay. Email is far too centralized now, at a handful of companies, in a handful of data centers. So in that regard, running a mail server on a VPS at one of the popular providers is kind of missing the point.

My local cable ISP doesn't allow incoming or outgoing connections to port 25, nor incoming connections to port 80. So at least for now, I can't run a mail server in my home. I've thought about switching to DSL, but then I would take a major hit in speed, in both directions.

Luckily, I have another option. There's a hosting provider where I live (Wichita, Kansas) that offers KVM-based virtual machine hosting. So I'll get a VM there, and if the service is any good, I'll move there from Linode. The pricing isn't competitive with Linode, let alone DigitalOcean, and I doubt that the connectivity is as good, since the server will be in a building here in Wichita rather than a real data center. But I'm willing to try it, in order to support a local business and fight the centralization of the Internet.

Hmm... It sounds easier to just run an instance of Zimbra community edition in a VM.


This is what I do, but it requires more than a $5/month VPS to run it.


True, but $10/mo should cover your needs fine. https://www.digitalocean.com/pricing

It's definintely time for an open source alternative to GMail... but I think everyone knows this isn't it.

These tools like exim, horde, dovecot, etc. have been around and worked for decades but wouldn't it be great to have fresh solutions that weren't so ancient and archaic?

You say ancient and archaic, I see ridiculously well tested and battle worn.

Seriously, what would make a "fresh" solution better than a long standing one, in this case?

You don't have to reinvent the wheel. Presumably the "better" solution would be using most of the existing code under the hood. But wouldn't it be nice if setting up an email server consisted primarily of typing something like "apt-get install email-server" and then setting the domain name and adding a few accounts?

I can agree with that, but I don't see it so much as a "fresh" solution as it is an improved UX on top of the existing ones. Of course, the reality is likely that the actual complexities of the problem will make any good UX hard to do. That is, realize that for the vast majority of us, gmail and friends are this "fresh" program.

That's essentially how it happens in most good package managers. You install a choice of MTA/imap/antispam, set up the domain name and unix accounts and SSL certs, and up and running at this point.

Most of the modern smtp and imap servers have very sane defaults.

I made a very extensive search for an open-source webmail solution that exported a REST api. Unfortunately, none of them do (that I could find). Most of them are from the XMLRPC days.

I'd definitely contribute to a kickstarter for a good FOSS REST Email server.

  | good FOSS REST Email server
You should specify what you're talking about here. An all-in-one solution? An SMTP server? An IMAP server? A webmail client (e.g. SquirrelMail / RoundCube)?

I imagine it would be similar to what i've been looking for.

Central app that manages the storage, sending (server to server) and receiving (server to server) of email exposed over a rest api

Out of curiosity, what was the reason for not picking a more traditional Dovecot + Postfix setup?

Debian decided to use exim4 by default, so I figured that it would be better supported / documented. Which for the most part was true. There is no reason why postfix+dovecot couldn't work equally well - I was familiar with none, so I chose what seemed like the past of least resistance.

Exim has had five serious vulnerabilities since 2010; two root privilege escalations and three remote code execution bugs. No confidence.

Agreed. I used Exim4 for a while, but after too many issues, I eventually switched to Postfix, which have been run smooth ever since.

I would have thought a client side encryption plugin that will seamlessly encrypt/decrypt all your Gmail sent between yourself and any other user running said plugin would be a simpler option. Adding common mail suppliers as it goes forward.

You could use something like Mailvelope [0]. You still have to manually encrypt things, but the process is fairly straightforward.

[0]: http://www.mailvelope.com/

Was hoping to see more discussion of backups. There are a bunch of possible approaches (depending on level of desired security, what the VPS provider offers, and how much you trust them), but for a mail server, there ought to be something...

I currently use mbsync to backup my all mail folder on gmail. There is a way to set it to never delete on your local side, turning it more into a backup than a sync.

I run that daily, and Crashplan takes care of the rest. Sure it doesn't save the folder structure and what not, but I'm ok with that.

Use OfflineIMAP to sync to a local disk.

Sync is not backup.

I've started defining 'hacker' as someone who's willing to 'eat their own dogfood' as it were. Someone that is willing to spend time working on the nuts and bolts that lead to some kind of productivity rather than just being productive with the tool / service to begin with.

I used to classify myself as a 'Hacker' and still do when it's something I want to learn more about. Most of the time, however, I'm more interested in just getting the benefits rather than tinkering with the internals. Sometimes, I'm a Hacker, sometimes I'm a consumer.

Does anyone have a gmail exporter?

i.e. something that imports email from gmail WITH labels.

This is a proof-of-concept someone wrote, but I have no idea how tested it is: http://git.zx2c4.com/gmail-notmuch/

It syncs Gmail labels to notmuch tags.

As I understand it: by default, Gmail treats labels as IMAP folders, which means a standard IMAP exporter will end up with multiple copies of messages with more than one label. The workaround is to only export the "All Mail" folder, and then add support for Gmail's custom attributes, namely retrieving the X-GM-LABELS [1] attribute for each message and then doing something with it (in this case, storing it as notmuch tags).

[1] https://developers.google.com/gmail/imap_extensions#access_t...

I believe gmvault.org should be fine for that.

You can use offlineimap to do that. Every label is an imap folder. offlineimap allows what they call as "folder filters", using which one can write a short python function parameterised over the folder name.

Why make it so hard? Why not just install virtualmin [0] on a Debian (or whatever Linux you prefer) server and get it over with. You also get web hosting, DB hosting, mailing lists, webmail and more as a bonus. And you don't have to worry about security updates. Just install and create your virtual host, and modify DNS for your domain. Couldn't be easier. Oh, and it's completely free.

[0] http://www.virtualmin.com

Just a general fyi, there are $10 credits for Digital Ocean if you decide to sign up. The one I used this month was OMGSSD10, but it may only work for this month.

I've got (and am very happy with) a DO VM, but in the context of securing mail - I'm not sure using a US based server (or a server owned by a US based company). (And, though I'm in Australia and have an English passport, I'm not sure UK or Australian servers are any better…)

Know any hosting companies in the travel lounge of a Moscow airport?

If you want a really good and really easy to setup mail server I would recommend SmarterMail. It is also free for 1 email user. I have used their product for about 10 years. Note that it is Windows server based.


Another relatively painless way to set up a mail server on your own box is http://www.iredmail.org. ( + http://z-push.sourceforge.net/soswp/ for ActiveSync push mail) So far I am fairly happy with it.

Does anyone know, if I used z-push for a product, would I be liable for not buying a license from Microsoft, for Exchange ActiveSync? It seems they are fairly litigious with activesync patents.

The protocol is patented [0]. If you want to offer your product in the US (and other places that care about software patents), you need a license.

Also note that Z-Push is AGPLv3 software. Unless you can get it under another license from Zarafa, you have to release your product with an AGPL-compatible license.

[0] http://www.microsoft.com/en-us/legal/intellectualproperty/IP...


Interesting. I remain a step behind this one in that I'm using offlineimap to sync local maildirs from google's servers and then using mu and mu4e in emacs. Means I get to use the Gmail Android client which is actually very good.

as for antispam..http://www.mxhero.com/ is auto and easy, and i think http://spamcheck.sourceforge.net/ is very nice. u don't get as many controls but it sends quarantine digests, and runs much lighter than mxhero. http://www.scrolloutf1.com/ looks nice too. at work we sell spamtitan, which i admit is really nice..tons of config with nice gui, good defaults, easy setup..but its non free...

Why use notmuch when one can have standard (RFC5228) Sieve support?

linode has nice guides...e.g. https://library.linode.com/email/postfix/dovecot-mysql-debia... though for postfix, having something like postfixadmin makes it nicer.. or perhaps something instant like http://www.xeams.com/ will do..

An interesting solution if you have something to hide I suppose.

As has been stated time and again, most people don't. The danger lies in politicians, ceo's and other figures of authority who do and can be blackmailed. Rather than a few hackers setting up their own SMTP servers I think a more powerful solution lies in keeping focus on the actual problem, the out of control NSA program.

awesome... its time move away from proprietary, snooping services such as gmail. Hopefully setting up such a service should become easier ( may be less than 5 steps ) with better cloud VMs. Then even non-tech savvy people can have their emails away from snooping.

Run your own server FTW

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact