
Mail-in-a-box: easy to set up modern SMTP/SMTPS server stack - api
https://github.com/JoshData/mailinabox
======
thisishugo
It remains a constant frustration to me that Postfix, Exim, Cyrus, Dovecot and
the like still feel as if they belong very much in the "here be dragons"
territories of the Unix world. Configuring these systems is an exercise in
constant frustration and bafflement.

They're such a pain to use that since becoming the sysadmin in charge of our
work email servers, I gave up running personal ones as well and just pay
FastMail to deal with it for me - ain't nobody got time for that.

I often dream of taking a sabbatical and writing modern, user-friendly SMPT
and IMAP[0] servers. If only so I could use them at work on my return and save
myself a lot of time and stress.

[0] Possibly even with Exchange ActiveSync support - Microsoft licenses the
protocol, but I've no idea what it costs [http://www.microsoft.com/en-
us/legal/intellectualproperty/IP...](http://www.microsoft.com/en-
us/legal/intellectualproperty/IPLicensing/Programs/exchangeactivesyncprotocol.aspx)

~~~
jauer
It is very much "here be dragons" but a lot of that is because of legacy
compatibility issues from when mail was delivered directly to local users.

If you don't care about local access (IMAP & POP is good enough, no mutt to
the mail spool for you) Dovecot is a huge improvement on everything else. You
configure it to listen directly for LMTP and use MySQL for user information
and Maildir for data it's almost as easy as running a random php+mysql webapp.
Even serverside filtering with sieve "just works".

The inbound SMTP/Spam filtering stack is still a PITA but that's because of
security issues (spam).

DBmail (dbmail.org) is one rework that seemed sane but stores all messages in
a RDBMS and I didn't want to deal with scaling it at work (little ISP, ~10k
mail users, ~1.5 million messages per day including spam) but it was fine for
personal use.

I've been playing with homegrown POP & SMTP servers that use a s3 compatible
datastore as the backend but that's a side project. People go crazy over email
so I really want a simple to operate, sane, zero point of failure mail
system...

~~~
mike-cardwell
Re "zero point of failure mail system": I also use Dovecot, and I learnt about
DSync recently, although haven't played with it yet. It provides two way
synchronisation of mailboxes between two Dovecot servers, so you can store the
same mail on two boxes, in completely different locations.

The clever thing is, it can recover from a split brain scenario completely,
safely, and without any losses. If your two servers can't see each other for a
few hours and you make conflicting changes on both of them, then it apparently
is able to recover completely and entirely automatically when the connection
comes back up.

[http://wiki2.dovecot.org/Tools/Dsync](http://wiki2.dovecot.org/Tools/Dsync)

You could have one remote server in a DC, and the other locally in your
office. Point your mail clients at the one in the DC, but then configure your
office router to intercept connections to the DC server and re-route them to
the local office server. When you're in the office, you hit the local office
server. When you're outside of the office, you hit the DC server.

[edit] Any chance your can elaborate on your S3 test setup? I've considered
something similar. I would be interested if there was any mail server software
that already does this. The alternative would be to use an S3 based filesystem
as your store, but this doesn't seem very efficient. You'd definitely want
local caching of messages in this setup.

~~~
jauer
So, for S3, I cheat and have my own Riak CS cluster so storage is fairly quick
and cheap. I should probably use Riak unless the message is too big and then
put the body in CS but I was thinking that straight-up S3 is a good lowest
common denominator for storing things.

Basically when I started out learning Go I saw a presentation by bradfitz
where he made a POP3 server that used twitter for the backend and thought hey,
access S3 instead. He had a smtp library on github so I hacked up something
that accepts all mail and stores it in a S3 bucket too. I haven't hacked in
things like user auth so it's all very proof of concept/useless at this point.

I'll have to take another look at DSync. The SSH batch job per user scared me
off last time. I might be able to use it for HA/Failover.

------
jgj
With every solution that comes along to solve the "host your own email"
problem, and every guide that takes you step-by-step, I always find myself
just wanting a huge wiki-style site with all the topology of a good, secure
email/groupware server explained in plain English rather than a package that
purports to do it all for me. Something with breakdowns/discussion about the
different options that exist for the different pieces that make up the whole.

It's one thing to easily obtain my own personal mail server, it's another
entirely to understand and be able to maintain/improve it.

~~~
dsirijus
I found this series of articles very helpful, if a bit verbose -
[http://arstechnica.com/information-technology/2014/02/how-
to...](http://arstechnica.com/information-technology/2014/02/how-to-run-your-
own-e-mail-server-with-your-own-domain-part-1/)

------
morsch
Very nice. Comes with Roundcube so you've got a webmail client available.

    
    
      - DKIM signing on outgoing messages (opendkim).
      - The machine acts as its own DNS server and is automatically configured for SPF and DKIM (nsd).
    

I remember many people running their own SMTP server used to have massive
issues with their outgoing mail being silently or non-silently ignored or sent
to the spam folder. Does this take care of this? "Mostly" really isn't good
enough.

~~~
Niten
> I remember many people running their own SMTP server used to have massive
> issues with their outgoing mail being silently or non-silently ignored or
> sent to the spam folder. Does this take care of this? "Mostly" really isn't
> good enough.

Frankly I've had mixed results over several years of hosting my own personal
email server, even with a valid PTR, SPF, and DKIM. Most SMB and personal
email systems I interact with are fine, as aside from Bayesian filtering those
tend to rely heavily on blacklists which I am responsible enough to keep
myself off of—easy given that I'm the server's only user. And I've never had
an issue sending to Gmail users.

Microsoft is the real f-up in this regard, originally with Hotmail and now
with the updated Outlook.com. Even though I'm on no blacklists and pass both
their SPF and DKIM checks:

    
    
        Authentication-Results: hotmail.com; spf=pass (sender IP is XXX.XXX.XXX.XXX) smtp.mailfrom=redacted@example.com; dkim=pass header.d=example.com; x-hmca=pass header.id=redacted@example.com
        X-SID-PRA: redacted@example.com
        X-AUTH-Result: PASS
        X-SID-Result: PASS
    

my messages still end up in the Junk folder.

If you visit their support page you're recommended to join their partner
Return Path's Sender Score Certified Email program to stop junk filtering of
your legitimate messages. That's great, if you're a medium-sized organization
that doesn't mind paying their certification fee; but Return Path won't even
allow an individual to register (I've tried). I'm familiar with Hanlon's
razor, "never attribute to malice that which is adequately explained by
stupidity", but this at least _feels_ like a racket.

Unfortunately I don't know what else to do, other than continue to advise
everyone I know not to use Outlook.com and friends.

And to finally answer your question, if this VM runs its own DNS then it
probably takes care of SPF and DKIM adequately, but you'll likely have to
contact your hosting provider for the PTR record, which is at least as
important. But even following those best practices you can still have delivery
issues such as I have described above, leading to the unfortunate reality that
if you _really_ need a message to be delivered you're probably better off
sending it through Gmail.

~~~
mike-cardwell
I was suffering the same problem about 7 years ago on my own personal mail
server. I discovered that if I padded out the message headers with a load of
junk headers to push the message size up beyond about 50KB, then the otherwise
exact same message would arrive in the Inbox at Hotmail rather than being
blackholed.

[https://lists.exim.org/lurker/message/20070614.130838.42d1bd...](https://lists.exim.org/lurker/message/20070614.130838.42d1bd79.en.html)

I haven't done this for many years though as it was such a terrible hack. I do
wonder if it still works, or if the threshold has changed though.

~~~
danieltillett
I am going to try this out and see if it still works. If it does I will be
sacrificing a goat in your honour when the next full moon comes around :)

~~~
mike-cardwell
If it still works, please prod me via the contact details in my profile. Would
love to find out it still does :)

~~~
danieltillett
I will. One thing I can add is Microsoft hates the word PayPal - mention
paypay in your email and it gets killed, take out that one word and it goes
straight through.

------
wtbob
Please, _please_ do not release this sort of thing for Ubuntu; target Debian
instead. It's far easier to build Ubuntu support upon a Debian base than vice-
versa.

------
chrissnell
Be aware: Rackspace (and surely many other cloud server providers)
intentionally submit their IP blocks to DNS blacklists to dissuade spammers
from using their hosts to send/relay mail. I'm not sure if the other big
players do this but it wouldn't surprise me. Same goes for most consumer-class
cable modem ISPs.

~~~
mike-cardwell
Do you have any evidence for this? It seems unlikely to me...

~~~
mike_hearn
He's talking about something called the PBL (policy blocklist). The idea is it
contains IP ranges that aren't "supposed" to send mail, like consumer ADSL
ranges. People who want to send mail from home directly are supposed to do so
via their ISPs SMTP servers, which may be configured to relay but only from
IPs the ISP controls.

~~~
mhurron
There is no reason you can't set up your SMTP server to use your ISP, or other
SMTP service like gmail, as a smart relay.

Personally I have a VPS that I relay mail through.

~~~
0x0
One downside with ISP outbound relaying is SPF; if you want to use it you need
to figure out how mail exits the ISP and keep the SPF records up to date; and
even if you do, suddenly all the other customers of that ISP can fake mail
from you while passing SPF.

Another downside is you lose logs of and insight in the mailq and the
recipient smtp server responses.

------
mstrem
This is typical, I have just done the whole process myself a few weeks ago
from scratch (fresh CentOS install) and now this comes up.

However I do not regret it at all. I had Linux experience and it took me one
day and a half work to get it all working very nicely.

I am happy with my config: Postifx, Dovecot, RoundCube, SpamAssassin, ClamAV

Server supports unlimited domains and user accounts - SSL is required for all
connections, I only allow IMAP and I have configured it with two valid free
StartSSL certificates:

    
    
        One under mail.domain.com (for email clients)
        One under webmail.domain.com (for the webmail)
    

And both of course do not show warnings and green padlock is always nice.

The thing that scared me the most was outgoing emails being dropped - however
to date I have delivered fine to all main email provider - followed a few
simple rules:

    
    
        Ensure you have both SPF and TXT correct DNS records
        Ensure you have IPv6 configured properly (Google was  rejecting due to this) 
        Set your reverse DNS
        Set your machine hostname etc. in postfix

~~~
dav-
You seem to have learned a lot in the process. Can you point to any good
resources that helped you, specifically with security concerns and outgoing
mail reliability? I've already got a mailserver running with a similar setup
as you, but I haven't invested much effort into security or making sure mail
providers accept my emails (Gmail sometimes marks them as spam and AOL rejects
them all together).

------
_pius
This project is up for Knight Foundation funding. If you want to see it
funded, speak up here:
[https://www.newschallenge.org/challenge/2014/submissions/mai...](https://www.newschallenge.org/challenge/2014/submissions/mail-
in-a-box)

The guy behind it, Josh Tauberer, also wrote GovTrack, which was (and still
is) important for anyone who builds things with Federal government data.

------
codexon
Please put a strong rate limit on the dns server to prevent it from being used
for ddos attacks.

The last thing the internet needs is more clueless users with dns servers.

~~~
joshdata
Hi,

Mail-in-a-Box uses 'nsd', which is a non-recursive DNS server. I don't think
it can be used to relay DNS queries in a DDOS attack. But if you meant
something else, please let me know and/or file a ticket (I don't think I'll be
looking here for replies).

[https://github.com/JoshData/mailinabox/issues](https://github.com/JoshData/mailinabox/issues)

------
josho
Even better is to host your personal email on a server at your home. SMTP will
retry sending mail if your server goes down for a period, so uptime isn't
critical.

I've been doing this for several months with no issues, and it's pleasant
getting out of google's targeted profiles.

~~~
clarry
I wish more protocols were built with such robustness and decenteralization in
mind. If you try to host a web site at home and it gains some attention among
netizens, you might have a bad day. Not because your connection is dog slow,
but because all those people will complain about the way you host.

The internet is too darn centeralized, people can't just put services out
there.

~~~
josho
I often feel the same about how centralized the 'net has become. Two issues
that seem to make it worse:

1\. ISPs providing asymmetric service (and it's been getting worse, it used to
be 10:1 dl:ul, but when I upgraded speed last year it grew to 20:1).

2\. Why do we have a plethora of centralized services, when ideally a smart
firmware in your home router can handle pages like about.me or simple blogs.

------
wvh
Setting up your own mail server is a thorny problem. I've been setting up my
(family's) own mail server, and despite having lots of experience, it has
taken a month already. Even when encryption, authentication, spam and sieve
filtering, IMAP and webmail work, there's still no SMTP backup or alternative
server in case you accidentally end up being blacklisted somewhere. I'm not
sure investing money and effort is really worth it for just a few people's
mailboxes.

The whole process keeps getting more complicated as the internet grows more
hostile and end-user requirements increase.

This project sounds interesting and much necessary, but the real problem to me
is the economy of scale. If 10 Unix guys sit together and set up 2 servers for
20-30 personal mailboxes, the time and money might be worth it. Setting up all
those services, filtering and redundancy by yourself for just yourself – and
monitoring them continuously – is wasteful and painful, even if you like to
tinker with Unix systems.

There used to be a few geek-oriented ISPs around that offered SMTP/DNS backup,
spam filtering and similar services so you could off-load some effort, but I
guess they've fallen prey to low-margin virtual server business.

------
peterwaller
It does greylisting by default. Greylisting is awesome at cutting down spam,
but we recently noticed that it was preventing a major mail provider from
getting through, among others, because it seems that this major provider
doesn't use the same IP twice (at least sometimes).

If ignoring email coming from a major provider isn't an option (it isn't.),
then what are your options?

~~~
mhurron
You can whitelist from GMail while still greylisting from everyone else.
Simply query DNS for their SPF records and use that for the whitelist, Google
is good about keeping that up to date.

GMail sends from a farm, it's random where the exit ends up being.

I only have that problem from GMail, Hotmail and Yahoo don't seem to have this
problem.

------
robomartin
This is neat. I've been running my own email forever and appreciate the
difficulty involved.

I do have a question:

Does hosting your own email truly isolate you from government surveillance?

If a million people use this to host their own email on AWS, what's to keep
the NSA from ordering Amazon to tap into your data?

You almost have to host it on your own hardware in the garage in order to add
a significant layer of isolation. Even with that in place there are ways to
access your communications through your ISP. Harder, not impossible.

What's reality?

~~~
smacktoward
_> Does hosting your own email truly isolate you from government
surveillance?_

Not by itself, no; the government can still get to you by getting access to
the machine you host it on, or the switch the mail flows through on its way in
and out from the outside world. Hosting your own just reduces the number of
third parties the government can try to shake down to get access to your data.

~~~
robomartin
I guess my point is that unless you have the machine in your home nothing
really changes: They shake down the hosting provider and they are in.

In other words, what's the difference between buying email service from
GoDaddy and renting a VPS from them to host it yourself?

Not being critical at all, just trying to understand if there's something
fundamentally different about Mail in a Box that makes it more government-
secure than other approaches. The description seems to list privacy from
government snooping as the primary motivator. Is it truly effective at that?
If so, why and how?

As I understand things the very idea of being secure from snooping is an
illusion so long as your email is on a machine that you don't physically
control in your home or office. And, even then, the best you can do is encrypt
all traffic to make it more difficult --not impossible-- to capture your data.

Am I wrong?

~~~
marssaxman
What's wrong with installing Mail-in-a-Box on a machine that you physically
control in your home or office, then?

------
jamestomasino
Are you serious? I just spent all weekend building myself almost the exact
same setup from scratch. It was a royal pain. I wish I'd seen this a week ago.
Nice work.

------
sz4kerto
A Docker image would be brilliant.

~~~
fideloper
Sounds like its a little more than running the setup script, as you need to
choose a hostname and then setup Reverse DNS. That could make a docker
instance harder to create if there's customization needed.

The irony of Docker IMHO so far is that it almost makes installation that
needs configuring with user-specific values harder, unless they know how to
build docker instances themselves.

~~~
radq
Can't say I agree, for example Discourse's Docker setup is pretty easy and
installing a Discourse forum does involve specifying a number of variables.

[https://github.com/discourse/discourse_docker](https://github.com/discourse/discourse_docker)

------
illuminated
Although I do run at the moment my own email server with postfix, dovecot and
the rest, I'm thinking for some time to replace that with Kolab [0] - a
similar package of "everything email hosting related in a box" solution with
an extra advantage of supporting active sync which I find very valuable.

[0] [http://kolab.org/](http://kolab.org/)

------
pjc50
I used to run my own mail server. I stopped because of two reasons:

1) You have to be your own 24-hour support, or accept that you will lose or
bounce mail. You can be stuck with realising that it's failed while you're on
holiday.

2) You have to be your own antispam. I've had my own email domain for fifteen
years; it's on basically every spam list out there. Gmail is very good at
this. At the moment I rely on my hosting provider who filters out most but not
all of it.

3) You can have tedious issues delivering outbound mail, even if you're not
using SPF.

In my opinion the right level to run email is "small cooperative" (10-100
users), where cooperatives share software, experience and maybe even staff but
are administratively and physically separate.

------
agentultra
It would be nice if this could be ported to something like Ansible so that we
could add more host targets easily (ie: Fedora, CentOS, Arch, etc).

Great job. Mail configuration is pretty hard in the number of steps and amount
of configuration that needs to be done to get it right.

~~~
jgj
[https://github.com/al3x/sovereign](https://github.com/al3x/sovereign) is
linked to in the repo. sounds like what you're looking for.

------
xyzzy123
This is cool, but it doesn't actually help you with mass surveillance unless
the _other_ side of your conversation is doing that too. What point running
your own email server if your friend is on hotmail?

Also, even if you both run your own MTAs STARTTLS does not help much except to
reduce the scope of the passive-only surveillance dragnet, because the
configuration does not require signed certificates.

Finally, if both parties are technical enough to run their own email servers,
um, why not just use GPG?

From a technical point of view, I put this squarely in "fun, but a boondoggle"
territory.

~~~
mike-cardwell
> Also, even if you both run your own MTAs STARTTLS does not help much except
> to reduce the scope of the passive-only surveillance dragnet, because the
> configuration does not require signed certificates.

Exim and Postfix (and hopefully others to come) are both working on adding
DNSSEC and DANE support right now. Which means in the not too distant future
it will be possible to publish cryptographically signed data in the DNS that
your MX has TLS support, and what the fingerprints are.

> Finally, if both parties are technical enough to run their own email
> servers, um, why not just use GPG?

OpenPGP encrypted email leaks all sorts of information/meta data.

~~~
xyzzy123
The metadata problem is so hard that if you actually solve it you are not
using email anymore. This is why I'm saying it's a boondoggle. Even in the
best case where my mother and I are running our own STARTTLS secured MTAs, oh
wait, timing and message size.

GPG gives you actual privacy but minimal resistance against traffic analysis.
All this other stuff gives you NO end to end privacy, AND minimal resistance
against traffic analysis.

Messaging is a weird special case and GPG is behind the times (because people
are coming to realise that repudiability and FS are important), but it's still
a heck of a lot better than even _working_ TLS slapped on SMTP (which doesn't
even currently exist).

~~~
mike-cardwell
The security of email will never be perfect; that is obvious. But there is
still legitimate and real value gained from making small incremental
improvements to it. If you can hide the To/From/Subject of a message from a
passive or even active MITM (because of DANE), then that's a good thing.

------
brianshaler
> It is a mail server in a box aimed to be deployed securely into any cloud
> infrastructure. It provides no user interface to send or check one’s mail
> but implements all of the underlying protocols that other applications (mail
> clients), such as Google K-9 for mobile devices, Mailpile, and Mozilla
> Thunderbird, can interoperate with.

The combination of Mail-in-a-box and Mailpile seems pretty exciting.

Since running your own mail server is opening a new vector for attack, I'd be
curious how Mail-in-a-box is going to handle security updates.

~~~
joshdata
Don't know yet, but if you have thoughts please open an issue!

[https://github.com/JoshData/mailinabox/issues](https://github.com/JoshData/mailinabox/issues)

------
bananas
Rather interesting. I still like building the whole thing with postfix +
dovecot by hand myself. Perhaps I'm wrong in the head.

One thing to watch out for: Yahoo is a complete bastard to deliver to. One day
they'll decide to block you (if they don't immediately) and the only way out
is to fill in a web form which either allows you 6 months of delivery if
you're lucky or blocks you with no hope of resolution for 6 months.

Total assholes they are.

------
542458
Oh wow. This sounds awesome! About a year and a half ago I tried to set up my
own mail server on a cheap VPS, but could not get it to work for the life of
me. It's sort of a stressful thing to set up as well - failure is both high-
consequence and invisible, and I'm never quite sure if what I'm doing is
secure or correct. Count me as a fan of the idea - I'll try setting this up
later!

------
fredsted
If you need email, but also web/ftp/git/mysql/dns, try Virtualmin -
[http://www.webmin.com/virtualmin.html](http://www.webmin.com/virtualmin.html)
\- you can easily run this on a $5 digital ocean server.

You basically follow a web installation wizard, create a domain and then you
can use Roundcube (included) or any IMAP client. Couldn't be easier.

~~~
creeble
A +1 for Virtualmin. Does all that th OP's server does (I think) plus can do
it for multiple domains and has a simple web ui. Free, and their support forum
is helpful.

------
coherentpony
Why has nobody just made a Docker image that anyone can download and use for
free? Or even just a Dockerfile. That way it's a text file and people can
contribute and improve the setup.

Just my two cents.

Edit: Looks like my idea is far from
original<[https://news.ycombinator.com/item?id=7634677](https://news.ycombinator.com/item?id=7634677)
>

~~~
nfoz
> Why has nobody just made a Docker image that anyone can download and use for
> free?

Because you haven't done it yet. :)

~~~
coherentpony
Touche :)

Hours in the day, unfortunately.

------
Oculus
After toying with the idea for a while, I've settled on using a service like
Mailgun or Sendgrid is the best decision when it comes to setting up email.
Otherwise, it's just too much work for little in gain (as far as I'm aware).
I'm curious to hear what others think.

~~~
dangayle
I love Mailgun and use it for every project that needs to send out email, but
you have to admit it doesn't address the OP's purpose: To decentralize and own
your own email

~~~
jradd
CTRL + F `mailgun` yayy...oh..nm

------
denibertovic
The part that bothers me is that is's all shell scripts... Why not use a
modern provisioning tool now-a-days?

This looks way more interesting to me:
[https://github.com/al3x/sovereign](https://github.com/al3x/sovereign)

------
JimmaDaRustla
I'm going to have to try this out - I tried setting up a mail server once and
failed because it was too difficult. Now I'm paying $30/year for service from
NameCheap.

~~~
johnpowell
I set up my own by hand and it worked great for about a year. Then emails
started to not get sent and incoming ones were vanishing too. Then it all went
to hell and just completely stopped working.

I gave up and went to fastmail. Setting it all up wasn't a problem.
Maintaining it was a nightmare.

------
rdl
Huge advantage to hosting your own mail due to third party doctrine, so this
is awesome, even if it is hard. Mail might be the one service I would be most
reluctant to outsource.

------
dale-cooper
Does anyone have a cheap VPS provider to recommend for this? I usually use
digital ocean but from what i hear, most of their ip space is blocked
everywhere due to spam issues.

------
_asciiker_
I for one, am still sticking with qMail, qMail-toaster to be precise. Over the
years it is proven to be very reliable even when it takes beatings like no
other.

------
hardwaresofton
After being conquered by the difficulty of sendmail configuration, this is a
breath of fresh air.

+1 to what codexon said about the rate limit, but this seems absolutely
amazing

------
GigabyteCoin
Mail server setup is certainly a common problem that needs fixing.

Good luck with your project. I know I will use it.

------
ausjke
what about Citadel, the UI looks stone aged but it works as the in-a-box thing
just fine.

------
nemoniac
File-system encryption with something like encfs would be a great addition.

------
snambi
Thanks for doing this. setting up mail is so hard, most people give up.

------
Torn
Noone here backed Mailpile on kickstarter then?

[https://www.mailpile.is/](https://www.mailpile.is/)

------
er0k
how does this differ from iRedMail?

------
miles_matthias
This is nice if your main goal is to take privacy back. For me, Gmail is not
only free, but provides great mobile apps and email innovation like filtering,
search, and a tabbed inbox.

In short, Gmail provides more value to me than being a simple SMTP server, so
I'm willing to pay with my advertising eyes.

~~~
zAy0LfpBZLC8mAC
Unfortunately, you are also paying with my privacy (assuming I were to send
you an email).

