Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: WildDuck – Self-hosted modern email server (wildduck.email)
222 points by andris9 8 months ago | hide | past | web | favorite | 81 comments

I would love to set this up but I've seen too many posts on the front page of HN about how gmail is blocking their incoming mail and they are powerless to do anything about it.

If your hosting email on a residential ISP, your gonna have a rough time. Comparatively, hosting on any of the notable VPS providers (DO, Linode, OVH, etc) has had fine deliverability in my experience, though emails may occasionally get flagged as spam.

Beyond that, the only insidious behavior I've seen has been Outlook.com hosted addresses accepting email, but then not delivering the email to the Inbox or Spam. Mysteriously, this issue disappears whenever I start going up the IT chain on the reciever's side. If the MTA is going to drop email, it should reject it rather than accept it (as otherwise the Outlook MTAs are falsely accepting mail that they will not deliver).

IIRC though DO tries to make sure a released IP is returned back to you if you re-provision quickly, you run the risk of losing your well groomed IP for outbound mail if there is an issue with your droplet.

Additionally, IIRC they forbid outbound SMTP on their Floating IP system. (I can't find an official doc on this matter). Either way, you can't get a PTR for your floating IP, which can harm deliverability[1]

For that reason, you may want to consider AWS & an Elastic IP that you can own, groom, and move around [2]

[1] https://ideas.digitalocean.com/ideas/DO-I-1023

[2] https://aws.amazon.com/premiumsupport/knowledge-center/ec2-p...

Furthermore, DigitalOcean silently drops all SMTP traffic on IPv6, which can be really confusing since you'll simply timeout and it isn't mentioned in the documentation. They do that because they implemented IPv6 against all standard industry practices (which were helpfully gathered in a RFC) and within hours of launching IPv6 the few /64s from which they were allocating customers /124s to landed on blacklists.

It's been 5 years since and they still have no plans to do IPv6 properly and allocate /64s [0].

This is the main reason I switched to Linode, who will happily allocate you a IPv6 range (/116, /64, /56) that can be rerouted between VPSes with a simple ticket (takes <1 hour).

[0]: https://ideas.digitalocean.com/ideas/DO-I-1366


I wish I'd seen this literally yesterday.

Don't destroy your droplet then? Digital Ocean supports reinstalling VMs in place without destroying the VM. Floating IPs for a mailserver (esp. outbound) sounds a bit silly, mail infrastructure is more fault tolerant than other systems, to the point that a minute or two of downtime is far from the end of the world.

Receiving email is no issue on residential ISP.only issue is sending, And to get around that you simply use an external SMTP server and your problem is mostly solved.

Yeah, that is one workaround that is reasonable to use. I'm just a bit weird and want to do delivery to other servers without an intermediary :P

It's just something you have to do anyway since more ISP prevent sending directly from home connections. That means it will cost some money, but there are also SMTP services with free tiers for several thousands of emails per month.

ISP mail relays generally work, not sure why you'd pay to add another MiTM besides the one that already exists (your ISP).

> hosting on any of the notable VPS providers (DO, Linode, OVH, etc) has had fine deliverability in my experience, though emails may occasionally get flagged as spam.

I can share a polar opposite experience: Yahoo and Outlook would categorically reject anything DO at the SMTP level with a 502. Luckily enough, at least they were explicit about it. But no leeway, not even accepting the envelope body.

So YMMV, but it will almost certainly be poor :)

I don't run my mailserver on DO (I know a few people who do tho), but I've had few issues operating one on a no name host and another on OVH. If that was a persistent issue, it might be advisable to move to a host that isn't blacklisted.

You would use SMTP delivery service instead of moving around wishing for a lucky host / IP.

My mail from DO doesn't get rejected, but ends up 100% of the time in spam.

Why not use SES or SendGrid, both are so cheap for small volume.

Sendgrid is usually spam or redundant notifications, depending on host they might not make it through without manual whitelisting. Additionally, using a totally unrelated SMTP rela to deliver mail defeats much of the point of a self-hosted mailserver, you'd be better off using your local network's SMTP relay.

It definitely does not defeat the purpose.

You control the way you authenticate users, how to run spam filtering, etc.

I run my own on Linode with postfix/dovecot/rspamd linked to LDAP for auth that gets routed through SES (probably some cents per month) and it's working good.

These days it's not easy to circumvent being flagged as spam running your own node, because you can look like any spammer until you prove yourself you're not one which takes time and usually not the best idea and you'd rather use an external relay who will provide you with a better reputation from day one.

If you stick with your own delivery route and friends tell you, your mail isn't arriving as they get flagged as spam, that's your fault.

DO is a major source of spam. Hetzner is closely behind. OVH is third.

We are now flagging all mail coming from their IP space as junk, but without blocking it, because there are (very) occasional legit senders there.

I've seen many people complain about it, but never had a problem myself.

I'm wondering if they are actually spamming gmail users, or maybe if gmail and similar large providers are biased against new domain registrations, and people starting new mail servers often have new domains.

Or maybe they don't do the basics like working DKIM, SPF, TLS, etc and they get penalized for it.

As a partial workaround Gmail will forward mail for you from you@yourdoma.in if you set it up properly on your account. So you could use Gmail for your outgoing mail and use something like this for incoming. If you wanted to get really clever maybe you could even use Gmail as outgoing only when sending to @gmail.com addresses.

> While WildDuck is more secure by definition than most alternatives > ... > You can run WildDuck on any system that supports Node.js,

Node.js and more secure by definition? That sounds _very_ strange to me...

For secure by definition I meant:

* written in memory safe language

* no special permissions needed (besides binding to privileged ports at start but there are workarounds) so no need to be able to chown or spawn workers under different user id's

* no file access for the running daemon, once the daemon starts then it does not read nor write to the local file system

* no spawning shell commands

each of these things is a target for a different attack vector

I get that this is a meme, but v8 is one of the most battle-hardened runtimes out there, plus you don't manage your own memory or concurrency primitives, which are the sources of most security issues.

v8 may be badass. but the quality of many of the libraries in the node ecosystem is .... questionable at best. Lots of newbs with no clues about security throwing out packages that people end up with in their dependency trees and never stop to ponder the security / quality of.

Maybe because "Microsoft: 70 percent of all security bugs are memory safety issues"

I guess they mean “Secure by Design” but it doesn’t inspire confidence indeed..

Do large companies provide data security with very well-known languages?

I feel like self hosted emails is gonna be the next big thing, kinda like the pi-hole trend. This is awesome

Been running a postfix/dovecot/spamassassin stack for years. Absolutely love it. Want to wrote up a joke server side filter that bounces emails that are too snarky by sentiment analysis? That's a simple dovecot-pigeonhole-sieve filter. Lots of fun for all.

I just googled pi-hole and now I'm converted. Thank you!

What's old is new again... Back in the 90's, it was very common to run your own email server. I've been hosting my own since 1996.

Always interesting to see new MTAs! I'm not convinced that storing mails (including attachments) in MongoDB is a good solution. What's wrong with the filesystem? It's designed for that purpose. You can (and should) of course store metadata in a DB to make searching easier, but when it comes to the actual emails I think the filesystem is the best place. One less thing to break. Easier to backup, mangle etc.

It's significantly easier to make an external data store resilient than to try to do that with a local filesystem. Backups, clustering, HA are all easier when there's a strong separation. In addition things like querying and lookups will be faster if you're utilizing a database

A filesystem doesn't have to be local. There are plenty of filesystems to choose from with different properties.

Using maildir (the current standard for mail storage) with any kind of remote file system is not the best environment to have. Maildir stores every email into a separate file and then uses rename() for every flag change. Marking 1000 messages as "Seen/Unseen" or adding a flag or moving to Trash means 1000 rename() calls in the file system. Multiply that with thousands of users also logged in to the system and doing their own things - you get many millions of inodes and rename()'s flying around all over. Good luck keeping a consistent copy or not crashing the mail server software as it waits for some queued rename() call over network FS to finish (and expects it to take nanoseconds as this is how long it takes in a local FS) while blocking everything else.

I don't see how storing the mails in a database will improve the situation. You will still have to update 1000s of records when you mark 1000 messages as seen/unseen or move them into the trash. And the DB will then write the changes to a filesystem.

One of Maildir's design goals was to make it safer to use on networked file systems such as NFS (compared to mbox).

Well, scaling a filesystem does not really sound easy to me. Scaling a dbms is rather common, I guess.

I rather question, why do they store their binaries in mongodb? imho, s3 would be the perfect fit for that requirement.

Back when I was hosting my emails myself, my main complaint was that it required plugging many pieces together, in a traditional Unix fashion. This was before configuration management was really a thing, so it resulted in a Frankenstein setup that worked but was afraid to touch.

If there was a more modern all-in-one-binary solution handling MTA/DKIM/IMAP/webmail with sane defaults I would maybe go back to self hosting.

mailu.io is an open source alternative to poste.io and bundles a classing email stack in an easy to use package. But it’s still quite complex under the hood. There is also an upcoming project called maddy( https://github.com/emersion/maddy) that tries to package everything in a single binary, written in Go. I’m excited about about this and hope that the good folks that are developing it will take it to a usable state.

> all-in-one-binary solution handling MTA/DKIM/IMAP/webmail

Sounds like a very wrong idea there. One mistake in part of it will crash or get breached in the entire stack. You know why postfix runs many small binaries.

See poste.io, it solves exactly this problem. (I've created poste because I administer multiple smaller mailserevers and thus scratching my own itch)

This is really impressive. Perhaps do a Show HN for it?

I use https://mailinabox.email/ it does pretty much everything for you very easy just host on linode.com.

These guys are selling a box, too: https://thehelm.com

Wish I could learn more about their box. But when I go to the web site I get a giant "Get updates and news from Helm" pop-up that blocks all other content except a sliver of an animation on one side of the screen.

How do I know if I want news and updates from Helm if I can't even find out what Helm does?

Is JMAP support planned?

No, at least not any time soon. JMAP is a beast that should not have been invented. It tries to convert TCP/IMAP (already a bad protocol by todays standards) into HTTP/JSON, just using a different transport while keeping the spirit.

Instead WildDuck has its own simple REST API to build email clients against (https://api.wildduck.email/). It is not standard by any means and there are no wide client support but it is really easy to integrate with other projects as it is basically just a wrapper against database access, just like you would access blog posts or whatever via an API.

Just to add to this point, I wanted to implement JMAP after only half completing an IMAP server implementation. Unfortunately, its just as bad only "this time with HTTP and Json". The batching advantages of JMAP could be generically done by using HTTP 2 and/or GraphQL.

I shall reply to your comment and your parent comment in one go here:

I work on FastMail’s web interface, which is now based on JMAP. Here are my opinions on the matter.

What papaf says is flatly not true. What andris9 says is generally unreasonable.

At its core, the API part of JMAP is an RPC framework, with defined semantics for object synchronisation and querying. The most important part of it is those semantics, and they are necessarily more complex than what web developers are generally used to. Simple traditional-REST APIs or the likes of GraphQL lack those object synchronisation semantics altogether, and are wildly inferior for practical email clients. There is just no comparison at all.

There is more to JMAP than just the API calls, though. What I think is the most obviously important part is that it also defines a push mechanism, including the ability for email clients to be notified of changes; this is tied into the object synchronisation semantics with the ability to ask questions like “what changed between these two states?”—so that your email client (be it on the web or not) can be told “something changed in the emails” and efficiently fetch the delta and apply it to the user interface and any persistent storage you may use. This is something that the shown WildDuck API does not appear to support at all, and is in fact something entirely unsupportable on such a style of API. These things are hard to do properly, in a way that will give any semblance of performance or efficiency.

(It looks like WildDuck’s web interface does have some way arranged so that changes get pushed to the interface, but that doesn’t appear to be part of its public API, and I can’t imagine that it’s actually a good, flexible and efficient solution. I haven’t looked into how it is implemented at all, however. I’m open to discussion of what I’m confident will be its shortcomings, if you don’t like the aspersions I am casting.)

Now to return to papaf’s remark: JMAP’s method batching is not achievable with HTTP/2 or GraphQL; the key difference is backreferences, whereby you can establish data dependencies between method calls, so that you pass part of one method call as an argument to subsequent method calls. To pick a common example of something that the FastMail web interface does, this allows you to express in one HTTP request the following: “find the first ten emails in Inbox, collapsing threads (so that you get one message from each thread only); then, given those, get the threads that they are in; then, given those threads, get the basic details of the emails that are contained in those threads.” In the absence of backreferences, you would need probably three round trips, or a special case in the API that limits flexibility.

I will add that JMAP can be usefully combined with HTTP/2 so that you can issue multiple batches of requests simultaneously and use the results as they return.

Please read through all the content on the front page of https://jmap.io/. It provides good justification for why quite a few things are how they are.

Remember this most important fact: JMAP is an object synchronisation protocol.

Would it be best to wait to write a JMAP server until the spec is finalized? Or should I consider the current one mostly final?

I’ve had one look at it and while the protocol looks really good, it also looks like I’d rather not implement it more than once ;)

The spec is currently in the editor’s queue for publication. I believe that means that no further changes are anticipated.

This is cool. The world could definitely use more email server implementations, even if JS isn't my cup of tea.

Have you considered supplementing Mongo with block storage like S3, though? I've found it to be a pretty much perfect match for storing immutable mail messages.

For messages no, for attachments yes. WildDuck parses stored messages and separates attachments from rest of message data. Deduplicated attachments are then stored to MongoDB as GridFS files but this could be just as well S3 in the future. Message data needs to be in Mongo because of indexing and speed, attachments do not have such requirements and could be stored anywhere.


Russian inside me chuckled.


For the non-multilingual (like myself).

You didn't translate the TLD, which is .org, not .opr.

You learn something new every day! Google translate didn't help me there :)

Made in Estonia as I see. Congrats ! Very cool project, I really like the GUI part.

Nice! I recently migrated my home server from Ubuntu to freebsd and was lazy to migrate my postfix/dovecot/sieve/all-the-other-parts and eventually kept it running in a VM instead.. This gives me the idea to try and migrate to Wildduck as it seems easier to deploy.

Two questions that I have:

1. Is it possible to plug it into a mysql db instead of mongo?Self hosted apps are heavily relying on SQL db, so it feels more fluent to have a sql support

2. Is there a way to migrate existing postfix/dovecot mail server to this if I end up liking it?

1. no. WildDuck is built to use MongoDB sharding to be able to easily scale both vertically and horizontally without single point of failures - each component of WildDuck (DB, IMAP, POP3, MX, SMTP) can be replicated. SQL does not fit too well into this model. If you want to use SQL then you could look into DBMail

2. migrating is possible only syncing via IMAP. There is a maildir importer but it is not open sourced (yet)

How do you handle spam filtering for self hosted servers? I know there are black lists for SMTP servers, but I mean the content based filters that are updated based on recent spam signatures?

I run a few servers that use Virtualmin + Postfix. I'm militant about outbound mail, and I make sure my accounts don't get compromised. If my mail queue reaches a certain size, I get notified and I go mop up the mess. Most recently it was an unprotected form that wasn't linked to, but existed.

As for inbound spam, I can use RBL's but the most effective thing has been to block email servers that have rDNS that is either mismatched or nonexistent. I also host at a provider that isn't known for having their IP's on the "forever banned" lists.

> As for inbound spam, I can use RBL's but the most effective thing has been to block email servers that have rDNS that is either mismatched or nonexistent.

This is also a recipe for never receiving mail from the SMB you do business with. (The same is true for enforcing SPF hard fails, unfortunately.)

If you really want to be smart about this, use an SMTP soft error (greylisting) or an SMTP greeting pause. This effectively singles out spam bots, while leaving your regular mail traffic unaffected.

Speaking as someone who runs a commercial antispam SAAS, content filtering is increasingly irrelevant, and isn't nearly as effective as reputation filtering or bayesian classification.

Back in the day I ran an anti-spam service, users would point their MX to my server(s), I'd do the filtering and pass on valid mails to their real hosts.

It was a great learning experience, and I got to see a lot of volume. Sadly I never had enough customers to make the jump to the big-leagues, but it was fun.

(I'd do proper SMTP-time rejection, but also archive rejected mails in a database for 30 days. Letting you browse / search through the rejected stuff - just in case I made a mistake, and also to see how good I was doing.)

Even then content-scanning was not the best, but I used a lot of heuristics and I was quite proud of some of them. For example I "defeated" fast-flux hosts, via heuristics to lookup the number of addresses for a domain. Even large domains such as gmail only have 1-4 address-records, though of course their anycast and multiplexed. If you got a mail from a domain with 7+ MX records it was 99% spam.)

At one point I documented damn near everything, after I'd shut down, but it seems I left the domains I used expire. I should see if I can dig out copies.

(I used qpsmtpd as my core, and wrote a ton of glue code and custom plugins. Later the qpsmtpd-proxy was ported to javascript as Haraka - but I didn't covert my code even as I ported some plugins back.)

Bayesian classification is what I mean by spam 'signatures'. You run your own classifier?

A million settings, but ASSP in front of postfix has worked well for me for quite a while.

rspamd works pretty well for me.

Wow I was looking for something like that since ages. Best part, it's open source an made with Node.js/Redis, a stack I deeply love :)

"it does not touch the filesystem"

How does this work exactly? Mongo would have to touch a FS at the end of the day, no?

Sure, MongoDB does manage files but for the email app this is behind an abstraction and possibly in a different machine. I was more referring to maildir/mbox where the email server has to write and read files while juggling with user/group permissions, invoking shell and such. WildDuck just writes and reads stuff from a database like a regular web server software.

Got it. I suspected that's where you were going with that but just wanted to clarify. Thanks.

See also https://forwardemail.net. Thank you Andris for making this possible.

Thanks, this is so timely. I was looking for something like this for weeks. All I want is IMAP server which can be managed via IMAP client api.

Cool project. Been looking for something like this for ages. But still cant figure out from the docs how to send email.

What's the difference with mailcow (a fully dockerized email solution) ?

Mailcow packages Dovecot, Postfix etc. in an easy to use solution. WildDuck has its own and quite different implementations for most of these things.

I would love to try this.

Is there a way to configure catch-all addresses for domains?

Does it come with official Docker image? :)

I don’t maintain a Docker image myself but the best version you could consider semi-official would be this https://github.com/astzweig/docker-wildduck

I’m using this image. Can vouch for greatness.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact