
Show HN: WildDuck – Self-hosted modern email server - andris9
https://wildduck.email/
======
55555
I would love to set this up but I've seen too many posts on the front page of
HN about how gmail is blocking their incoming mail and they are powerless to
do anything about it.

~~~
StudentStuff
If your hosting email on a residential ISP, your gonna have a rough time.
Comparatively, hosting on any of the notable VPS providers (DO, Linode, OVH,
etc) has had fine deliverability in my experience, though emails may
occasionally get flagged as spam.

Beyond that, the only insidious behavior I've seen has been Outlook.com hosted
addresses accepting email, but then not delivering the email to the Inbox or
Spam. Mysteriously, this issue disappears whenever I start going up the IT
chain on the reciever's side. If the MTA is going to drop email, it should
reject it rather than accept it (as otherwise the Outlook MTAs are falsely
accepting mail that they will not deliver).

~~~
jasonjayr
IIRC though DO tries to make sure a released IP is returned back to you if you
re-provision quickly, you run the risk of losing your well groomed IP for
outbound mail if there is an issue with your droplet.

Additionally, IIRC they forbid outbound SMTP on their Floating IP system. (I
can't find an official doc on this matter). Either way, you can't get a PTR
for your floating IP, which can harm deliverability[1]

For that reason, you may want to consider AWS & an Elastic IP that you can
own, groom, and move around [2]

[1]
[https://ideas.digitalocean.com/ideas/DO-I-1023](https://ideas.digitalocean.com/ideas/DO-I-1023)

[2] [https://aws.amazon.com/premiumsupport/knowledge-
center/ec2-p...](https://aws.amazon.com/premiumsupport/knowledge-
center/ec2-port-25-throttle/)

~~~
tambre
Furthermore, DigitalOcean silently drops all SMTP traffic on IPv6, which can
be really confusing since you'll simply timeout and it isn't mentioned in the
documentation. They do that because they implemented IPv6 against all standard
industry practices (which were helpfully gathered in a RFC) and within hours
of launching IPv6 the few /64s from which they were allocating customers /124s
to landed on blacklists.

It's been 5 years since and they still have no plans to do IPv6 properly and
allocate /64s [0].

This is the main reason I switched to Linode, who will happily allocate you a
IPv6 range (/116, /64, /56) that can be rerouted between VPSes with a simple
ticket (takes <1 hour).

[0]:
[https://ideas.digitalocean.com/ideas/DO-I-1366](https://ideas.digitalocean.com/ideas/DO-I-1366)

~~~
richjdsmith
Damn....

I wish I'd seen this literally yesterday.

------
markvdb
> While WildDuck is more secure by definition than most alternatives > ... >
> You can run WildDuck on any system that supports Node.js,

Node.js and more secure by definition? That sounds _very_ strange to me...

~~~
Tankenstein
I get that this is a meme, but v8 is one of the most battle-hardened runtimes
out there, plus you don't manage your own memory or concurrency primitives,
which are the sources of most security issues.

~~~
masukomi
v8 may be badass. but the quality of many of the libraries in the node
ecosystem is .... questionable at best. Lots of newbs with no clues about
security throwing out packages that people end up with in their dependency
trees and never stop to ponder the security / quality of.

------
dead_mall
I feel like self hosted emails is gonna be the next big thing, kinda like the
pi-hole trend. This is awesome

~~~
acidburnNSA
Been running a postfix/dovecot/spamassassin stack for years. Absolutely love
it. Want to wrote up a joke server side filter that bounces emails that are
too snarky by sentiment analysis? That's a simple dovecot-pigeonhole-sieve
filter. Lots of fun for all.

------
Tepix
Always interesting to see new MTAs! I'm not convinced that storing mails
(including attachments) in MongoDB is a good solution. What's wrong with the
filesystem? It's designed for that purpose. You can (and should) of course
store metadata in a DB to make searching easier, but when it comes to the
actual emails I think the filesystem is the best place. One less thing to
break. Easier to backup, mangle etc.

~~~
nijave
It's significantly easier to make an external data store resilient than to try
to do that with a local filesystem. Backups, clustering, HA are all easier
when there's a strong separation. In addition things like querying and lookups
will be faster if you're utilizing a database

~~~
Tepix
A filesystem doesn't have to be local. There are plenty of filesystems to
choose from with different properties.

~~~
andris9
Using maildir (the current standard for mail storage) with any kind of remote
file system is not the best environment to have. Maildir stores every email
into a separate file and then uses rename() for every flag change. Marking
1000 messages as "Seen/Unseen" or adding a flag or moving to Trash means 1000
rename() calls in the file system. Multiply that with thousands of users also
logged in to the system and doing their own things - you get many millions of
inodes and rename()'s flying around all over. Good luck keeping a consistent
copy or not crashing the mail server software as it waits for some queued
rename() call over network FS to finish (and expects it to take nanoseconds as
this is how long it takes in a local FS) while blocking everything else.

~~~
Tepix
I don't see how storing the mails in a database will improve the situation.
You will still have to update 1000s of records when you mark 1000 messages as
seen/unseen or move them into the trash. And the DB will then write the
changes to a filesystem.

One of Maildir's design goals was to make it safer to use on networked file
systems such as NFS (compared to mbox).

------
nicolaslem
Back when I was hosting my emails myself, my main complaint was that it
required plugging many pieces together, in a traditional Unix fashion. This
was before configuration management was really a thing, so it resulted in a
Frankenstein setup that worked but was afraid to touch.

If there was a more modern all-in-one-binary solution handling
MTA/DKIM/IMAP/webmail with sane defaults I would maybe go back to self
hosting.

~~~
efesak
See poste.io, it solves exactly this problem. (I've created poste because I
administer multiple smaller mailserevers and thus scratching my own itch)

~~~
eps
This is really impressive. Perhaps do a Show HN for it?

------
toomuchtodo
Is JMAP support planned?

~~~
andris9
No, at least not any time soon. JMAP is a beast that should not have been
invented. It tries to convert TCP/IMAP (already a bad protocol by todays
standards) into HTTP/JSON, just using a different transport while keeping the
spirit.

Instead WildDuck has its own simple REST API to build email clients against
([https://api.wildduck.email/](https://api.wildduck.email/)). It is not
standard by any means and there are no wide client support but it is really
easy to integrate with other projects as it is basically just a wrapper
against database access, just like you would access blog posts or whatever via
an API.

~~~
papaf
Just to add to this point, I wanted to implement JMAP after only half
completing an IMAP server implementation. Unfortunately, its just as bad only
"this time with HTTP and Json". The batching advantages of JMAP could be
generically done by using HTTP 2 and/or GraphQL.

~~~
chrismorgan
I shall reply to your comment and your parent comment in one go here:

I work on FastMail’s web interface, which is now based on JMAP. Here are my
opinions on the matter.

What papaf says is flatly not true. What andris9 says is generally
unreasonable.

At its core, the API part of JMAP is an RPC framework, with defined semantics
for object synchronisation and querying. The most important part of it is
those semantics, and they are necessarily more complex than what web
developers are generally used to. Simple traditional-REST APIs or the likes of
GraphQL lack those object synchronisation semantics altogether, and are
_wildly inferior_ for practical email clients. There is just no comparison at
all.

There is more to JMAP than just the API calls, though. What I think is the
most obviously important part is that it also defines a push mechanism,
including the ability for email clients to be notified of changes; this is
tied into the object synchronisation semantics with the ability to ask
questions like “what changed between these two states?”—so that your email
client (be it on the web or not) can be told “something changed in the emails”
and efficiently fetch the delta and apply it to the user interface and any
persistent storage you may use. This is something that the shown WildDuck API
does not appear to support at all, and is in fact something entirely
unsupportable on such a style of API. _These things are hard to do properly,_
in a way that will give any semblance of performance or efficiency.

(It looks like WildDuck’s web interface does have some way arranged so that
changes get pushed to the interface, but that doesn’t appear to be part of its
public API, and I can’t imagine that it’s actually a good, flexible and
efficient solution. I haven’t looked into how it is implemented at all,
however. I’m open to discussion of what I’m confident will be its
shortcomings, if you don’t like the aspersions I am casting.)

Now to return to papaf’s remark: JMAP’s method batching is _not_ achievable
with HTTP/2 or GraphQL; the key difference is backreferences, whereby you can
establish data dependencies between method calls, so that you pass part of one
method call as an argument to subsequent method calls. To pick a common
example of something that the FastMail web interface does, this allows you to
express in one HTTP request the following: “find the first ten emails in
Inbox, collapsing threads (so that you get one message from each thread only);
then, given those, get the threads that they are in; then, given those
threads, get the basic details of the emails that are contained in those
threads.” In the absence of backreferences, you would need probably three
round trips, or a special case in the API that limits flexibility.

I will add that JMAP can be usefully combined with HTTP/2 so that you can
issue multiple batches of requests simultaneously and use the results as they
return.

Please read through all the content on the front page of
[https://jmap.io/](https://jmap.io/). It provides good justification for why
quite a few things are how they are.

Remember this most important fact: JMAP is an object synchronisation protocol.

~~~
Aeolun
Would it be best to wait to write a JMAP server until the spec is finalized?
Or should I consider the current one mostly final?

I’ve had one look at it and while the protocol looks really good, it also
looks like I’d rather not implement it more than once ;)

~~~
chrismorgan
The spec is currently in the editor’s queue for publication. I believe that
means that no further changes are anticipated.

------
Felz
This is cool. The world could definitely use more email server
implementations, even if JS isn't my cup of tea.

Have you considered supplementing Mongo with block storage like S3, though?
I've found it to be a pretty much perfect match for storing immutable mail
messages.

~~~
andris9
For messages no, for attachments yes. WildDuck parses stored messages and
separates attachments from rest of message data. Deduplicated attachments are
then stored to MongoDB as GridFS files but this could be just as well S3 in
the future. Message data needs to be in Mongo because of indexing and speed,
attachments do not have such requirements and could be stored anywhere.

------
konart
>андрис@уайлддак.орг

Russian inside me chuckled.

~~~
elken
Andris@wildduck.opr

For the non-multilingual (like myself).

~~~
zingmars
You didn't translate the TLD, which is .org, not .opr.

~~~
elken
You learn something new every day! Google translate didn't help me there :)

------
scblzn
Made in Estonia as I see. Congrats ! Very cool project, I really like the GUI
part.

------
mgamache
How do you handle spam filtering for self hosted servers? I know there are
black lists for SMTP servers, but I mean the content based filters that are
updated based on recent spam signatures?

~~~
mark242
Speaking as someone who runs a commercial antispam SAAS, content filtering is
increasingly irrelevant, and isn't nearly as effective as reputation filtering
or bayesian classification.

~~~
stevekemp
Back in the day I ran an anti-spam service, users would point their MX to my
server(s), I'd do the filtering and pass on valid mails to their real hosts.

It was a great learning experience, and I got to see a lot of volume. Sadly I
never had enough customers to make the jump to the big-leagues, but it was
fun.

(I'd do proper SMTP-time rejection, but also archive rejected mails in a
database for 30 days. Letting you browse / search through the rejected stuff -
just in case I made a mistake, and also to see how good I was doing.)

Even then content-scanning was not the best, but I used a lot of heuristics
and I was quite proud of some of them. For example I "defeated" fast-flux
hosts, via heuristics to lookup the number of addresses for a domain. Even
large domains such as gmail only have 1-4 address-records, though of course
their anycast and multiplexed. If you got a mail from a domain with 7+ MX
records it was 99% spam.)

At one point I documented damn near everything, after I'd shut down, but it
seems I left the domains I used expire. I should see if I can dig out copies.

(I used qpsmtpd as my core, and wrote a ton of glue code and custom plugins.
Later the qpsmtpd-proxy was ported to javascript as Haraka - but I didn't
covert my code even as I ported some plugins back.)

------
SillyPosition
Nice! I recently migrated my home server from Ubuntu to freebsd and was lazy
to migrate my postfix/dovecot/sieve/all-the-other-parts and eventually kept it
running in a VM instead.. This gives me the idea to try and migrate to
Wildduck as it seems easier to deploy.

Two questions that I have:

1\. Is it possible to plug it into a mysql db instead of mongo?Self hosted
apps are heavily relying on SQL db, so it feels more fluent to have a sql
support

2\. Is there a way to migrate existing postfix/dovecot mail server to this if
I end up liking it?

~~~
andris9
1\. no. WildDuck is built to use MongoDB sharding to be able to easily scale
both vertically and horizontally without single point of failures - each
component of WildDuck (DB, IMAP, POP3, MX, SMTP) can be replicated. SQL does
not fit too well into this model. If you want to use SQL then you could look
into DBMail

2\. migrating is possible only syncing via IMAP. There is a maildir importer
but it is not open sourced (yet)

------
niftylettuce
See also [https://forwardemail.net](https://forwardemail.net). Thank you
Andris for making this possible.

------
js4ever
Wow I was looking for something like that since ages. Best part, it's open
source an made with Node.js/Redis, a stack I deeply love :)

------
indigodaddy
"it does not touch the filesystem"

How does this work exactly? Mongo would have to touch a FS at the end of the
day, no?

~~~
andris9
Sure, MongoDB does manage files but for the email app this is behind an
abstraction and possibly in a different machine. I was more referring to
maildir/mbox where the email server has to write and read files while juggling
with user/group permissions, invoking shell and such. WildDuck just writes and
reads stuff from a database like a regular web server software.

~~~
indigodaddy
Got it. I suspected that's where you were going with that but just wanted to
clarify. Thanks.

------
srameshc
Thanks, this is so timely. I was looking for something like this for weeks.
All I want is IMAP server which can be managed via IMAP client api.

------
amolo
Cool project. Been looking for something like this for ages. But still cant
figure out from the docs how to send email.

------
F_r_k
What's the difference with mailcow (a fully dockerized email solution) ?

~~~
andris9
Mailcow packages Dovecot, Postfix etc. in an easy to use solution. WildDuck
has its own and quite different implementations for most of these things.

------
Ente
I would love to try this.

Is there a way to configure catch-all addresses for domains?

------
mamon
Does it come with official Docker image? :)

~~~
andris9
I don’t maintain a Docker image myself but the best version you could consider
semi-official would be this [https://github.com/astzweig/docker-
wildduck](https://github.com/astzweig/docker-wildduck)

~~~
Aeolun
I’m using this image. Can vouch for greatness.

