I used qmail for about ten years. It was remarkably good from a sysadmin's perspective:
- the config system mostly used the name of a directory where lots of software would use a filename (qmail.ini => /var/qmail/conf), the name of a file where other software would use a variable name, and the contents of the file as the value of the variable. This is really, really easy to read and write without writing a parsing library.
- separate executables do separate jobs, with well-documented interfaces between them. Oh, and if a part didn't need root, it didn't run as root, and if a part didn't need to be the same user as another, it didn't share users. While it was easiest to replace something at the edge of the graph, it was entirely possible to replace middle jobs as well.
- qmail was the original design home of Maildir. I approve of various Maildir++ implementations (nesting folders is useful) and you could add support for them by replacing your delivery agent.
- qmail expected to be controlled by daemontools, a supervision system that has inspired/been blatantly copied and improved upon as runit, perp, s6, and nosh. (I have a suspicion that nosh is what most critics of systemd is really looking for.)
- fast.
- nearly bug-free.
What qmail gave up in exchange for all this: modern features and required antifeatures... which, it appears, notqmail is going to implement. Huzzah!
> the config system mostly used the name of a directory where lots of software would use a filename (qmail.ini => /var/qmail/conf), the name of a file where other software would use a variable name, and the contents of the file as the value of the variable. This is really, really easy to read and write without writing a parsing library.
this djb-ism has seriously affected a lot of the ways i unix, to date. envdir is such a great idea.
djb needs his own distro. He's written his own init, has thoughts about package management, config management, etc. He probably doesn't need his own kernel but a BSD or Linux kernel with DJB controlling userland would be interesting.
Agreed. Having had the opportunity to encounter, use, setup, and then later on, read the code and learn from qmail early on in my computer career I think it helped learn about proper system design with an eye towards security. Back when qmail was released there was nothing like it, the most popular MTA at the time was Sendmail and we all know how great that was :)
qmail had a great impact on system design for many years and especially in the mail delivery area. Things like Maildir redefined how even MUAs store and use email. djbdns also provided a breath of fresh air at the time it was released, worked much faster and much more secure than something like bind.
DJB also had many other (likely even more impactful) roles in the software industry, leading the fight against the encryption software export restrictions of the 90s.
I liked a lot of things about qmail, but I found it annoying as a sysadmin, like in the 1.07 days, because you had to do all this patching. I eventually wrote a script that would take all the sources and patches that one needed to run a reasonable mail server, and turn them into an RPM that could be installed, because I couldn't just provide the patched, resulting RPMs.
There were a lot of great things about qmail. I really liked the envdir and how secure it was. But eventually I found the attitude behind it very tiring and switched to Postfix, which is a really great program.
Yes, for a long time it's been the case that to run qmail you have to really really want to run qmail. That's why I put so much effort for so many years into cross-platform packages of qmail in pkgsrc, and that's why we have "Meeting all common needs with OS-provided packages" as an explicit goal for the notqmail project.
The question is if notqmail isn't a bit too late. I'm maintaining a qmail setup for a customer and the amount of duct tape we've put on that thing is rather ridiculous. As it is now, I'd rather replace the whole thing with a regular Postfix/Dovecot setup and be done with it.
Why not Postfix? I trust Wietse V. and Viktor D. a great deal, and I trust their coding abilities at least as much as if not more than DJB's for anything that isn't a cryptographic algorithm.
FYI, "I’ve learned more C, reduced build-time complexity, ..." probably isn't confidence-inspiring. I work with C a great deal and have for a very long time, and I though I'm very comfortable with C, I treat it with fear and respect -- I have found far too many security vulnerabilities just from staring at C source code to do anything else. Now, I too have "learned more C" over the years, but I wouldn't write a sentence like that as part of justification for some project written in C -- I would fear that others would wonder if I don't just know enough C to be dangerous. Perhaps one should not write "I've learned more C" as part of anything other than justification for moving to Rust.
Postfix has had 9 CVEs in 20 years, a few of them quite bad. [0]
qmail has had 1 CVE in 20 years, a local DoS. [1]
I trust djb to write correct code more than almost any other human on the planet. That includes all the non-crypto code found in qmail, daemontools, dnscache, ucspi-tcp, etc. If you’ve read it, you know how almost supernaturally careful and minimal it is.
Your points about mere mortals writing C are well taken though!
The moment you start patching qmail you're not running qmail any more. None of the CVE numbers are meaningful to your custom-patched qmail.
qmail was good when you could use it in its default configuration. Unfortunately, very few sites could (much less still can).
Which brings us to the question of who you trust more: The maintainers of a full-featured MTA who have maintained their software for 20 years or some person's patches to scratch their itch before moving on and never touching it again?
Can you revise that "9" number, because looking at it I feel like some of those are inappropriately filed against postfix.
I find it hard to pin a "postfixadmin" CVE on postfix, are we going to blame vmailmgr bugs on qmail? :-) The BSDDB one I'm on the fence about. "postfix_groups.pl" is marked as disputed. Another seems to be clearly a Debian package issue.
CVE-2011-0411 is interesting because, IIRC, qmail had no SSL/TLS abilities, and the hacks you had to put into place to enable it were abominations and likely opened up issues of their own.
Fine, let’s call it 7, since postfixadmin & the debian postinst script can’t be attributed to postfix proper.
I’m not sure this qualitatively changes the argument. CVE-2011-1720 and CVE-2008-2936 are examples of postfix vulns no one has found, or reasonably expects to find, in djb code.
> CVE-2011-1720 and CVE-2008-2936 are examples of postfix vulns no one has found, or reasonably expects to find, in djb code.
CVE-2011-1720 is related to SMTP authentication which qmail doesn't support out-of-the box. So. Yes you're right: You don't find that vulnerability in qmail because qmail doesn't have that feature.
Can you run an SMTP server without authentication these days? Not if you have users using you as a relay. So I guess you have to custom patch qmail and now you're back to not running djb code in the first place.
CVE-2008-2936 I grant you. Though once we are at doing hardlinks to symlinks, we might hit other edge-cases too, some might even affect qmail in some way.
Aside of the fact that I doubt your "most people" assertion here, it's beside the point that I was trying to make which is that a CVE affecting a feature of Postfix that qmail doesn't have to begin with is not a fair mark against postfix.
Because if you disable a feature you don't use, you won't be affected by the vulnerability.
If you use smtp authentication, you're not running djb's code and all advantages of djb's code are moot. If you don't use smtp authentication then that CVE is inconsequential and you can't count it as a black mark against Postfix with regards to the security of your specific setup.
> it's beside the point that I was trying to make which is that a CVE affecting a feature of Postfix that qmail doesn't have to begin with is not a fair mark against postfix.
I disagree. It is a fair mark against postfix if (as I claim) most users really don't need it.
Here's my reasoning to that point:
I think most email end-users either use their ISPs mail settings (who know who they are and don't need authentication), or webmail (which doesn't use SMTP for relaying anyway), or outlook (which doesn't need SMTP at all). Fastmail is becoming more popular and it doesn't need SMTP either. The end-users who remain are either business users (who could use a VPN which is better than SMTP AUTH in every way), and nutters with vanity domains.
If that's contended, and note I do believe there are probably end users who are actually still manually entering SMTP AUTH settings into email clients, but I would struggle to believe they're the majority and would need evidence.
Meanwhile on the enterprise, pretty much every machine in every enterprise environment I've ever worked in has been set up for local relay. Again, if you think they're all using SMTP AUTH I'd need to see evidence.
> If you use smtp authentication, you're not running djb's code and all advantages of djb's code are moot
Do you actually believe that if you apt-get install postfix that you don't have any of the advantages of postfix's development processes?
I'm not sure I can engage with that. It's like saying running linux invalidates "all advantages of djb's code" simply because you're (sometimes) not running his code (e.g. when you're in a syscall). Madness runs down that way.
For what it's worth, most of the qmail smtpd patches I've seen to add SMTP AUTH (including the one I wrote nearly twenty years ago) are sensible, integrate well with the rest of qmail. You might not get djb's warrantee, but I can't believe you gain nothing by starting from a solid foundation.
>I disagree. It is a fair mark against postfix if (as I claim) most users really don't need it.
if they don't need it, they won't be affected by that vulnerability because it won't be active.
>Meanwhile on the enterprise, pretty much every machine in every enterprise environment I've ever worked in has been set up for local relay.
In every enterprise setup I have seen, local relay was still protected using SMTP-AUTH for accounting purposes.
Plus in case of end-users, many ISPs still provide SMTP for mail delivery, all of whom need to support SMTP auth. That includes Fastmail and Gmail.
>but I can't believe you gain nothing by starting from a solid foundation.
Of course you do. But saying "Postfix is worse because it had a security vulnerability in a component that qmail doesn't even support" is unfair to postfix.
If you are running qmail with a custom patch then that custom patch will for sure have been exposed to less scrutiny than Postfix proper. It's not valid to extend the scrutiny given to vanilla qmail to all of the custom patches.
> saying "Postfix is worse because it had a security vulnerability in a component that qmail doesn't even support" is unfair to postfix.
Ok, well that's your opinion, and hopefully now you at least know why I have mine.
I can't imagine forgiving Microsoft for all those AD/SMB and explorer CVE just because Linux doesn't support those things, but it's not really that important to me.
> In every enterprise setup I have seen, local relay was still protected using SMTP-AUTH for accounting purposes.
I have no idea what you mean by "accounting purposes" -- it sounds like you're talking about users relaying and not machines relaying.
There is no way you'd get a SOC2 (or SSAE/SOC1) if you used SMTP auth to protect local relays, so perhaps we are talking about different things when we say "enterprise".
> If you are running qmail with a custom patch then that custom patch will for sure have been exposed to less scrutiny than Postfix proper.
I'm still not sure you can have it both ways. Why is it when Debian patches postfix that it's still (in your mind) "postfix proper" but when I download a well-distributed and discussed patch from a mailing list, and read it myself, that you somehow think I'm at higher risk?
>it sounds like you're talking about users relaying and not machines relaying
there is such a thing as machine- or even application-specific accounts that can be terminated or their credentials rotated.
Even better; you could use something like Vault (https://www.vaultproject.io/) to manage such application-specific credentials.
All of this is much better than relying on IP addresses never changing and never being reused.
>Why is it when Debian patches postfix that it's still (in your mind) "postfix proper" but when I download a well-distributed and discussed patch from a mailing list, and read it myself, that you somehow think I'm at higher risk?
I never said it was postfix proper. It really isn't.
Debian is known to f'up the security of its packages. But it's my hope that distributors have a better equipped security team and bigger exposure compared to some guy putting a patch on some website.
Debian also has a proper security update process that makes sure that patches go out to users quickly and painlessly. And because Debian is big enough, it's usually included in embargoed security vulnerability reports, so usually a patch will be out and sometimes even deployed by the time the vulnerability is known.
But the relative security of Debian-maintained postfix vs. Debian-maintained custom-patched qmail is the same as postfix proper vs custom-patched qmail.
However, vanilla whatever, in my opinion is superior to custom-patched whatever. And while postfix is usable vanilla, qmail isn't. It was bordering unusable back in the 90ies and it definitely is unusable nowadays.
qmail doesn't support TLS, it doesn't support spf, much less domainkeys or dmarc. It doesn't (as we're talking about) support SMTP auth, it doesn't support ipv6. It doesn't support any other common anti spam measures like greylisting or rbl checks, nor does it support a plugin system that would allow for easy plugging them in.
Heck, it doesn't even support rejecting mail at SMTP time so it's one of the worst offenders for backscatter which is one of the big annoyances for mail admins.
At this point, an unpatched qmail is as useful in fulfilling MTA roles as a closed port 25 is (which would totally be safer than both qmail and postfix)
Hence my assertion that everybody is running a patched qmail and none of these patches ever enjoyed the same scrutiny as even the security abominations that are sendmail and exim.
And worse: None of these patches were ever reviewed for side-effects of arbitrarily combining them.
Nearly every custom patch was made against a vanilla qmail because there is no central project to manage these patches. So every site runs a different version of qmail, none of them vetted at all for security.
This is a mess and we should really discourage this rather than dreaming of long-gone times where vanilla qmail at least was useful to a minor degree.
Partially-related update: I just looked at the qmail code: This thing is written in K&R style C and sometimes uses single-character variable and function names. I would argue nobody but djb can actually safely make changes to this code-base. This is all from a bygone age. Let it rest.
Sorry, we think it's rested plenty long enough :-)
You've given some excellent examples of how qmail got this way, and what notqmail needs to change to be viable. I have my own running-in-production solutions to most of them -- for instance, https://schmonz.com/qmail/rejectutils for SMTP recipient rejection and https://schmonz.com/qmail/acceptutils for user-facing AUTH and TLS.
These may or may not become part of notqmail. But we believe that together we can carefully and safely evolve notqmail to meet modern needs.
I agree. But if the software is lacking essential features so that everybody patches them back in it's a very ineffective strategy because anybody's home-grown patches are worse than properly maintained features.
I can put a machine on the net with a closed port 25. That's the most minimal implementation of an MTA and it's entirely secure. Guaranteed.
The thing is that qmail was severely lacking features back in the 90ies and it has received zero feature additions by its original author since then, so it's safe to assume that it's now lacking even more features to the point where anybody running qmail these days is nearly guaranteed to be running franken-qmail full of third-party patches which are surely less well maintained than proper Postfix.
Just a count of CVE, while interesting, I think misses calibrating by how much attention each has gotten. Given its popularity I'd assume much more security research and testing on postfix.
If we were starting from scratch, I’d be first to say let’s pick something safer than C. But we’re starting from where DJB left off, so there’s not much left to decide about language. Our roadmap aims to provide mostly Unix-process extension points such that new code can be written in any language.
Postfix is great, and I certainly hold Viktor in high regard (haven’t met Wietse). It’s just that some of
us really like qmail. :-)
Hey! I'd not noticed you were involved. EDIT: Where are my manners. Hope you're well too!
So, yeah, qmail is fine as a starting base, but there are important things that a modern MTA/MSA needs to have:
- MSAs need to support user authentication in
SUBMIT (no open relays)
- MTAs and MSAs need to support TLS
- MTAs should -really, need to- support DANE
for authenticating SMTP servers
There's more stuff too (delivery options, filtering, ...), but protocol-wise, the above are critical.
I'm not sure that portability is such a problem that you can't use Rust. Rust runs on all modern OSes, which is all you need to support in a new MTA/MSA. And Rust is fairly easy to get into, actually.
Vanilla qmail was never an open relay unless you really intended to configure it as one. It's just that the only relay controls were IP-based. Back in the day, before AUTH was widely supported, I used a really nice "relay-ctrl" add-on from Bruce Guenter for POP-before-SMTP.
For mandatory user-facing TLS and AUTH on port 587, and opportunistic server-to-server TLS on port 25, I've written https://schmonz.com/qmail/acceptutils. For SMTP recipient checks, I've written https://schmonz.com/qmail/rejectutils. It's too soon to say how notqmail will solve these problems, but they're solvable and we intend to solve them. For DANE... definitely not there yet.
Not that I disagree, but to inject some clarity: Python would be a perfect fit too. The major MTAs are written in C not because they need to be, but because C was the only serious option in the mid-late 1990s. Any modern language would work.
I used to run Qmail on a system that handled ~2m users. The main limitation was IO capacity. Today the M.2 NVMe SSD in my laptop can do substantially faster IO than the rack sized storage array we used for that mail system.
If you're building a competitor for Gmail or Hotmail, then maybe that performance difference will matter. For almost everyone else it's largely irrelevant.
EDIT: I'm confused by the downvotes for this. If someone has practical experience that suggests that performance is a real concern for this type of use, I'd love to hear about it. For my part I built and ran such a system for a couple of years, and spent a lot of time looking at the performance constraints. CPU usage just was not a factor even on ~20 year old (by now) hardware, running on 4-5 machines where the beefiest one had either 2 or 4 cores. All our optimization efforts consistently went on trading increased CPU usage for reducing IO use, because that is where the bottlenecks were. That's been reinforced with other mail systems I've run over the years, including large scale mail processing with PHP (not my choice) - every such system I've seen have been IO constrained rather than CPU constrained.
A few years back I ran `qpsmtpd` which is a flexible mailserver written in Perl. It was extremely easy to hack upon, so you could write plugins for SMTP-time spam-detection, rejection, etc.
These days the project is stalled, I think most of the development moved over to a nodejs based project.
Anyway the project was pretty simple to work with, and scaled wonderfully:
For most sites, performance is a total non-issue for providing the feature-set that qmail provided.
And if you are a site that processes enough email for performance to start becoming an issue, you're probably better off with a custom solution that's tailored to your specific issues
Postfix cannot do what qmail can do. Specifically, users can create and destroy email addresses themselves with .qmail files. This is different and more powerful than Google's plus-addressing approach. There are many, many other features of qmail's implementation that Postfix cannot compete with.
Postfix is not alone in this; no other MTA/MDA can do what qmail can do.
I've got a similar thing with my postfix setup though. My users can go to their account portal and apply for an alias, which when approved (to avoid taking of important names like postmaster etc.) is added to their otherMailboxes LDAP attribute. They can delete the alias themselves when they're done with it.
I'm tempted to start auto-allowing new aliases and just having a list of names that can't be used (postmaster, webmaster) etc, that some services use for domain ownership verification.
As an aside I do hope qmail offers similar limiting abilities. It wouldn't be great if a user could make themselves receive webmaster@domain.tld emails!
They can’t, unless qmail has been configured to know they control mail for domain.tld. The admin configuration mechanism for this (control/virtualdomains) is simple and powerful.
Sorry I'm confused. If your mail server is setup for control of a domain (e.g. domain.tld), then what stops someone from putting postmaster@domain.tld in this .qmail file?
People can put whatever they want in their own .qmail files. That doesn't affect anything unless qmail believes those .qmail files are relevant to what it's delivering. A user controls .qmail files for a domain if and only if an admin has configured qmail to delegate that domain to that user.
Rust is not a magical language. It's just got the right features to be a C replacement, which is mostly: a simple type system (much simpler than C++'s or Java's) that has the right extensibility (traits, no inheritance) and can have an ABI that programmers can reason about due to monomorphization (no v-table dispatch).
All the borrow checker stuff amounts to: just enough pain that to avoid it you must design better APIs.
Yep, after reading the post I'm not any less impressed by (or interested in) Rust. If the post's claim were "Wait until Rust changes more slowly before investing more heavily in it," I might be able to agree.
For a project like notqmail, I would worry about portability. qmail runs on a lot of platforms and getting Rust bootstrapped is a bear.
For any new MTA/MSA you don't need to run on old HP-UX systems. You can elect to only run on systems that support Rust, which is to say: all the modern OSes.
Just wanted to add that Erwin Hoffmann has actively maintained s/qmail (and related djb tools) for quite some time. =) Please feel free to check his website:
Yes, and while I haven't run s/qmail myself, I have sometimes borrowed (and then usually refactored) some of his code, and have frequently collaborated with him on improving ucspi-tcp6, ucspi-ssl, etc. I'm still hoping he'll consider participating in notqmail development.
I have to wonder how much work is going to be required to bring qmail up to a modern standard. There's no doubt that the author is an extremely intelligent programmer with a deep focus on security. In the intervening period Postfix has really gained in popularity, and is shipped pre-configured out of the box on some major Linux distributions.
It is the default MTA for Fedora/Redhat etc for some years now.
I wanted to like qmail, it was definitely an improvement over the complicated disaster that is Sendmail, however, it always seemed to require patches to get it to do things that Postfix could already do out of the box. In addition I didn't like the dependency on the inetd replacement thing (daemontools?). It just felt so un-ergonomic.
In addition, the Maildir mailbox format (while very cool in its own right) usually required third-party software to be patched manually, for example WU-IMAP.
A day or two before the September 11 attacks, the UNIX time_t value rolled over to 1 billion, and went from 9 digits long to 10 digits long. The moment this happened, a comparison function in the WU-IMAP maildir patches failed, and our IMAP server started sorting and displaying all our emails out of order.
After scratching our heads and poring over the source code for a couple of hours, a co-worker and I found the problem and submitted a patch to the maintainer.
Funnily enough, the imap-maildir page still references this event even today (it probably hasn't been updated in years):
Under the section "10-digit unix date rollover problem".
But it drive home to me as a fairly young sysadmin at the time of the dangers (at least for the inexperienced) of using third-party patches just in order to enable a non-standard mailbox format (eg. Maildir).
> Did your installation stop sorting messages correctly on 9 Sep 2001 01:46:40 GMT? Find the fix to the 10-digit unix date rollover problem in the important updates section below.
Reading that I imagined someone running those patches on their mail server for two decades before suddenly realizing one day in 2019 that for the past 18 years their messages have been ordered incorrectly, ha ha.
It will be a lot of work, no question about that. Fortunately none of it will be about Maildir, as folks like you have long since ironed out all those issues (thanks!).
It's possible to package qmail in such a way that it's trivial to install and run, supporting many modern features by default. I've done it. Here's a demo: https://youtu.be/Vq6vu9T3vow
But that required a lot of decision-making and a lot of effort by the packager. With notqmail, we hope to make packaging much much easier.
I run my own email server as well, and I've been using postfix/dovecot quite happily with zero issues. I've been contemplating opensmtpd as well, though I'm not sure where that stands when it comes to being run in production environments. Also, reconfiguring mailservers is a PITA, I'd rather stick with a setup which just works.
Could someone explain why one might want to use qmail (or this, its newest incarnation) over postfix or opensmtpd? Would I gain anything by shifting to qmail or opensmtpd?
As far as opensmtpd is concerned, it is production ready and has been used in high volume environments sending mails to millions of recipients daily, handling multi-million queues.
The project was started in 2008 so when it comes to being run in production environments, I think we can claim that it's ready :-)
Now as for the what you would gain, I won't do proselytism, we have published a couple presentations on our website which describe how it works and some of the technical details behind it. You will also find lots of technical details on my blog where I post mostly about OpenSMTPD internals.
If you have a running Postfix, I suggest you don't shift but keep your working setup. If you intend to setup a new MX, you should definitely give a try and see how it goes for you.
Thank you very much! I shall have a look. I'm thinking of moving servers and setting everything up from scratch in order to modernize all my services and remove legacy cruft. I will definitely consider opensmtpd.
For the moment, we're targeting sysadmins who already run qmail (or netqmail). Much later, when we begin to approach feature parity with Postfix, notqmail might start being an interesting alternative. We hope that time will come. If you can't figure out why you'd run it right now, you're probably right :-)
We all killed SMTP when we pointed the morons with domains at gmail/apple/yahoo/o365, postfix was winning when it was ended- no reason to revisit it. All that matters for email is what Google/MS/Apple decide to dictate to us now.
- the config system mostly used the name of a directory where lots of software would use a filename (qmail.ini => /var/qmail/conf), the name of a file where other software would use a variable name, and the contents of the file as the value of the variable. This is really, really easy to read and write without writing a parsing library.
- separate executables do separate jobs, with well-documented interfaces between them. Oh, and if a part didn't need root, it didn't run as root, and if a part didn't need to be the same user as another, it didn't share users. While it was easiest to replace something at the edge of the graph, it was entirely possible to replace middle jobs as well.
- qmail was the original design home of Maildir. I approve of various Maildir++ implementations (nesting folders is useful) and you could add support for them by replacing your delivery agent.
- qmail expected to be controlled by daemontools, a supervision system that has inspired/been blatantly copied and improved upon as runit, perp, s6, and nosh. (I have a suspicion that nosh is what most critics of systemd is really looking for.)
- fast.
- nearly bug-free.
What qmail gave up in exchange for all this: modern features and required antifeatures... which, it appears, notqmail is going to implement. Huzzah!