Hacker News new | past | comments | ask | show | jobs | submit login
UUCP must stay; Fetchmail sucks (2001) (freebsd.org)
108 points by segfaultbuserr 9 days ago | hide | past | favorite | 75 comments

Fetchmail grew to be a mail transport swiss army knife and encompassed half baked functionality of a domain-wide MTA rather focusing on single user async email consolidation and collection for a logical single account, which is largely what this rant is about. Half of the half-bakedness is due to lack of MTA protocol support for MUA interactions on the receiving side.

pop3 was never meant as a spool for MTA to MTA email transfers, but was instead meant for an MUA to access. It's not fetchmail's fault that the envelope TO isn't available via pop3. That, at the time, email service providers bolted on entire domain delivery to a single mailbox, and then users of those services put multiple recipients behind that single mailbox. Fetchmail made this work as best it could. Also missing at the time was any kind of serious connecting-client-as-MTA or MTA-to-MTA authentication (rather, relying on DNS and TCP!), which might have made async, receiver controlled MTA initiation more workable.

This was also due to common SMTP MTAs not having good (or at least easy/obvious to configure) store-for-later-pickup functionality where an MTA could query for email destined for itself rather than be always-on connected to the internet waiting for another MTA to initiate delivery. Fetchmail filled a hole that existed in email delivery at the time, but it was really a hole that should have never existed. The quality of the fetchmail code is another, albeit serious, issue in itself.

These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.

These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.

I don't want to be "that guy" but we've known about this distinction since the 1980s. I know, because I was fringing in the discussions in standards ml during the x.400 years.

You are "that guy". I didn't say we didn't know about this distinction, I said we have a better understanding. It used to be the case submitting originating email on port 25 was accepted. Now, if you have that configured, it's wildly considered that You're Doing It Wrong™.

Many email clients, like Eudora and Thunderbird, were submitting email on port 25 for a long time, in the clear, where the only protection was very loose source IP address restrictions for outbound, otherwise you'd be an open relay. Using different servers for inbound vs outbound mail submission was a relatively rare thing too, because the routing can be a bear to set up. SMTPS (SMTP over SSL) on port 465 wasn't official and was locally used where it could be supported (by the client) and where an admin had bothered to configured it. Many email clients didn't have support for even using a non-standard port for any thing mail related, so even once the mail submission port 587 was standardized it was often a problem getting a client to use it or providing authentication on it.

But even the OP is saying that fetchmail is a problem because it's trying to be an a full, transparent MTA when it's really a glorified MUA. The fact that fetchmail (and its ilk) talk MUA protocols to receive email (pop3, imap) and can then turn around and reinject that message into the mail stream using SMTP to implement local delivery is where a lot of the problems with fetchmail come from (such as munging the headers or creating email loops with mailing lists).

Quite a lot of the anti x400 vituperation was about complexity. People didn't want to think about the distinctions and where open 25 was possible, with 110, the barrier to entry was low.

Setting up a proper mta/mua/msa respecting service was hard, and for many people remains hard. Exploring how to get utf8 clean across all the m* protocols for lhs@rhs reminds me how complex this can be.

I actually learned the difference between MTAs, MUAs, etc because of fetchmail. Fetchmail kept randomly crashing on select spam messages and my bug reports went unheeded (and I wasn't what anyone would call a programmer) so I needed a new way to download my mail. So I wrote a crappy Perl script. All it does is use standard Perl modules to download the mail, and pipes it off to an MDA (postfix, maildrop, etc) to deal with them.

After nearly two decades, no matter what system I plunk it on or mail provider I use, the same stupid script just keeps working.

I have personally found that as you get serious about keeping your own archive of email you need something like fetchmail, but not fetchmail.

I used getmail for years and it is solid. duckerude posted this elsewhere in the thread, where the author of getmail rants a bit about fetchmail. http://pyropus.ca/software/getmail/faq.html#faq-about-why

Once I left Gmail I was interested in something faster and more efficient, because I was using the transfer tool for more than just backup, it became the main way I receive email on my machine. Then the mail gets indexed (for text search) by another program and displayed by a third.

Anyway I found mbsync, which is just this amazing, efficient, very solid transfer program that's been around for many years just quietly doing its job. I almost wrote it off because its homepage is on sourceforge but if you poke around a bit you'll see it is actively developed, there's an active mailing list, etc. I love it. It's kind of interesting how this project I had never heard of just does its job well and works and the program by the semi-famous software pundit whose programming essay went viral really does not (from what I have heard).

fetchmail can run as a daemon and fetch from imap with IDLE. As far as I can tell, mbsync doesn't do this.

I haven't found a single other program that does this well; there are tools to sync entire directory trees that can idle, but they don't seem to support "download everything and delete it" the way fetchmail does.

Many e-mail services throttle new connections, so it's not the difference between getting your mail now and 1 minute from now, it's the difference between getting your mail now and 5-10 minutes from now.


Another thing keeping me on fetchmail; I have all my filters setup with maildrop. Most other programs want to deliver to a local folder directly. Maybe there's a way around this?

Same here (like another sibling reply). I really don't get the complaints.

For private email consolidation, I've been running fetchmail for probably 20+ years now without any hiccups or complaints whatsoever. I haven't had to touch my Fetchmail+Postfix+Maildrop setup in like 15+ years, except for the postfix part in the sense that nobody accepts emails from dialup IPs any longer, so everything I _send_ goes through the proper mail server, instead of my own.

My actual email provider (that I've also used for 20+ years, i.e. no, it's definitely not and never will be GMail) does throttle connections via POP3 or IMAP to 3 connections per 5 minutes. That's completely fine. I really don't care whether I get an email now or in 5 minutes from now. I check it way less often than that and I do definitely _not_ have notifications turned on for my email client. And when I do care (coz you know, some "2FA" BS email, I check via my provider's app or log into my server and start a `fetchmail`.

I just got started with mbsync and am having several issues with Gmail and Outlook.com. These are old accounts and I do have well-behaved IMAP emails from smaller hosts, but it would be nice to manage everything with mbsync. Trashing emails in mu4e and then syncing results in them being deleted from my local [Gmail]/Trash while they somehow remain in the remote. I thought Outlook.com would be fine considering they use more standard folders rather than the tagging system which seems to result in duplicates, but I have the same issues there. This is with Expunge None, which I expect would leave emails alone until the server empties them from the Trash.

The bizarre thing is that no one online seems to have these issues? The mbsync configs for Gmail just seem to work for people, or something. I do have auto-expunge turned off in Gmail, so I’m kind of at my wit’s end with it. It’s a shame because it is an amazing program. It synced 10,000 emails without a hitch. I only started getting issues when I tried to sync from my maildir to the IMAP store.

I’ve only run mbsync unidirectionally as a backup utility (in addition to tarsnap on my Mailserver). I wonder if the bidirectional features just aren’t used that much?

Also, Gmail’s tags don’t map nicely onto folders.

I use a fdm + s-nail combo. I know I can use s-nail over imap, but this way I am sure I always have my mail available.

I've used fetchmail for years to get email from IMAP inboxes (mainly Exchange server) and never had a single problem with it. Was surprised to see that there are major complaints.

Most of the usage complaints seem to be when using fetchmail to send off to another smtp server, not for just downloading mail.

There are also complaints about it being insecure and of low code quality, but those seem to rarely be the main complaint.

I used to use UUCP to get my email when I was on a dialup, non permanent connection. I was able to host my own mail server. There were a lot of technologies back then helping the internet stay somewhat decentralized. Now I, like most, just use Gmail.

What has become of our beloved free(digital)land?

Well, part of this is on us. Let's think on the reasons Gmail is so popular:

1. It's very easy to get to.

2. It has incredibly fast search that has 0 setup.

We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with. The best we have is .mdns which either works magically or perversely refuses to work.

Similarly for free text search, the software world simply hasn't delivered a lego-like solution for email search. You CAN rig up any number of open source projects but it is neither easy nor instant. And even other professional products like Apple Mail struggle with a mere gigabyte of email.

Despite the fact that it's 2021 and every successful email provider aggressively solves these problems, the open source world still debates about the utility of ubiquitous search or pretends that local networking isn't a pressing problem.

"We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with."

I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config.

The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as

     Desired email account: [________]@gmail.com
     Password:              [________]
     Verify Password:       [________]
     [X] I agree to have all my data used in arbitrary ways
It's not possible. There is no way to set up a server that easily, even in principle.

Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.

>It's not possible. There is no way to set up a server that easily, even in principle.

I partially agree, but I think we could get a lot closer than we are now. It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name, and that's tricky to do as part of a FOSS project. Maybe you could integrate with the APIs of a few registrars, but... it's not ideal.

As far as the "run thing on server" side of it goes, though, projects like Sandstorm[0] have gotten really far re: making it a simple process. I stood up this instance of Etherpad with a few clicks on a web UI, for example: https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQon...

Sandstorm in particular doesn't quite work for mail servers just because the software is heavily oriented towards webapps, but there's no reason a similar system couldn't work in principle.

[0] https://sandstorm.io/

The involuntary back burner started designing "how would I set this up as a business", and I definitely ended up with setting up docker containers. Sandstorm, unfortunately, is an example of being a bit too ahead of its time... sandstorm has a lot of stuff in it that just isn't that big an advantage to me vs. being in a Docker container. (I'm aware of at least some of the differences, but... it's just not that helpful to me for the most part.)

I think something that could be like sandstorm but using docker containers might be a good thing, but there's still the "you need to hand me your credit card if you want me to register a domain for you" which is a big barrier vs. the screen I showed above. A credit card off the top is already 4-6 entry fields, and everyone knows to be a bit nervous about entering those....

> It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name

I can't agree, getting a domain is much easier than it is to configure the abysmal shit that is Postfix+Dovecot+WebGUI. God forbid you want proper search as well.

Basically all of that could be done programmatically though. There are sane defaults that can be used for these things; projects like Mail-in-a-Box do most of that heavy lifting for you. (When all you want is a basic SMTP/IMAP/webmail server... a standard config can work just fine.) It doesn't even necessarily have to be done through a command line.

What can't so easily be done programmatically is DNS registration, because it involves money changing hands and there's no standard "registrar API". You'd have to either support an arbitrary number of differently-shaped APIs of different registrars or pick a "blessed few" to support.

Realistically, you could support half a dozen DNS providers and get a LONG way towards making it easier to roll your own.

GoDaddy, GCP/google=domains, AWS, Azure, DigitalOcean and a couple others would provide enough for a LOT of people. Especially since godaddy and google domains include dns, and DO doesn't charge for it.

I've setup a script to make it easier to manage a handful of dev domains (commit to a repo, and it updates via build/event pipeline) with DO.. was surprisingly easy. I don't think doing similar for the biggest cloud/dns providers would be all that hard to integrate.

Another issue that needs to be solved is 'bad actor'. Basically someone gaming the system and overloading it with spam. There is no real nice neat way to fix that. Instead it is a mishmash of blacklists/whitelists/blocklists and sorta intelligent alg filtering. Getting that all filled and working is not trivial either. Oh its 'doable' but kind of a pain for even someone with decent experience at it.

To add to that; with re-assigned IP's, widespread port 25 blocking /most/ ISP connections won't allow you to run an email server on a residential connection.

Spam/botnets have a lot to answer for.

Agree 100% on both of these, and #2 is seldom listed as a big issue. Had IMAP addressed search in a better way, I think it would have made a huge difference.

Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives.

Question is, is Gmail hegemony fixable now?

I think the best approach to breaking the Gmail hegemony is to replace email with something better. Some features I think such a system should have are:

1. End to end encryption 2. search as a first-class feature 3. decentralized/federated 4. standard minimal rich text format (maybe something like markdown), instead of an inconsistent subset of html 5. Fix some of the legacy limitations of email, like having to be ascii safe, line limit of 70 characters etc. 6. Possibly make it easier to have an identity that isn't tied to your service provider

The big questions are how would you get the general population to switch from email, and what to do about spam.

4 there's already a standardized rich text format for emails, called "enriched text" defined originally in rfc 1563 almost exactly 27 years ago, and last updated in rfc 1896 about 25 years ago.

It supports many things that markdown does not (setting font families, sizes, and color for one example), and nobody uses it anymore.

Another point is that markdown is still pretty readable as plain text, whereas html, or "enriched text" are a little harder for humans to read if it isn't specially formatted.

and afaik hardly any email clients support it. and even if yours does, chances are the recipient on the other end uses one that doesn't.

Right, so I don't know what inventing a new such format will do to improve things.

1> GPG.

2> Your MUA does that. Get one that doesn't suck.

3> Impossible and/or dangerous.

4> No. HTML is NOT part of a MIME standard, even.

5> That's an issue of netiquette, often you have to post lines over 70 chars in News servers, too.

6> That's a different issue.

1. GPG has a lot of problems. Perhaps the biggest of which is that there isn't a critical mass of people who use it, and it is pretty difficult to us at all, much less to use correctly.

2. I have tried several MUAs and haven't found one that IMHO "doesn't suck." Although search isn't always the problem. (The biggest problem I've run into is actually getting them to work with logging in using an SSO method like SAML or OIDC). And this point was specifically in response to the parents statment that "Had IMAP addressed search in a better way, I think it would have made a huge difference."

3. Not sure what you mean by that since email _is_ decentralized and federated. Meaning, if you use gmail as your email provider you can still send messages to people using a different email provider. As opposed to say something like Signal or Whatsapp, where you can only message people using the same service as you.

4. Well, some subset of html has become the de facto standard for rich text in emails. Although what subset that is depends on the client...

6. Different issue from what? It is a very real complaint I have with email. Changing your email address is a pretty big pain, possibly more so than changing your physical address or phone number. For many people, myself included, the thought of having to change my email address is a pretty good motivation for staying with the service I currently use. Now it is possible to use your own domain with many email providers. But doing so is beyond the technical skills of most people, and costs more for essentially insurance in case you stop liking your current email provider some time in the future. Unfortunately, I don't know of a good solution for this. At least not one that is scalable.

Yeah, none of the IMAP server authors really thought of search as a first-class feature, nor did they seem to want to do the development to build the backend support. Resource constraints were also an issue back then, as well as a client-first approach. Most large mail servers only had enough resources to run the IMAP and POP services themselves; adding search index maintenance would have overloaded their CPUs. (Not that they couldn't have added more CPU, but this was all pre-cloud.)

> Question is, is Gmail hegemony fixable now?

Maybe, if we get maddy working near-perfectly and kept up-to-date.

I do agree with you, but let's add:

3. dealing with spam is a massive PITA

Most of spam problems could be solved if mail software came with sane defaults. Sane rate-limits, sane IP reputation, sane security rules, some backscatter protection. Something. Most setups are so widely open because good configuration is arcane knowledge.

I use offlineimap/mu4e over Exchange account at work, in parallel to a personal Gmail account. Would not say now that Gmail has a perceptible edge there.

Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).

What about mu plus mu4e

Thunderbird plus global search

Recoll + client of choice


3. A whole lot of free storage, so most people don't have to worry about needing to delete old emails.

For the search problem, notmuch is pretty easy and has better search capabilities than gmail.


Spammers and abuse.

I used to run personal email server some years ago. The spam was harder and harder to manage - first local SpamAssasin was fine, the I had to tweak rules, then I tried distributed filtering (DCC)... and then I gave up and started paying for Fastmail. This ended up working much better than I could ever do.

Usenet was similar -- I remember looking at the newgroups which were at more than half spam (and some were 95%+ spam) and then just leaving for forums. (plus there were a major UX annoyance: unlike with email, there is no simple way to sync read status across multiple devices)

Our free(digital) land has been killed by a handful of bad actors. I no longer believe in "free for all" systems. The best we can do is many independent systems, each with its own rules and policies.

I run my own mail server with only Spamassassin for filtering, and I get little spam to my inbox in spite of my e-mail address being posted in plaintext across the internet. It has sufficed to 1) simply activate distributed filtering within Spamassassin, and 2) set a slightly more stringent standard than the default for regarding a message as ham, since any legitimately sent message will easily meet that higher standard. I also fed my entire e-mail archive to sa-learn, and what spam I occasionally get I also feed to sa-learn.

In ten years of running this mail server, I have only had one persistent spammer I couldn’t manage to get filtered with these settings, but then I just created a custom rule.

And _that's_ the work that is needed to have a spam-free-ish setup. Why can't FOSS ship these defaults and do these trainings semi automatically? Something totally could based on specific folder contents, but nothing does, it's always this "reinvent the wheel"-type of stuff.

Spamassassin is storage-agnostic, at heart it is just a script that takes messages piped through stdin. It doesn’t know where your preexisting e-mail archive is stored on disk. Storage is done by either your IMAP server or your e-mail client, and obviously it is up to you to tell Spamassassin where those messages are for Bayesian training.

Yep, that's the problem, it doesn't know. But for a good product it should know, these things should be possible to integrate trivially with the result being both reliable and automated.

Lucky you. I am not sure what I did differently, but I had those (including a custom script to run sa-learn on my "spam" folder), and ended up with close to dozens of spam messages per day anyway.

Greylisting and DNSRBL always worked sufficiently well for me, in addition to Spamassasin.

I think worst than the spam problem is the deliverability problem. Worst case I can hit the delete key on spam. (as an aside, most of it is immediately obvious from the subject line alone, which makes it hard to believe that it's such a hard problem). But getting mail delivered from a private server to major email providers like gmail, yahoo, outlook, is difficult.

... and "deliverability problem" is caused by spam. Because if it were easy to get mail delivered, spammers would do this.

Yep, I remember spooling usenet and email to my Atari ST over a 1200 baud modem over UUCP in 1990, 1991. Bang path and all.

And you're right, the centralization is crazy, and opposite of what techno utopians predicted/promised at the time.

Sadly, the upsides are just too good. I love decentralization, but I have to recognise that I've not lost a message since I moved to gmail in 2005; before then, losing entire mailbox files was not uncommon, as well as missing messages from unreliable mailservers, spam management, and so on. I'm sure I could have done better etc etc, but the bottom line is that with gmail-like services you don't have to. I enjoy doing sysadmin but not 24/7.

Oh hell yes, I basically don't delete email ever and as a result I have an archive of every email (pointless or not) since 2005. (Though I just checked and my first email on gmail is porn spam). Meanwhile everything I had from every account prior to that, dating back to 1991 or so, is lost forever. Apart from some very embarassing forever-archived usenet posts.

My first access to the web around '93 was via CERNs e-mail to web gateway coupled with UUCP exchange from my Amiga to my local BBS which exchanged with its upstream 4-6 times a day.

Yeah I used to dial into the local university's gopher "campus information server" -- which was just a dial-up to telnet to a gopher client -- hit the ^] telnet escape character, close the connection, and re-open to another host where I had a shell and go from there. If I recall right it was probably gnu.ai.mit.edu that gave out free shell accounts, or something else in that domain connected somehow to Stallman.

There was no ISP in my area back then, not even dial-up. Again, around 1991.

Me too! I ran AmigaUUCP on my Amiga, dialed up at 9600 baud to my upstream. This was around 1992 or so. I had Internet email and a Usenet feed of maybe 15 groups or so. Fun times.

> When mail is delivered to a POP3 maildrop, envelope information is destroyed. To combat this, you would need to tunnel the enveleope information in headers.

Isn't that exactly what Envelope-to is supposed to be? Protocol-level data? It's not really part of the e-mail, but added when sending it. If you forward the mail to another machine, there will be a new envelope.

I can see that it's all a gray-ish area with email (because email is a mess (which might be part of why it's been successful)) but I can't see a reason to complain about losing Envelope-to.

Would anyone that's well-versed in UUCP, Fetchmail and related mail protocols and technologies care to explain whether this discussion from 2001 (20 years ago!) is still relevant today?

This doesn't seem so much about UUCP vs fetchmail as about cathedral vs bazaar style of development. In hindsight, it's clear that both of these work, for better or worse, with each having its own disadvantages and benefits.

Think this story is a developer who submitted a patch to they guy who wrote the manifesto for open source. The patch was rejected for non-technical reasons, which led to the developer pointing out the irony of the situation. It's actually funny, and if you ever used fetchmail, you'd know just how accurate Terry Lambert's assessment of fetchmail ("abomination before God") was...

I'm glad someone else actually got it. Didn't think this was all that complicated, but context is everything, I guess.

Some context for those who are unaware: fetchmail was written by the same person (Eric S Raymond/ESR) as the "Cathedral and the Bazaar" essay.

If you have any interest in open source development (which these days we all should given that 95% of the lower parts of our stack are open source projects) it's well worth reading the essay/book.

If you don't want to do that, at least read the wikipedia page: https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar

Not really. I read it when it came out and remembered liking it (we were all on kind of a buzz about open source and Linux at the time) and then again much more recently for a thread on HN, and I found it to be mostly a rehash of other, better material --- Brooks and Kernighan, in particular. That, plus the modern context that what he was writing about was Fetchmail, and Fetchmail as it turns out is hot garbage, to me makes the whole thing very skippable.

It's also responsible for hatching one of the great canards of open source, "Linus's Law" (I'm sure Linus is thrilled with having his name attached to that), "many eyes make all bugs shallow", which has in the last 10 years become one of the great jokes of computer science.

I remember reading his essays (there were several more) and thinking, "He is seems so sure of everything, let's see how it plays out".

He wasn't completely wrong about everything, but the main premise - that careful planning and development yields worse results than guerilla ("bazaar") way of working - didn't pass the test of time. His main argument was the Linux kernel. But when you look at this project now, it seems quite far from the bazaar style. There are many bazaar-style projects that failed and many cathedral-style projects that do very well. So his main argument is moot.

We can treat this essay as a historical document, not something anyone should refer to as to how the open source development should be done (even though he has some good points about leadership style for example).

> We can treat this essay as a historical document

It's why I'm rereading this book today.

One thing I've been thinking about since morning was ESR's "Linus Law", "given enough eyeballs, all bugs are shallow". Today the idea is heavily criticized. The very existence of fetchmail by the author was a counterpoint - the code quality was below average, it had numerous security vulnerabilities, and arguably basing the whole book on fetchmail is the book's greatest flaw (not to say the criticisms of ESR's personal qualification from many hackers).

But if you read the original context...

> Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this: Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

The original context was strictly about beta-testing - about how the Linux kernel didn't collapse on its own weight despite the lack of a BSD-style "core" team and heavily scrutinized release. Instead, it achieved system stability with the help from a large number of beta-testers and co-developers who are willing to run on the bleeding edge. It's still how the kernel and the majority of the FOSS world. "Build today's git snapshot and see whether it explodes" can be quite effective.

Then ESR simplified it to...

> Or, less formally, "Given enough eyeballs, all bugs are shallow."

And now it means "people can find all bugs, because there are always a lot of eyeballs to read the source code". Now it can refer to all sorts of things beyond its scope. It is a serious mistake. Notably, security vulnerabilities are different - it usually does not have any user-visible effects and can only be triggered by carefully crafted malicious inputs, not normal uses. This is the reason it doesn't work for security. Thus, maybe a more accurate reflection of this rule could be: Given enough eyeballs, all user-visible bugs (e.g. crashes) are shallow.

At the end of the day, my conclusion was: Perhaps "Linus Law" was not as wrong as it seemed. ESR simply misinterpreted his observations.

The "law" isn't that "all crashers are shallow". That would be a silly law, because you don't need many eyes at all to notice crashers.

Actually, you do, especially in the kernel. No one person could ever replicate all the crazy combinations of hardware, software, and usage that people need.

I found this interesting post while reading some historical documents, so I submitted it to HN. It seems that most people have missed the point. I think the point is: while The Cathedral and the Bazaar was a good campaign pamphlet and a general overview of FOSS development, but ESR's attempt of using his fetchmail as an example was arguably the greatest flaw of the book ("to such an extent that it could damage the Open Source movement"). From a technical perspective, fetchmail had numerous problems and security vulnerabilities. ESR was also criticized for mismanaging the project by putting it in "maintenance mode" early and refusing to accept patches that could address those issues. After a group of new maintainers had taken over the project, one of the first things they did was to systematically eliminate countless buffer and integer overflows from the program.

Can confirm that fetchmail is garbage. Found bugs, reported them, no fix. Wrote my own tool instead. Such is life in open source.

I wonder if many folks who only joined the world of software development after DVCSs (and indeed, just VCSs) became popularized can understand why Fetchmail was not trivial to fork and fix.

The only time I ever interacted with ESR on fetchmail, I reported a bug for an issue I ran into (with Kerberos 4 IMAP), submitted a patch, and it was accepted without discussion in the next release.

Admittedly, this was 19 years ago, and it was a relatively small fix.

I've been using fetchmail every two minutes (daemon mode) for well over two decades and not once has it failed to do what it needs to do. So it's a bit of a stretch to call it garbage. Other than emacs and mutt, there's no other tool with such stable longevity in my toolbox.

Under ESR, fetchmail was garbage. However, a new team took over development in the early 2000s after ESR abandoned the program. Many parts have been re-written to address design criticisms.

Fetchmail was only supposed to be something like "offline IMAP" for someone with only occasional network connectivity, and whose mail program is too braindead to download and cache messages for offline reading when connected.

Of course, fetchmail wouldn't have the envelope info, because that occurred in an SMTP chat that is not part of the message landing in your inbox.

Something that accesses delivered mails in IMAP or POP3 boxes cannot be a full-blown MTA. E.e. we can't program a mail exchanger machine to accept the mails on behalf of hundreds of users, by delivering them into a giant mailbox, and then continuing the delivery path by sucking them out with a fetchmail-like tool, and forwarding to more forwarding hosts.

The whole complaint of this irate BSD person is that you can't do this kind of nonsensical thing.

There is a sort of valid criticism in that fetchmail (or at least did originally, I don't know about now) worked by re-sending the messages to the user over a mail transport mechanism, rather than delivering it into a mail directory structure. E-mail clients that download and cache messages certainly do not do anything of that sort. However, that shields fetchmail from having to know how the user's mail software likes to have the messages stored. Arguably, sending the mail is the right interface.

Though the envelope info isn't available, it's assumed that it was already processed by the remote mail server. The remote server did all he spam checks on the SMTP info, and whatnot, decided that the message is deliverable to the user, so there it is.

Fetchmail assumes that if the mail is in the remote box, then it's yours, and can be effectively forwarded. The envelope sender is Fetchmail itself.

If that is broken, so are mailing lists. When you receive a message from someone via a mailing list, you also don't have the envelope which the mailing list robot's server received. The envelope is that of the mailing list's own delivery; the mailing list robot is the MAIL FROM: sender.

For that matter, you are not even the original intended recipient! The mail was originally to the list, and now it is to you.

Not having an envelope isn't a show-stopping impediment to further forwarding and delivery. You just have to understand and accept that it's more of a forwarding style of delivery, like a mailing list robot.

Overall, it's almost like complaining that a dog which fetches the mail from your door and brings it to the kitchen is not a trained postal worker, does not understand and handle addresses right, and could not be reliably used to service an apartment building with six tenants, all by itself, such that the mailman could just put everything in one bin and leave it to the dog.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact