
This is the mail system - tbassetto
http://dewith.com/2012/this-is-the-mail-system/
======
orthecreedence
I completely disagree 100%. I think the mail system is exceptionally well-
functioning. It's a massively-scalable, decentralized, asynchronous messaging
system that supports large file attachments. At this, it excels. There's not
one big evil company controlling all messages and unless you're sending
massive amounts of emails every day, your messages almost always end up
exactly where they're supposed to. I'd challenge anybody to do better.

The author's main gripe is Mailer Daemon messages. It's not _email's_ fault
that people write shitty clients that don't display a pretty failure message.
If your car doesn't have anti-lock breaks, it's not the road's fault when you
hit a patch of ice and slam into a tree.

Email is great not only because it works simply and beautifully, but because
it's so extendable. People think it's broken. It's not. It's just simple...and
it's this same simplicity that makes it easy to build on top of. Look at Gmail
or Xobni. Both have incredible features that "email" doesn't have because they
took the time to build extras on top that make it work nicely.

Email is NOT broken. There's nothing to fix. Whatever fixes you think it needs
_you can make yourself_. The underlying platform works great. If you have a
problem, write/patch a client.

~~~
Niten
Exactly. It would be no small matter to come up with an alternative system
that has the simplicity, flexibility (including flexibility of archival), and
most of all the freedom of email – and have it work anywhere near as well in
practice as email does already.

(Google Wave showed promise... but it also introduced a ton of new complexity,
and given the pattern of messages "living" on the sender's server, it wasn't
clear how archival should actually work.)

It seems most people who advocate this notion that email is "broken" would
rather us all use some centralized, proprietary system akin to Facebook
messaging. This would be an unimaginable step backward for reasons that I hope
do not require elaboration.

~~~
zokier
Federated XMPP seems rather nice alternative if it were extended to support
asynchronous communications (eg servers would store messages when clients are
offline). Yes, it is XML based and XML is not fashionable these days. But
still I think it has potential if it were developed further.

btw Facebook messaging is partly based on XMPP so it's not completely
proprietary.

------
dsr_
Hi. This is the qmail-send program at yahoo.com. I'm afraid I wasn't able to
deliver your message to the following addresses. This is a permanent error;
I've given up. Sorry it didn't work out. : 72.1.169.10 does not like
recipient. Remote host said: 550 : Recipient address rejected: undeliverable
address: unknown user: "[list name]" Giving up on 72.1.169.10.

That's a pretty good error message. Most people glaze over as soon as they see
it.

Most people ignore the content of error messages as soon as they realize what
they're reading. An awfully large percentage of people ignore it entirely.

I don't know what to do about people.

~~~
pbhjpbhj
> _Recipient address rejected: undeliverable address: unknown user_ //

Hmm, it's okay.

It's a person masquerading as a computer masquerading as a person. This
doesn't really help IMO. Why can't it be

"Hi, this is the Yahoo! email support team. You sent a message to someone but
when we tried to pass it on the email address didn't work so we can't send the
email for you. Here's some things to try ... technical details and the text of
the email you sent are below [ _or_ attached]."

The response could also include info like that the domain is live.

~~~
davidcann
That sounds like someone at Yahoo is reading my email.

~~~
pbhjpbhj
Well the chances are slim because they cant read that much at a time.

Probably a computer at the CIA is reading it though.

------
zrail
Every MTA used to have their own way of reporting bounces. These days, the big
ones (at least by volume) seem to all obey RFC 3464[1], which defines a MIME
type and required and optional fields for delivery reports. Of course, the old
MTAs are still around and still delivering their bounces in a myriad of
different formats. The bounce processor I work on at work has about a dozen
different variants on "The message could not be delivered. There is a copy of
the original message/headers below this line". Those plus the RFC 3464-style
messages cover roughly 95% of what we see day to day. The other five percent
are so unrecognizable that we can't process them automatically.

[1]: <http://tools.ietf.org/html/rfc3464>

------
sneak
Please do no encourage admins to replace useful debugging output with useless
HTML shiny.

But, by all means, improve the text itself.

~~~
spoondan
You missed a crucial part of the post:

    
    
        Machines can read [these error messages] just fine, though.
        Here's an idea: machines shouldn’t slap us in the face. 
        They should help us along if they fail to do our bidding.
    

He's _not_ encouraging admins to replace useful debugging output. In fact,
it's the opposite: he's encouraging email client designers to actually _use_
the debugging output to let people know what went wrong. His mockup is quite
clearly of an email client showing users relevant recovery options based on an
underlying error message the client has parsed.

In other words, he's proposing that email clients treat MTA error messages
precisely as they already handle raw headers. Full headers contain useful
debugging information, but you aren't normally debugging.

~~~
ori_b
No, they can't. There's no standard format for these error messages that I
know of.

Maybe there should be, but that's a problem that has to be solved before the
shiny UI can be placed over it.

~~~
dfox
There is no standard format for the plain text part of bounce that your MUA
shows to you as it's just that: plaintext part to show when MUA knows nothing
better to do. RFC 3464 specifies format of additional MIME parts of message
that specify what happened in machine readable format. It's interesting to
note that it's exactly same mechanism as is used for sender-requested
notifications for delivered email in which case these messages are processed
by MUAs just fine.

------
jhferris3
Yes, the email system could be better. But for something that was designed
_decades_ ago it works surprisingly well.

Should somebody try to do some smart parsing of failure notifications to make
them more readable? Sure, I don't see why not. But don't pitch the baby out
with the bathwater here. The underlying system is not that bad.

~~~
jsprinkles
Actually, the underlying system is absolutely putrid, a sentiment that pg
agrees with based on his latest talk at PyCon. Mail is, unequivocally, the
worst part of my daily existence and you are kidding yourself if you're
calling it "not that bad".

Using pg's words, mail is a to-do list that anybody can put shit on, and I
don't have any control over who puts things on it. We live in an era of
technology that can do e-mail so much better, and replacing it must happen in
my lifetime. Everything, from the architecture to the messages themselves,
need an overhaul. The extraneous shit we bolt on to e-mail (SPF, DKIM,
PGP)...it's just making it a big, unwieldy mess.

~~~
vog
_> mail is a to-do list that anybody can put shit on, and I don't have any
control over who puts things on it_

That's a misuse of email on the _receipient's_ side, not on the system's side.
Email is about messages, not about tasks. Although there might be more
comfortable (i.e., more automatic) ways, there's nothing wrong with
maintaining a TODO list that is 100% separate from the INBOX.

For example, in my TODO list I describe _shortly_ (and in my own words) what I
consider to be important. Why should I waste my time scanning through the
INBOX everytime I want to check what needs to be done?

In other words: It's _me_ who decides what gets on my TODO list, nobody else!
I'm _in total control_ because I'm a free citizen living in a democracy, and
as such it's _my responsibility_ to organize my life. In particular, to decide
what to do (to-do) in my life.

If you decide to give other people access to your TODO list (e.g. by defining
your INBOX to be your TODO list), you're _voluntarily giving up control_ of an
important part of your life. Nobody forces you to do that. In fact, almost
every organization guideline strongly advises against that kind of nonsense.
(The most famous one being the "Inbox zero" series at
<http://www.43folders.com/izero>)

~~~
ssebro
I think that part of the idea is that reading a message, replying to a message
& all other things that you do with mail can be considered as "tasks".

------
anigbrowl
What annoys me about SMTP is that it's poorly implemented. Have you tried
sending a message to the postmaster at any large firm in recent years? Chances
are nobody even reads that. I clicked on the link to submit a tip to my local
news station last week; later I got a reply telling me that those mailboxes
didn't exist. The largest broadcast media organization in the Bay Area can't
even operate its own wesbite properly. So while SMTP is great on a technical
level, it's only as good as the people who administrate it, and system
administration is not as valued as it used to be because it's too labor-
intensive.

~~~
jodrellblank
Reminds me of the time I filled in a form on Microsoft.com and submitted it, I
think it got sent to an internal nonexistant address, was autoforwarded, and
then bounced to my email, containing the content of my original message and a
reassurance to the internal person that their email address wouldn't be
revealed to the outside world. It was.

------
hub_
I suggest returning the error in a .doc file so that people can read. At least
some.

And yes I get you idea, this is actually a RFC
<http://tools.ietf.org/html/rfc6522>

And the MUA (Mail Program) should just deal with it properly. But then you
have all these Bloatus Notes, M-SExchange and other broken system in the
wild...

~~~
kijin
.doc file? Why use a proprietary binary format when a suitably standardized
plain text format would work just fine? I don't even open .doc files from
unknown senders.

~~~
hub_
You obviously missed the sarcasm.

~~~
kijin
Sorry, sarcasm doesn't always come across clearly on the Internet.

~~~
whacker
Neither do email bounces!

------
pnathan
I generally prefer email to any other form of electronic communication. It's
sortable, archivable, easily formed into lists, decentralized, encryptable,
supports attaching images, well-known, implemented in a number of protocols...

While you could implement a shiny-for-the-laypeople status code parser, email
as a system works unbelievably well.

~~~
kijin
> _archivable, ... decentralized, encryptable_

This. I shudder every time someone mentions replacing e-mail attachments with
Dropbox/iCloud/etc., which somebody already did in this thread. With e-mail, I
am beholden to nobody; and if I pay some company to handle my e-mail, it's
only because I voluntarily chose to do so. I could drop that company at any
time, fire up my own mail server, and have a 5TB mailbox with 10, 20, 30 years
of searchable history if I wanted to. I can also access it with a whole bunch
of free and open-source software, from ESR's original fetchmail all the way to
the latest alpha of Thunderbird.

Every other messaging system that I've heard about, on the other hand, is
either held hostage to one company's proprietary platform or clearly not
intended for people who need to look up conversations they had 10 years ago.

So please, folks, by all means fix what's currently wrong with e-mail. But
please think twice before suggesting that everyone replace it wholesale with
some proprietary cloud-based TODO app.

~~~
aethr
I was one who suggested Dropbox as a replacement for attachments (although it
was really just an example, I don't even use Dropbox personally), and the
point I was trying to make is that you can either have flexibility, or you can
have standardization. The OP is suggesting we somehow standardize email error
messages, but this doesn't seem possible while email remains a free-form, very
simple protocol, which gets used for an infinite number of different purposes
by different people.

"Fix what's currently wrong with email" is so overly vague as to be basically
meaningless. Different people have completely different ideas about what is
"wrong" with email, and it is such an entrenched technology that you will
almost certainly never be able to make structural changes to it unless the
problem set it addresses is radically diminished.

~~~
kijin
I don't see any reason why something like SMTP error messages can't be
standardized if several major e-mail software vendors and related groups get
together and agree to implement, over a period of 5-10 years, an extension to
SMTP. Sure, it might not be as clean as rewriting the protocol from scratch,
but it would work for all practical purposes. A lot of features of HTTP that
we take for granted, like cookies and anti-clickjacking headers, were added
through a similar process. The problem is political and economic, not
technical.

I also don't see anything wrong with having a free-form, vendor-neutral,
decentralized messaging protocol that can be used for a million different
purposes. As long as the underlying protocol itself is reliable (e.g. it has
proper error handling) and extensible (e.g. you can add custom headers and
MIME types that other software won't mess with), innovators are free to
implement whatever functionality they want on top of that protocol. Heck,
Richard Stallman uses e-mail to browse the web! If you restrict the protocol
to what you think should be its only purpose, you're unnecessarily limiting
what other people can do with it. That would be like restricting HTTP to
hypertext documents only, no AJAX, no API calls, no streaming.

IMO, e-mail doesn't need to _become_ a TODO list or notification system. When
I say "fix it", I mean we should clean up old cruft from the protocol (e.g.
make it 8-bit clean) and make it extensible so that it is not too difficult
for interested parties to _implement_ a TODO list, notification system, etc.
on top of it if they want to. People will obviously disagree about how exactly
that should be done, but people disagree about everything anyway.

------
gnu8
You want to strip out all of the useful information needed for debugging mail
problems, and replace it with web 2.0 bullshit ajax and lickable icons that
won't render in most mail clients?

Sorry you couldn't send your attachment but go back to your blog and leave us
alone.

~~~
spullara
They are suggesting that the client generate those UIs based on the message in
the email, not that the message have that UI in it.

------
aethr
All mail systems have different rules for what is "acceptable", what is
perceived as "spam", etc. One mail server may allow attachments up to 10MB,
another 20MB. Some mail server somewhere may allow attachments up to 50MB but
will not allow attachments with a .exe extension, and due to the vast and wide
uses of email it actually is important for this flexibility to be available.

Given the amount of variablility around mail handling, relays, spam filtering,
etc, I'm not sure that it's entirely feasible to "standardise" error or
bounceback messages. RFC 3464-style messages get us part way there but it just
doesn't approach something as "nice" as HTTP error codes (which themselves
still aren't handled in a standard way by web browsers, they are handled on a
case by case basis, if at all, by web developers).

I think this problem is only going to approach being solved when some of the
more edge-case uses of email (such as attaching/sending files) are deprecated
by more targeted, more efficient services. For example, lets say everyone
started using Dropbox or YouSendIt to send files instead of attaching them in
emails. And alongside that perhaps a separate protocol such as iCal or similar
takes over the "meeting arrangement" space. Eventually the usefulness of email
may be narrowed down into a small enough band of "send and forget" messaging
that someone can actually, finally, come up with a better, more efficient and
usable alternative.

Until that happens, or something else equally radical, email is something
we're just stuck with, worts and all.

~~~
pbhjpbhj
> _One mail server may allow attachments up to 10MB, another 20MB [...] not
> allow attachments with a .exe extension, [...] important for this
> flexibility to be available._

 _[...] I'm not sure that it's entirely feasible to "standardise" error or
bounceback messages._ //

Well couldn't we use something akin to SPF - ie a TXT field in the domain's
DNS record with a shorthand for the acceptable receiving protocol in
operation, kind of an RPF.

So you'd have something like:

alicious.com TXT v=rpf1 deny-ext:exe,reg,dll max-size:15MB req-field:sender
ip4:xxx.xxx.xxx.xxx/16 req-format:txtonly req-other:spf ...

would tell the MTA (or the forwarding relay) that this address will likely
reject emails containing exe, .reg, .dll files; emails over 15MB; emails
without a sender field; emails apparently originating from servers outside the
given address range; emails not in text format; emails from domains without a
SPF; ...

Whoops got carried away there.

What would be the problems with this sort of scheme?

[FWIW there's already a milter called RPF but that's not what I'm proposing
here I'm talking instead about a policy to give to a sender to tell the sender
what you intend to accept so that they can choose to match your requirements
and avoid having the mail rejected. Obviously an MTA/MUA can choose to receive
mail that doesn't adhere to the policy if they wish.]

~~~
aethr
From a technical standpoint something like that could work to a limited extent
(enough to be helpful in some situations, when both parties are using modern,
up to date MTAs / mail clients). It still doesn't fix the error message from a
bounce, it's more like a hint to the email author to say "this email _might_
not be accepted for the following reasons: x, y and z".

Ultimately though, the inertia to implement such a thing on existing MTAs is
somewhere close to the difficulty of implementing IPv6. Except the need for
IPv6 is verging on dire, and not many people seem to be rushing to fix that
one. How will we convince people to act with any expediency over something
that is, at most, an inconvenience?

~~~
pbhjpbhj
The MUA would do the work up front, query the TXT field for the rpf and tell
yiu up front "I can't do that, $user wont accept attachments over 15MB". Then
as the OP suggested the MUA can offer to zip, upload and link, or whatever.

------
justauser
Status codes:

[http://www.iana.org/assignments/smtp-enhanced-status-
codes/s...](http://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-
enhanced-status-codes.xml)

<http://tools.ietf.org/html/rfc3463>

I'm guessing most e-mail client providers just haven't found a reason to
prettify the responses. If anyone is up to the task, then can you explain
PGP/GPG so the entire world can also understand and use that?

------
jlawer
Client side parsing won't be as reliable as you expect.

SMTP bounce codes are notorious for changing on a per domain basis whenever
someone feels like it.

I work with StrongMail supporting their Email Marketing Platform and their
solution is to have a central database that is updated regularly for all major
ISPs to basically be able to have a conversion process between the SMTP bounce
codes and the 30 classifications for mail failures StrongMail groups failures
by (such as User not existing, blocked due to spam complaint, service
temporarily offline, etc). This conversion process has to be updated regularly
to keep up with changes.

Server side generation of multi-part NDR reports including HTML email could be
a solution, i am not sure if it violates the rfc to send NDRs in multi-part,
it could perhaps need an amendment.

But if you want to show this then your in luck since all the major SMTP
servers besides exchange are open source, go produce a patch set for postfix,
sendmail and qmail.

------
Tichy
My main problem is that spammers hacked the "sender unknown" error message
long ago and use it to spam me (they send mails pretending to be me, so the
error message gets delivered to me). So I had to filter them all away.

I wish Thunderbird's spam filter would check if I actually sent the message
the error message refers to.

------
emmett
I really like this idea.

You can apply this to every transactional email, not just to bounce messages.

Off the top of my head:

\- Vacation auto-replies

\- Receipts

\- Error reports (ok, that might be a bit coding specific)

------
Shank
Part of this would have to do with the fact that mail servers are never really
updated with new "features."

\- Sendmail was first released in 1983 initially. \- Postfix was released in
1997 initially.

Though both are getting maintenance updates, and new processing features, none
have improved the actual error path system. Maybe someone should write a
better MTA?

------
4ydx
It is a perfectly reasonable error message that mail clients should try to
beautify. The message itself is absolutely necessary, but should perhaps be
cleaned up by the client. All of the calls to write a new "protocol" by the
posters on that blog make me wonder if they even understand what the word
means.

------
wazoox
This could be a good idea to improve bounce management from the client, if
only for the reason that 80-90% of the world population simply don't
understand english at all -- all bounces AFAIK are always in english, for a
start.

------
waratuman
Its old, it doesn't look nice. But it is surprisingly reliable.

------
xpaulbettsx
Coming up with ideas to fix Email in 2012 is like coming up with ideas to fix
IM - why bother?

~~~
pyre
Why not? If any of them turn out to be good, then we've improved something,
haven't we?

