
Why it’s so hard to innovate in the e-mail space (2014) - mathouc
https://medium.com/@collinmathilde/why-its-so-hard-to-innovate-in-the-e-mail-space-9874e08e3426
======
Animats
It's hard to make money in that space, yes.

I've considered merging mail and messaging, so that email protocols are used
to do the things people do with instant messaging, Telegram and such. The
first step is to get away from queuing mail.

A first step is a new kind of mail forwarder, one for machines that don't have
local mailboxes (which is most non-mail servers). It acts as an SMTP server
and client. When an email comes in, and the sender reaches the end of the DATA
section, the forwarder checks its forwarding list to see where this mail is
supposed to go. It opens a SMTP connection to the destination and sends the
email. The incoming SMTP connection is held open while this is in progress.
Any problems are reported back to the sender immediately with an SMTP status
code. There are no bounce emails, no queuing, and no possibility of joe-jobs.
All mail is handled immediately or rejected immediately. (This only applies to
single-address messages, and only to "To" addresses. Everything else is bulk.)

The next step is an IMAP server which has the same immediate orientation. When
it gets an incoming email, it sends a push notification to any connected
clients, while holding the incoming connection open. If at least one client is
responding, a SMTP success status is returned. Otherwise, you get some status
such as 251 (User Not Local, Will Forward.)

Then, a threaded email client. It would help to have the convention that an
email with subject but no text, text but no subject, or text the same as
subject is displayed like an instant message.

All of these can be done independently, and are backwards-compatible. If you
have all of them, email is equivalent to instant messaging - you can tell if
it was delivered, it goes through immediately, and it works like a
conversation.

Of course, there's no big-money startup in this.

~~~
zombition
I tried some similar things when making Zombition, but I found that blurring
the line between email and instant message mainly results in a more
complicated UI and more details for end users to remember. For example, with
instant messages you generally have some status information about whether or
not the other user is online, whether or not they may have read whatever you
just sent them, and whether or not they're typing a reply. If you want to add
that into a email interface, how do you indicate to your end users that
another user will never have associated status information because their
account is on a server that is incompatible with the method you proposed?

Plus the behavioral differences between email and instant message didn't blend
well: was I going to wait in a mixed email/instant message conversation for a
response, or was I going to move on to the other conversations in my inbox and
send a reply or archive each so that I could move on to doing other things?
And if I wanted to temporarily leave an instant-message-like conversation so
that I could deal with some more email-like conversations, would the person I
was chatting with know whether or not I would return?

Eventually I stepped back to look at what additional value a user gained by
blurring instant message and email in this way. Fundamentally they're both
asynchronous text-based communication, and the things you can actually do with
them aren't strikingly different. The biggest difference is in the
expectations that a user has for each.

Anyway I still wanted to "innovate in the email space", so I decided to go a
more extreme route, and I made my own pull-based protocol (superficial
similarities to IM2000 -
[http://cr.yp.to/im2000.html](http://cr.yp.to/im2000.html)) that retains
backwards compatibility with email while adding abilities like stateful
plugins (JavaScript apps/widgets) that can share data through the new
protocol, editing messages after they're sent (SMTP recipients receive the new
version after the update), and inline replies to parts of messages to enable
nested conversations. If you're interested, I posted it earlier this afternoon
here
([https://news.ycombinator.com/item?id=10693535](https://news.ycombinator.com/item?id=10693535)).

~~~
zAy0LfpBZLC8mAC
> inline replies to parts of messages to enable nested conversations.

Just to be sure: You know that that is really easy with email as it existed
until about 20 years ago, including BBS networks, usenet, etc., and that it is
still trivial with good email clients? That was a solved problem that made
email really efficient, until some clueless developers and users came along
and disinvented (is that a word?) the solution in order to make email easy to
use (or so they claimed).

~~~
zombition
Actually I didn't know; thank you for telling me! (That also makes me realize
that I could likely do it in a much easier way than I currently am...) What
are the email clients that allow that kind of behavior?

~~~
zAy0LfpBZLC8mAC
Well, essentially, all of the traditional text-only clients do (such as Mutt,
which is what I use), I think Thunderbird does as well, beyond that, no clue,
but there probably are others.

Traditionally, if you chose to reply to an email, the mail client would put
the mail you were responding to into your editor, with > marks in front of
every line. The only socially acceptable way[1] to write your answer was to
insert your responses in between those quoted lines, without > marks, and to
delete everything of the original mail that was irrelevant to establish the
context for the reply.

That's how easy it is.

Usually, a good mail client would also color quoted lines so that it's easy to
see and read only the response lines in a received email, so you'd only have
to read the quoted text if you weren't sure about the context.

Clueless developers and users around the turn of the millenium then somehow
didn't understand the purpose of providing you with a quoted version of the
email you replied to, and started putting the reply above the quoted mail,
thus ending up sending a copy of the mail they just received back to the
person they received it from ... and nobody seemed to notice how idiotic an
idea that actually is, and so it became sortof the new norm in large parts of
the internet to attach large blobs of completely useless and often close to
unreadable text to every mail, up to the point where some people even invented
justifications for the behaviour that mostly tend to describe how this
"feature" allows them to work around the lack of proper thread handling in
their mail clients.

Obviously, easy quoting traditionally relies on plaintext only mails with
fixed (short) line length, but format=flowed encoding has since been invented
(the original RFC is from 1999) to accomodate screens and windows of variable
width while maintaining backwards compatibility with old clients.

[1]
[https://www.netmeister.org/news/learn2quote2.html](https://www.netmeister.org/news/learn2quote2.html)

~~~
zombition
Ah right. I was meaning something more like this so that you can skip the
quoting business entirely and have the visual hierarchy match the conceptual
hierarchy: [https://s3-us-west-2.amazonaws.com/zombition-
static/20151208...](https://s3-us-west-2.amazonaws.com/zombition-
static/20151208/inline-reply.gif)

> Clueless developers and users around the turn of the millenium then somehow
> didn't understand the purpose of providing you with a quoted version of the
> email you replied to, and started putting the reply above the quoted mail,
> thus ending up sending a copy of the mail they just received back to the
> person they received it from ... and nobody seemed to notice how idiotic an
> idea that actually is, and so it became sortof the new norm in large parts
> of the internet to attach large blobs of completely useless and often close
> to unreadable text to every mail, up to the point where some people even
> invented justifications for the behaviour that mostly tend to describe how
> this "feature" allows them to work around the lack of proper thread handling
> in their mail clients.

Yes, I think the rationale must run something like "What if user A deletes the
entire conversation they were having with user B, and user B replies to the
deleted conversation, but user A doesn't know what the reply is about because
they deleted the entire conversation, have a terrible memory, and are too
embarrassed to ask user B to explain what's going on? Let's just send the
entire conversation as quoted text every time."

~~~
zAy0LfpBZLC8mAC
> Ah right. I was meaning something more like this so that you can skip the
> quoting business entirely and have the visual hierarchy match the conceptual
> hierarchy:

Well, if it's supposed to be compatible with all the HTML crap out there, and
even most MUAs' broken encoding of plain text email, it might be hard, but in
principle, I'd think this should actually be rather simple:

Simply specify an algorithm that defines how to segment a text/plain body into
paragraphs and how to thus generate MIME object-scope IDs per paragraph (by
simply numbering them) and use that same structure to provide UI elements per
paragraph. Then, if the user writes a per-paragraph reply, generate a
multipart/alternative body, with one text/plain part in traditional quoting
style, optionally with format=flowed, and one alternative application/fancy-
paragraph-foo part which contains in some sort of serialization format
references to the message+paragraph IDs, that paragraph's text, and the
corresponding reply text. This way, a receiving MUA that doesn't support
application/fancy-paragraph-foo will display an ordinary quoted plaintext
body, and also, if a communication partner uses different MUAs, you can still
reply to an email sent using an MUA without application/fancy-paragraph-foo
support and still have the feature work if the reply is then read in an MUA
that does understand it. Also, including the text of the original paragraph
makes sure you can still display a message sensibly when the displaying MUA
doesn't have the mail it's in reply to - you'd just have to be careful to not
misrepresent the authorship information as reliable.

> What if user A deletes the entire conversation they were having with user B,
> and user B replies to the deleted conversation [...]

* g * (is there any way to escape asterisks to their literal meaning?!)

Well, the rationale I've heard is that it's easier to make sure new
participants in a conversation can read up on what has happened so far ...
which obviously would be solved far better by attaching a thread of
message/rfc822 attachments when adding a new participant that the recipient
could navigate using their MUA's usual UI instead of having to read a close to
unreadable unstructured mess of emails.

------
CM30
Nice article. But I think there are a few other things they missed out that
make building an email client so difficult:

1\. Consistency. Okay, this is always going to be a problem, since rendering
anything more than plain text in emails is a minefield that makes the bad old
days of Internet Explorer vs Netscape Navigator look like a utopia. But people
still expect their Amazon/eBay/Google/whatever account/confirmation emails to
look decent, so you have to figure out a basic HTML parser as well. And then
try and make it somewhat work with people who are designing their messages
with stuff like Outlook's Microsoft Word engine in mind.

2\. Spam. I'm surprised this wasn't really mentioned in the article, but it's
arguably the biggest issue any email provider, client developer or email
server software maker has to deal with. You have to make sure your software
can figure out what messages are unwanted, then either send them to the spam
folder or delete them. Most startups don't have to figure that out, since
their systems are closed and spam prevention can be done on sign up. Not so
much if you're building an email client where any Tom, Dick or Harry can send
messages to anyone with no real way to verify if they're human or not.

So yeah, few other issues.

~~~
xs
Unless you're building an MTA from scratch, I feel like the spam problem is
solved by addons the MTA has, like SpamAssassin for Postfix.

~~~
feld
If I ever get a chance to revamp an email system I will probably use rspamd
instead. Those "add-ons" are painful.

[https://rspamd.com](https://rspamd.com)

[https://rspamd.com/rspamd-slides.pdf](https://rspamd.com/rspamd-slides.pdf)

~~~
baudehlo
Check out Haraka, built by an old SpamAssassin developer (ie me). It's all
that and has custom anti-spam features like the Karma plugin too. It's in
place at some large installs (eg craigslist, who went from 20 postfix servers
to just 7 Haraka servers).

~~~
feld
I'll look into it, thanks!

------
xs
I'm trying to build an email platform too, yes it's hard. But I thought it was
only because I'm a beginner and 1 man team.

Some of the reasons why I think it's hard:

* I don't think security was a big deal to many of the mail applications when they were being built. I tried and gave up so many times trying to find a strong hashing algorithm that both dovecot and ruby supported. There was practically no documentation on this and it felt like I was venturing into territory never before seen.

* Parsing emails to be display correctly seems impossible. Sometimes emails have "<br>" in them and sometimes they have "\n". But what if the "\n" doesn't mean newline but it's just what someone wrote in the email? They come in a variety of mime types and formats. It sometimes seems impossible to do this.

* Not everyone uses the RFC standards! I thought the RFC said that a subject can only be 78 chars long. Yet I get emails all day long that go way beyond this which cause major problems in my code. AM I THE ONLY ONE AROUND HERE THAT CARES ABOUT RFCS?

~~~
feld
Look at
[http://archiveopteryx.org/badmail/](http://archiveopteryx.org/badmail/) for
your storage backend.

Strict RFC implementation, email normalization. Bad emails won't kill your
email client 30 years from now.

------
tmail21
At a very high level, email is primarily used for discussions. With this view,
successful innovation in the e-mail space requires innovation in the nature of
discussions themselves. Tinkering around with better usability or feature
improvements will not be sufficient to overcome the massive inertia of
'traditional e-mail'.

We are a startup that just launched
([https://tmail21.com](https://tmail21.com)) that aims to rethink the
discussion itself. In our view, one of the major problems of e-mail (and chat
for that matter) is that the discussions are not goal-oriented. So,
discussions/threads can meander around with no outcome or accountability.

So, we enhance the notion of discussions to be goal-oriented.

Now, once one has the notion of goal-oriented discussions, a natural next step
is to evolve a goal-oriented discussion into a Lean Process (which is
basically a goal-oriented discussion with more structure) . Examples of a Lean
Process might be a Blog Article process, a Product Deployment Process, a
Feature Design Process, an Issue Escalation Process etc.

We've done all this in an email-LIKE interface. I guess we'll leave it to
others to decide whether this constitutes innovation in the e-mail space :)

~~~
bitcuration
Pretty good endeavor that you essentially created a BPM application, that is
only as if email used as a BPM platform which is seen prevailing in corporate
than other enterprise tools (SAP, Oracle etc.)

I actually propose instead of re-inventing email, create addons to extend what
email message can be used/read/interacted with. For example, high school
teaching today has been reliant on gmail for offline homework assignment and
discuss between teacher and kids. There are some tools in that space
aggressively exploited by schools to conduct their "innovative" teaching
efforts.

Gmail, or microsoft outlook would be the best go-to destination for the
majority of who can afford an in-house IT team, or shadow IT in large
enterprise who's tired of the lack of evolution from SAP, IBM etc.

Still, Gmail and outlook are slow in adopting or providing email as an open
extendable platform for BPM application purpose.

Then there are everyday use of email other than BPM, notification, online
purchase receipt, offline discussion, even transmitting files, photo etc. via
email are somewhat common with email client.

The longevity of email is exactly the lack of rigor of the intended use of it,
BPM or not, email is flexible to conduct any discussion to an open goal, or
without a goal.

To understand email better, one has to compare email to SMS, and social media
along with how mass used these tools.

~~~
tmail21
Thanks for your insights bitcuration. I agree that email is flexible enough to
do things like goal oriented discussions, but it is just so painful. The best
one can do is fling attachments around and hope to get all the versioning
right and avoid inadvertent branching.

TMail21 does other email-like things like transmitting files, notifications,
full search etc.

It also does interesting things like giving every thread a unique tracking
number, which allows it to be tied into the broader enterprise ecosystem. The
idea is that just like the URL transformed the Internet (i.e. the web),
Tracking numbers can transform 'email'. Now, TMails can be referred to from
Chat, Voice, IM, Apps, Spreadsheets etc.

Another thing we do that is very hard to do in email is things like Certified
Mail, Certified Forwards, Transactional Guarantees, Certified Diffs, Non-
repudiable audit trails etc.

All of these capabilities are however means to an end to make TMail21 the
first true BPM tool for regular business users (rather than for coders and
'process analysts)

------
vonklaus
There are tons of reasons why this is so hard. Leaving aside all of those
there are 2 that I see as the biggest:

* A 100x improvement over current email is needed for a switch.

* It has to be 100x cheaper than the current product.

While you can't multiply by 0 and I meant that facetiously, users won't pay
for the software or only niche consumers. Which is why email providers that
get paid focus on things like security, collaboration and and backup. These
products are essentially 90% other things and 10% email. examples:

* dropbox * slack * telegram * silent circle * sms

obviously I am not going to name everything in the space. I can't think of a
company that has email at it's core. maybe 'hushmail' or some small encrypted
mail providers but what profitable business is at it's core an email provider

~~~
AlexB138
MailChimp? Constant Contact? I may be wrong, I'm not terribly familiar with
either, but I believe they're largely focused on email.

~~~
bradlys
Mailchimp and such aren't for everyday emailing between you and me. They're
for emailing large lists of users one email a day or week or whatever and
using tracking information based off referral links users click on in the
email.

------
hackuser
Why are messages separated by transport? That is, why are messages sent
to/from me via one transport (e.g., SMTP) read and composed in one
application, and those sent via another transport (e.g., SMS, Twitter) read
and composed in another? Why, as a user, do I give %#@%! what the underlying
transport is?

I think the focus on a particular technology (SMTP) rather than a service
(messaging) is why email clients don't advance. Bizarrely, three other
transports often are integrated into email clients: NNTP, voicemail, and fax.
I can only guess that it's because of a completely arbitrary reason: They are
older.

Probably chat clients should also be integrated. I should be able to send my
contacts an instant or time-shifted message.

~~~
aetherson
Because the different transports have vastly different abilities and
limitations that are a source of frustration if you attempt to obfuscate the
differences between them and aren't served by a unified UI?

~~~
hackuser
Could you be more specific? For example, comparing SMS and SMTP, the two most
prominent transports?

~~~
zombition
SMS is used to send tiny chunks of text, and SMTP can be used to send message
content in HTML format and rich text in addition to plaintext. I haven't
worked much with rich text format, but HTML format for sure would be
relatively impractical via SMS (XML adds a lot of characters, so you would
have a lot of little chunks to send if you wanted to send HTML over SMS).

~~~
hackuser
I agree with that. I don't mean that I want to send my email messages visa
SMS. I mean that I want all my messages sent/received and read in one
messaging client, whether they are transported via SMS, SMTP, etc. If the HTML
message should go via SMTP, I expect my software to deal with that. I want the
content; the transport is an obscure detail for developers to worry about
(speaking as a typical end user).

~~~
toomuchtodo
You're asking for a magic genie content router that does not exist. It does
not exist because _its a hard problem_. Not only would you need to
interoperate with open standards, you'd have to attempt to integrate with
unwilling participants (Facebook, Google Hangouts). Are you prepared for that
level of pain?

~~~
Ntrails
You also have to consider that facebook already do this. They tried to wrap up
email, messages and an SMS replacement on your phone into a single package
where they own the whole thing.

I look at iMessage and I think it works pretty perfectly, SMS when no internet
or to a non apple user, iMessage whenever possible. What I want is for
whatsapp (and I suppose facebook and plausibly email) to do the same all in my
single messenger application.

But. Ignoring the need to get the tech owners to consent to this new
"trillian", I'd also have to trust the app itself. Good luck making money on
that.

------
mikhailt
It will be easier to build a new email spec that's more feasible/modern with
security built in mind than it is to build a sustainable email client that
tries to comply with various email services (GMail is the worst offender for
not being consistent with IMAP spec).

Mailmate is my fav OS X client that focues more on power user features but it
has a boring classic interface which doesn't bother me at all. Despite having
integration support with SMIME/PGP, Markdown reply, powerful search features,
and so on that makes me productive, the only thing I hear when I recommend
this to other people is that its UI is ugly. :|

~~~
akvadrako
It's not as ugly as Thunderbird or a web-app. At least it follows most of the
OS X conventions. Anyway the existence of Mailmate shows how the premise in
the article is totally wrong - it's not hard to create a decent modern email
client - it already exists and only takes one developer.

A better title would be: Why is it so hard for users to use a good email
client?

------
grayclhn
Isn't this taking a somewhat narrow definition of "the e-mail space"? Slack,
Twitter, Facebook, Snapchat, etc. all seem like successful innovations that
lots of people use to communicate instead of email.

------
unsignedint
The biggest problem is that E-mail demography tends to be broader than many of
the other services.

Some of E-mail users are very conservative and specific about how they handle
their E-mail. (Some of which are very risky behavior, which I actually tried
to correct by explaining why that's bad idea, but ultmately given up...)

There are demography between novice and expert. Novices tend to not care about
changes so much; some even won't notice changes, as long as features are
somehow accessible. Experts may have some particular tastes, but they'd be
quick to find solutions to change things around when they have to. But people
in between tend to be very vocal about any changes and they can't (or not
willing) to find and accept "solutions" \-- they will be the first one to
complain when UI elements like buttons shift around the screen. Unfortunately,
as universal E-mail gets, a lot of E-mail users fall under category. (Though,
in different ways, "experts" can be very particular about how they handle
things can be very stubborn when there's no good reason presented to change --
but it's a bit different context.)

I can't even imagine how tough it will be to migrate some of people I know to
anything other than what they are used to.

------
lassejansen
As the article mentions Unibox: If somebody is interested in giving the beta
version of Unibox for iOS a try, send me an email:
lasse.jansen@eightloops.com.

Unibox groups emails by person. It's different, but it really helps you
staying organized.

[https://www.uniboxapp.com](https://www.uniboxapp.com)

------
J_Darnley
Because. It. Doesn't. Need. It. Email isn't broken.

~~~
luxpir
Not necessarily broken, but certainly outdated. What do you think about spam,
encryption, attachment sizes?

Imagine being able to publish your email address online without worrying about
spam; even in your profile you've been forced to obfuscate your own a little.
Or to know that every message you send can only be read by the recipient and
nobody in-between - without having to explain PGP to non-techs. Or being able
to send a 50-100MB file without resorting to third party or public hosting -
any of those features would be a bonus, in my view.

Not broken, but can be improved, don't you think?

~~~
J_Darnley
You make a good point. Why would I obfuscate like that? My email is out there
in git logs, mailing list logs, lists stolen from other websites, lists bought
from other websites.

I can't see that encryption will ever work end-to-end as we want. People are
too stupid to learn most things. Technical solutions rely on big corporations
that we trust to do the Right Thing(TM). The Right Thing(TM) is impossible
from US companies and probably impossible from European companies.

Attachments. I don't remember the last time I sent a large one. As far as I
know most of the size limitations are artificially implemented by companies
that don't want to spend that much on storage. Other limitations such as
Gmail's block on executables is because people are stupid and will run
everything they get sent.

People will never not be stupid so we can never have nice things.

------
merih
Such a great timing for this post, as Dropbox announced that they are shutting
down Mailbox: [https://blogs.dropbox.com/dropbox/2015/12/saying-goodbye-
to-...](https://blogs.dropbox.com/dropbox/2015/12/saying-goodbye-to-carousel-
and-mailbox/)

------
11thEarlOfMar
E-mail as a paradigm has been optimized about as far as it can be. We need a
new paradigm. It shouldn't look like e-mail.

Google Wave was an attempt, but the chemistry wasn't right. Someone one will
find the right mix...

------
rrggrr
Mixmax. Unrollme. Sanemail. Zapier mail parser. There is more innovation
taking olace than many realize.

------
tsunamifury
This isn't thats complicated:

Because email is dying and is already mostly optimized.

~~~
JustSomeNobody
Email is most certainly not dying.

~~~
tsunamifury
Yes it most certainly is on a secular decline. It is no longer growing at
rates that are keeping up with the online user base. New internet users are no
longer even setting up email accounts.

Email is dying. Its not dead, but it is no longer the go-forward internet
communication platform.

~~~
abeing
> New internet users are no longer even setting up email accounts.

What percentage of new internet users do not have any email accounts? I
haven't been able to find any reliable data.

~~~
oh_sigh
Seems like a strange claim, considering many sites that new internet users use
will not let you sign up without an email account

~~~
abeing
I agree. I'm skeptical and would like to see what data might support that
claim. I've not been able to find any myself.

~~~
tsunamifury
1.2 Billion Whatsapp users with phone number only

800 Million Messengers users as phone number only

20+ Apps with 100m + users as phone number only.

Together these PN based winners outstrip the combined usage of almost all web
apps. I mean you are all downvoting me -- but its kind of funny how out of the
loop the thinking is here.

------
wodenokoto
Why do people blog about what their company is building without so much as
mentioning the name of their company or product or even a hyperlink?

In the author profile at the very bottom of the page, in small letters, there
is a URL (but not a link) to

[https://frontapp.com/](https://frontapp.com/)

and the email client the author works on is apparently called Front. But why
should her readers ever care about that?

~~~
mathouc
You're right. When I wrote this article I was more interested in replying to
Des Traynor's article than promoting our product. I've added a link (even if
I'm not sure people care)

~~~
horsecaptin
Front is a great product. It lets us manage emails and SMS support together!

