Hacker News new | comments | ask | show | jobs | submit login
Email made me miserable, so I decided to build my own email app from scratch (slate.com)
273 points by dmbaggett on Dec 14, 2015 | hide | past | web | favorite | 159 comments



"Maybe the dream of an email-free future isn't dead; maybe it just means a future in which email is merely a sliver of our communication rather than the whole pie."

Email is victim of its own success, like cars & highways in America.

Highways are not bad per se, they were and are an incredible revolution. But too much of them means the value we collectively subtract constantly diminish. We need alternatives: walkable neighborhoods, bike lanes, tramways, trains, bus, etc.

It's the same with email, we need alternatives. Chat services like Slack & HipChat are good examples for internal communication.

Email's decentralized nature and ubiquity mean it's the perfect tool for external communication. I don't see why this would change.


Last week I did a visualization of all email clients in time from '79 to now (http://email-apps-timeline.missiveapp.com). It shows that there is a lot happening in the email space right now, even if the technical barrier to entry, as the author implies, is so high.


Great chart! Some of those new email clients look great. I wonder if any of them will become solid and popular enough.


I am still waiting for the application that kills email and the need for clients..


Given that the set of folks that are interested in and capable of making widely used end-user communcations software, and the set of people who are not interested in making walled gardens appears to have no overlap, you're going to be waiting for approximately forever. :(


ONE DAY MY TIME WILL COME!


:)

Just as long as part of your glorious future is a fully open and Libre protocol, federated servers, and -at a minimum- feature parity with email I'm 100% okay with it. :D


The barrier to entry isn't high at all. The author was able to create an email client in 3 weeks. That's not a lot of time.

Yes, making a solid GUI app will take a year. But that's the case with any solid app, and I doubt email is significantly harder than other productivity apps.


It impossible to overstate how wrong you are. Read some of my comments in this thread.

One of the absolute worst mistakes you can make as a developer is to assume that what you don't know or understand must be easy. That's the path to ultimate project doom -- don't do it.


I never said that I thought it was easy; I just don't see the email-specific barrier to entry. Syncing is hard in general, dealing with legacy systems is a pain, parsing poorly compliant file formats is annoying... But you have that everywhere.

What do you expect? That writing an email client should be as easy as cobbling together a map view with Google Maps? If that's your point of reference, then yes, there is a barrier to entry.

In another reply you said that your MVP took 4 years; but looking at your website I see something that tries to compete with Outlook. That's hardly what I'd consider an MVP.

I'm pretty certain that I could develop a solid email app in a year. Of course, it wouldn't do everything that existing mail clients do, and it wouldn't be compatible with every email provider, and it would probably be limited to a single platform. But it would do some things differently enough that it would be worthwhile for a few people.

If you think there's an insurmountable barrier to entry, you need to take a step back. There is absolutely no need to compete with the best existing clients on a feature-by-feature basis to enter a market. Focus on a small niche, and go from there.


> I just don't see the email-specific barrier to entry. Syncing is hard in general, dealing with legacy systems is a pain, parsing poorly compliant file formats is annoying... But you have that everywhere.

Have you ever written a production grade email client?


Of course not, otherwise I wouldn't have written "I'm pretty certain that I could develop a solid email app in a year".


Of course you haven't.

So you're claiming that you don't see the email-specific barrier to entry. Which is to say, you're arguing against it. While people who have authored such software are claiming the opposite.

Yet you appear to have zero willingness to entertain the idea that those who have come before you have learned what you don't yet know?

Very arrogant.

There most definitely are email-specific barriers to entry. No reason to list them out again as @dmbaggett (among others) have already been kind enough to give up their time doing so (for the benefit of all to learn).

Not buying what you're selling. So I'm sure you'll continue with your arguing and snap back with a terse reply.

Carry on.


I'm sorry if I offended you. I felt that my argument was discredited by appeal to authority.

I absolutely believe that the difficulties dmbaggett lists are real. I've experienced some of them myself on the few occasions that I've written code that integrates with IMAP and POP3. I don't assume that I could find a solution to all of them in a short time.

But here's what I think: These difficulties are not "a barrier to entry". You don't need to solve all these problems to enter the market. You don't need to compete head on with the best competitors to enter a market. You don't need to make an email client that works for everyone.

You just need to focus on specific audience, and solve a problem that no-one else does. If you do that, people will use your product even though it falls short in other regards. And you can leave the hard parts to the big companies.


Your points are valid, no need for them to be blindly discredited. You obviously have real world experience.

The key here is your approaching the problem from a "here's what I think" standpoint and not a "based on the past 5 years of me doing this" standpoint.

Again, it boils down to no having learned, what you don't yet know.

More thoughts...

I do have to say, though, that integrating email features into software (not an email client) these days can be trivial. For example, Rails makes a lot of it trivial.

But that is a very poor litmus test.

The difference between that and a full blown, production level, narrowly focused, tailored towards a specific audience, solving problems that no-one else does email client... are worlds apart.

You have 2 choices today, third parties (Gmail API, JMAP, Nylas, etc...) or diving into a raw IMAP connections (as an example).

The former provides general purpose layers, abstracting the complexity (to some degree). The disadvantage being they are general purpose, features everyone has.

The latter provides the flexibility to create what has not yet been created. And/Or backfill what the general purpose solution lacks (typically a lot).

In general, your approach to software, products and an MVP is spot on. Obviously you have experience doing this.

The point that I, and others make, is that email does not fall cleanly into that same approach. Why? For all the reason already listed plus a TON more not listed.

This is where you are unwilling to be open minded, and as mentioned previously, it's a classic mistake seen many times before.


Wow, that's great. One immediate bit of insight from the top right of the chart is how the modern email client is almost universally an iOS app, with other platforms if you're lucky.


Agreed...but I wonder if that's because ios folks tend to be more interested in products with more design built into them, in other words there is more of a market for a wider variety of products...? Or, if maybe non-ios users tend to think of email as a utility and figure "design-y" products for such "mundane" things as mail are "meh". Mind you, that's not my opinion, but just curious myself.


really nice, thanks! another client for your timeline: https://en.wikipedia.org/wiki/Minnesota_Internet_Users_Essen...


Awesome vis !


As far as I'm concerned chat and email are essentially the same thing: messages that get sent from an individual to a group of other identified individuals (as opposed to a broadcast platform where you don't have to identify all the user in advance). There's nothing to stop you making a UX for a mail app that feels like a chat app, or for a chat app that feels like a mail app.


Here's one difference: Chat is interactive and email is time-shifted. Users may use them other ways, but those are the fundamental designs.

In fact, I think many problems people have with email boil down to response times. If there was a way to senders could designate how soon they need a response, recipients would be face fewer interuptions and deal with email in batches when it fits their schedule.

Email has the priority field, but so few use it that you can't rely on recipients being aware of it and there are no social conventions about its meaning.


I don't think there's really anything fundamental about them that means that chat has to be interactive and email has to be time shifted, I know many people who deal with chat messages in batches much later than they're sent, and people who will respond to email within seconds. I think that's a ux choice, just like encouraging long vs short messages.

I do like your idea of designating a response time, but it probably requires too much from the sender. It might be that you could train rules on the client that could do a good job of working out how quickly things should be responded to so your client could batch appropriately.


I only do email at 09:00 and 15:00 at the moment, I've never had anything bad come of it. Honestly, if something is time-sensitive high-impact, sending an email in a fire and forget manner is close to a firing offence. That's just not an acceptable way to communicate urgency.


That works for some professions/working environments. I did that when I was working in an office and my entire team was in one 2x6 cubicle row. Email becomes much less important. Chat was nearly nonexistent. Now as a consultant working from home, chat is my lifeline (sometimes we even use IM while we're on the phone with each other to exclude some phone participants from a side conversation) but email is everything.

I have clients who aren't on my company's chat system and can't call me because I'm on the phone all day. Chances are if they call I can't pick up. Then there are the client's vendors who will not call me for any circumstance, because that's not their place. They're not paying me, and I hold no control over their wrk. But we'll email back and forth.

Our email system has no "urgent" flag, it's up to you to read them all as they come in and prioritize properly. If you're lucky, you'll have a PM who will spot an urgent email before you do and ping you on chat if you're a few minutes late.


On one hand I agree. On the other hand, I've had clients who did not agree: Guess who would have been fired!


Not really. I sometimes send chat messages when the users are offline knowing they will be read later. Likewise I have email chats where we respond more or less instantly (when my friend doesn't have a chat client). Its a bit clunkier to use, but works.


Eh, chat can be time shifted as well. My team regularly posts chat messages to Flowdock even when I'm not online (and I do the same). It doesn't get overloaded because the # of users are limited to a small team.


Maybe I'm a dinosaur all my experience with Chat comes from IRC in the 90's (which was horrible - net splits anybody) maybe it has gotten much better now. I wouldn't know I use email heavily today. Chat - rarely.

The key difference to me is that Email is broken down into discrete messages by sender and subject it is easy to archive retrieve and resend (forward) those messages with additional attachments if need be. If you can do that with Chat now great.


A chat app that happens to speak SMTP or whatever.


> Highways are not bad per se ...

To pick a nit: in the US, neither the Interstate Highway system nor the State Highway systems have much to do with car-centric design in cities and neighborhoods. [0] I used to live in a vaguely large-ish city that had both a State Highway, and an Interstate Highway that ran through it. These were two four-to-ten-lane [1] roads out of many, many, many other roads of widely varying widths and lengths.

The city was car-centric not because of the two highways that ran through it, but because the city managers never, ever made alternatives to private car ownership a priority of any sort.

[0] I mean, other than vaguely encouraging individual car ownership by making it relatively easy to cross large distances in an automobile. But, both highway systems play a big part in making inter- and intra-state shipping economical. Not only do we derive a huge amount of good from this property of the system, properly funded and designed public transit systems can also take advantage of these roads to enable city residents to ditch their cars.

[1] The Interstate varied in width, depending on where you were. Once you got out of the city, it was most frequently a divided four-lane road.


Seems to me you could create an email client that works a little more like Slack but not as ephemeral.


I think e-mail now in 2015 is like the fax machine was in the late 80s early 90s useful but rapidly becoming obsolete. Sure it will hang on for some reason just like the fax machine has but everyone else will have moved on.


The most remarkable thing about this article to me was that a journalist with moderate programming ability was able to build an email client. I think as coding becomes more accessible we'll start to see truly novel approaches and applications. People say C succeeded because of UNIX, and JavaScript succeeded because of the web. I'd guess the next language will succeed because it makes true software development fully accessible to people who haven't spent years learning how to do it.


Reading his background (http://cperryk.com/) it says "I built my first website as an 11-year-old." "I earned my first dollars building websites for family friends and fellow students' parents." So he may be a bit more experienced than the impression he gives in the article.


Chris Kirk, the author of this piece, interviewed me briefly. I have to admit that when he told me he was writing his own email client for the story, he became my tech journalism hero.


It does sound pretty impressive; I've written a (console-based) mail-client, and between fighting with different approaches to UI, scripting, and MIME issues, I've a lot of respect for somebody tackling the problem.


Am I the only person in the world who thinks email works pretty well? Maybe I just get less mail than many people, but I just delete (or mark "read" and ignore) most of the marketing ham mails I get. I read and respond to the actually-important messages I get for work or from personal contacts. Actual spam is pretty well filtered.

I vastly prefer email to Facebook messages, or Slack notifications, or any of the other purported "solutions" to communication.

Why do people hate email so much?


As the author explains in the first few paragraphs, he was basically fine with email until he was thrust into a job with a much higher email volume. That's been the story I've heard from most people who hate email.


Exactly, the problem isn't email, it's the level of communications you believe you are expected to maintain. Maybe you are expected to maintain this level, in which case you have a problem that no level of email-client fixes will solve.


> For example: I often find myself completely losing track of something that urgently requires my response, or struggling for several minutes to find a single email from a few weeks ago. That’s because email was not designed with the assumption that anyone would get hundreds of messages a day. To a computer, a direct email from your boss looks the same as your water bill. It's on you to put it where it needs to be. IMAP provides only two tools for this: mailboxes and flags. You're expected to sort your messages into mailboxes as if you're sorting paperwork into folders, and/or flag them with a limited set of colors. Do this long enough, however, and you'll soon end up like I did, rifling through an elaborate hierarchy to find the proper folder. The more complex your system of organization, the longer it takes you to organize or find each individual email, and the less useful the system becomes. Eventually, the time you spend maintaining the system outweighs the time it saves you.

It's not so much email-the-communications-system, but "email" as a collective noun for all the different communications demands from other people and system he faces.


I have no problems with it either. It is highly customizable and you can do pretty much what you want to do with so many email clients.

I'm guessing these are just people that don't know about filters and how to unsubscribe from unwanted mail.


it's a mixture of:

conversations: conversations I need to monitor but not necessarily do anything about; conversations I need to respond to; conversations I was added to for some stupid reason and will be unable to remove myself from so they just need to be muted; conversations not yet sorted into any of the aforementioned categories

alerts: some very urgent, some just background noise

anti-alerts: sometimes I'd like to be notified if eg my daily company revenue report doesn't show up!

action items: things I need to follow up with on a variety of dates in the future; unfortunately, most email clients don't have a "remind me of this on a calendar date" easily integrated to their messages

spam, shopping email, mailing lists, requests from people I don't know and haven't emailed before, shopping confirmations, notices from my credit cards, skiing discussions from my ski house, invitations to a friend's birthday party, etc.

most email clients basically don't give any tools to support the above workflows so I have to hold all that state in my head and stuff falls through the cracks.


The ~10 or so messages each day that I feel I should really answer, but not now. They always pile up.


I find very interesting how the author describes some of the shortcomings of IMAP. The article seems like a nice starting point for requirements to create a new email protocol.

Like the author, for years i've been bothered by email, but as someone who has been maintaining mail servers for 20+ years my biggest issue is the server side of things.

SMTP, POP and IMAP have served their purpose way beyond their original scope. I think going to the drawing board and coming up with a replacement is long overdue.


I started typing out the requirements[0] I'd want as a user. It's only a start and doesn't have any implementation, but any discussion would be appreciated!

I wanted to focus initial on the things I hated as a receiver: spam filters (too many false-positives that don't correct when I report them as such), dropped messages (I cannot get messages from http://www.eternal-september.org/ on gmail because of reasons only $DIETY could extract from google.), lack of encryption and authentication.

[0]: https://github.com/jimktrains/email_ng

(For IRC just use the same channel i registered for discussion on a new http (freenode @ #http_ng) (https://github.com/jimktrains/http_ng)


> ...lack of encryption and authentication.

PGP (or the MSFT-specific equivalent) handles this. [0] I know that people love to pile hate on the UX of PGP, but Engimail has worked perfectly fine for me for ages.

[0] Yes, I know that PGP doesn't -and can't- encrypt addressing information. We could address a big chunk of that by creating informal groups of sysadmins who configure their email servers to only speak over TLS-encrypted channels to other servers and clients.


Yes, but I can't even get technical friends to use PGP:-\

I'm proposing it be tied more tightly into the protocol. Some of which is so that there is infrastructure to "vouch" for a sender address (for a period of time) so that they can send you things that would bypass spam checks. (Think password resets, or account notices from a bank).


I think fastmail has proposed jmap as a replacement for imap: http://jmap.io/


From JMAP description on github:

> It aims to be compatible with the IMAP data model,

Unlike this project, I think user requirements should drive the protocol's design, not a need to be compatible with SMTP/IMAP.


Compatibility with the rest of the network is a user requirement.


It should not be, that's the main premise of my idea: create a new email protocol that completely replaces both SMTP and IMAP based on actual use cases instead of being constrained by 40+ years of legacy protocols and extensions.

Client and servers using this protocol would not be required to implement SMTP/POP/IMAP at all, bridge services could be built for that.


It sounds like a good decision, but you've just replaced protocol complexity for implementation complexity. At least a single group of designers and programmers can work through protocol complexity. Implementation complexity is going to impact everyone who tries to implement it - any many of them will just give up.


Yes, this. You need to remain compatible with the rest of the world. IMAP plus every extension in the world isn't even that awful, if everyone supported all of them. The problem is that they don't, so you have massive complexity of servers to support.


I get a feeling that this is what the Nylas N1 folks are doing with their sync server.

They created their own JSON protocol and documented it surprisingly thoroughly, given that their desktop mail client is the only implementer right now: https://nylas.com/docs/


I've always thought an effective if brutal strategy might be the one that won't even deliver your email to you if you're not at Inbox Zero, and then it delivers a day at a time so you might end up days or weeks behind. And maybe it would give you clues, like, "you have 47 additional emails waiting from this sender", so if they're spammy, you could just unsubscribe right there and it would delete them upstream without you even having to look at them.


do people really have this problem? I never remember being bombarded by a single source (be it a person or a service), email becomes a burden because there are many sources and the level of urgency is alternating even for the same source through time


Yeah I manage that by treating all mail as low priority and every chat as high priority. People chatting for spam or to bring attention to a low priority inbox gets told to send an email. This way I'm sure not to miss critical messages without relying on pepople self tagging


I would think this could be a real situation if you have a problem customer (read: demands more than their SLA) and you're not ruthless/effective in your communication.

Sometimes not responding in those situations can lead to better outcomes than replying in haste or frustration.


A few years ago I wrote a webmail client to access my mail on the server host. I had developed a web server (in Scheme no less) that was/is running (now on a VPS) and at the time it seemed only natural to extend it with webmail functionality.

The effort was successful (for some definition of success), though it proved a to be a daunting task. It taught me a lot about email, that it's riddled with arcane protocols and dark corners, these "features" pretty much without consistency or order. On top of that there was constructing the UI needed to actually use the thing.

Would I do it again if starting from scratch? I don't think so, it sure seems the hard way to go about it. No doubt I could recommend it as a great learning exercise, but for creating a product I don't think it's the pragmatic way to go.

I've considered publicly sharing my project, but not before refactoring some prodigiously opaque, obscure and convoluted parts, and who knows when I'll have the time (or courage) to tackle it.

Email remains at once a great wonder and horror, yet for all that email continues to be widely used, and remains essential to most everyone. I still use my email client most every day, despite its warts. I guess it's proven its worth after all.


Why use email as a to-do list, when it's obviously so poorly suited for the role? Why not use email for things email's good at, and use one of the wide variety of to-do list tools for task tracking?

When you need to refer to an email from a to-do item, there's ways to do that; some clients support direct hyperlinks, and others can be dealt with by just including the date/time and sender or subject of the email in the to-do item's notes, so that you have the information you need to quickly find the message via search. You can also include excerpts from email, or even whole messages, directly via copy-and-paste; after all, it's not as though they are going to change once received, so there's no sync problem to manage.

Granted, this might seem like being more effort than just mapping task list semantics onto email. But the article under discussion here chronicles its author's essentially having reinvented every task abstraction all the way up to the project, solely in order to overcome the inherent drawbacks of trying to use email as a task list. It seems impossible that this could be easier than just finding a task list tool that has the features you need, and learning to use it alongside your preferred email client.


Quite a lot of people use their inbox as a task list where the task is "respond to / deal with this email". The problem here is that such a task list is world-writable and chronologically sorted.


Right, I get that, but the point I'm making is that the tool isn't suited to the job it's being used for. "Deal with this email" could mean anything from an immediate off-the-cuff answer to a trivial information request, to a project that takes days to complete; email clients generally provide no good method (or just no method, period) of capturing and tracking such tasks, because that's not the job they are designed to do.

Worse, as you rightly note, if your email account is also your to-do list, anyone can add to it as frequently as they like. This may or may not make an actual difference in the amount of work you have to do, as for example in a case where you get ten different emails all pertaining to the same task, but it might well have a psychological effect, and it certainly makes it harder to keep track of what you actually need to do.

I mean, I don't know. I'm not here to tell anyone they're wrong to do what they've found works for them. But every time the subject comes up, I hear a lot of people saying that what they're currently doing really doesn't work for them, and they've been as yet unable to come up with a better way. From the entrepreneur/innovator side, these pleas tend to be met with yet another attempt to solve problems with email which only exist as a result of making email do things it's neither designed nor well suited to do. I think people would be better served by instead suggesting the use of a tool which actually is suited for the job they're trying to make email do. There are lots of task list managers out there, and almost any of them will do the job more ably than yet another attempt to cram email into a paradigm to which it so plainly isn't suited.


> Why use email as a to-do list, when it's obviously so poorly suited for the role? Why not use email for things email's good at, and use one of the wide variety of to-do list tools for task tracking?

I've no idea. Some mail clients even give you one-click ways to directly turn emails into todo tasks (ex.: Thunderbird with Lightning, and I think Outlook as well).


Outlook certainly does, but the problem remains that an email has a defined grain, the message. Tasks might be much more granular. It's not very easy to make multiple tasks out of a single email, nor track progress separately on those tasks.

I'm currently in the progress of streamlining my to-do list, but it's currently a private git repo of text notes, with some shell scripts to extract to do items from arbitrary notes in the notes directory. I'm probably going to start using emacs's org mode for this capture and tracking.

I use a few lightly nested folders in Outlook (most are single level hierarchies) and I use the to-do flag simply to make the emails related to my actual to-do list easy to recall. Task tracking is external to email, and the to-do is simply a holding area for "sorted, not completed".


Because email is a tool you already have.

"Go get a proper application for that" entails a lot of work, including things that can be cognitively difficult like sifting through the giant sloshing sea of half-finished code that is the Internet to pull out a few good candidates. So it's not hard to understand why using a tool you already have (and already understand) would be more appealing.


I suppose it doesn't have to be a proper application; I've seen people manage to-do items in Windows Notepad and it seemed to work for them, God knows how. I tend to think in terms of task list management tools because I've gone to the trouble of setting up a CalDAV server and integrating all my computers and devices with it, but the general point could (and, as you point out, really should) be less "you need to go find an app" and more "try thinking about to-do lists in any terms other than email, because anything else will probably be less painful".


I wish I could check email in Evernote. Or a Gmail where I could edit notes. If somebody from Evernote is listening..


You'd only reinvent Emacs.


The repo is here https://github.com/cperryk/slatemail I've no idea how to run the app. It seems to be a graphical node.js application.


Hey guys — I'm Chris Kirk, the author. It's an nw.js application (formerly known as node-webkit). Nw.js is basically a framework for building native desktop clients with Node.

Unfortunately I haven't been able to give the client much attention since writing this article. Slate was amazing for giving me a month to work on it and report on that experience, but it's not in the email business. I come back to SlateMail from time to time, but right now the app is not exactly in a usable state. Nylas N1 (https://github.com/nylas/N1) is similar -- an extensible email client based on Web technology. Nylas has a whole team of people on it, and it's in active development. So if SlateMail interests you, I recommend you check Nylas N1 out!


Thanks for the kind words about N1! I'm one of the Nylas co-founders. :)

I'd love to get your take on N1, especially given that you've dug so much into email the past few weeks! Shoot me a note? mg@nylas.com

Edit: Just saw this article was actually written in February :P


Nylas N1 looks really nice, thanks for telling about it :D


No README.md in the root of a git repo is definitely a bad sign.


It's a personal project that the author didn't expect anyone would ever look at. He's also a journalist, not a software developer. It's an accomplishment that he had the determination to complete his project, even if it is rough around the edges.


He's a journalist, so I'm sure he expected people would be looking at it. Reasons were probably more due to inexperience and lack of time.


I bet if you asked him nicely or kindly pointed out that he's missing that, he might add one.



Great! States what it's, why you should not use it and what to use instead. Such an information is helpful.


It uses 'node-webkit'. If you just throw node-webkit at it, it might work!


May be the author is a bad programmer — but undeniably, he's a great tech journalist.


Hah, very true. Though, we all start off as bad programmers =P


He dwells too much upon the age and cruft of email, but this article reads more like a layman learning for the first time about Hofstadter's law, general programming project mis-scoping, and dealing with technical debt he himself created... which is kinda cool to hear about from "the outside".


To be fair to him, though, pretty much every programmer I know since I've been working on (http://inky.com) has asked me why it's hard. Email is partly hard because of the historical cruft, but there are genuinely difficult design/abstraction challenges with email as well, and even experienced developers tend to mis-predict how hard it is to make a "real" mail client.

The one exception to this was Carl de Marcken, one of my co-founders at ITA Software, and chief scientist there. He predicted -- correctly -- that it would take us 4+ years to get to an MVP.


I agree, at Missive (https://missiveapp.com) it took way longer than we originally thought to get to MVP, ~around 1.75 year (and we only support Gmail/Google app for the moment)


Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(


What are some of the most surprising complexities?


There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples:

- sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy

- conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics

- properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle)

- it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't).

- instant full text search over a million message inbox is hard, but this is a real-world use case

- converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole

- making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work)

- implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app

In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.


You've managed to miss a few of the biggies I'd count:

- the charset parameter is nowhere near as correct as you'd like it to be - many RFCs are not followed in practice. In particular, RFC 2047 is literally more often incorrectly implemented than correctly implemented.

I also once recall trying an admittedly weird message (a message whose body is of type message/rfc822) on 4 different IMAP server implementations and getting 4 different answers for part numbering.

I'd say building an email client is harder than building a web browser because at least the people who write the specifications for the latter have been trying to thoroughly document all the idiosyncrasies and how to handle them.

Also, screw CFWS.


Oh God, IMAP part numbering. I'd blocked that out until now. And regarding RFC 2047, I still remember when I first learned about RFC 2231. It really reads like an April Fools joke. Actual text from this RFC:

Character set and language information may be combined with the parameter continuation mechanism. For example:

   Content-Type: application/x-stuff
    title*0*=us-ascii'en'This%20is%20even%20more%20
    title*1*=%2A%2A%2Afun%2A%2A%2A%20
    title*2="isn't it!"


You (well, the spec writer) forgot the semicolons (it's errata'd). :-)

And now I'm stuck trying to remember off the top of my head whether or not the first few lines are permitted to use quoted-strings in the first place.

With regards to RFC 2047, the worst header I've seen to date is this: Re: [Kitchen Nightmares] Meow! Gordon Ramsay Is =?ISO-8859-1?B?UEgR lqZ VuIEhlYWQgVH rbGeOIFNob BJc RP2JzZXNzZW?= With My =?ISO-8859-1?B?SHVzYmFuZ JzX0JhbGxzL JfU2F5c19BbXiScw==?= Baking Company Owner

I've been unable to find any possible decoding of those encoded-words that make sense.

... I can't wait until everyone decides to just use UTF-8 for all the headers. Except then I get to yell at everyone who uses Windows-1252 headers right now. >:(


I think sanitizing email is a solved problem with this:

https://github.com/cure53/DOMPurify

I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.


Probably not fast enough for us, but it is a very nice library. (We use libxml2.)

The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester:

  https://emailprivacytester.com/
Try it on your own mail client!


We have no speed problems using this in production at FastMail (we're kind of known for our speed in fact…). We use it at render time, both on desktop and mobile.


Also, Google has a tool called Caja that is open source:

https://github.com/google/caja


You mention that the client is a really heavyweight app. A quick look at the packages for Windows, iOS, and Android suggests that you're using Python, specifically the CPython interpreter. Would you choose something else if you were starting today? In particular, I assume that stuffing Python into the mobile apps was challenging. These days I'd use .NET/Xamarin or the Elements compiler (www.elementscompiler.com) when starting on a new multi-platform client-heavy application like this.


If starting today, I'd probably try to use Rust.

However, using Python actually makes it much lighter weight than it would be running all native code, because Python bytecode is quite compact, and we use small C extensions for things that need to be fast. Being able to link with C code is hugely important for us, and I don't know if .NET/Xamarin would allow that..?

Getting Python to run on a POSIX environment is actually pretty trivial. Windows RT is harder, which is one reason we've never bothered with a Windows RT port. And on iOS you are limited to using a single core under Python, because you aren't allowed by the O/S to fork.

The biggest pain I've had with Python is memory management: it's almost impossible not to leak memory in a Python program because the garbage collector is quite primitive and reference counting is just so error prone. For an email app you need to be able to run for days/weeks without crashing or blowing up memory usage; you can't just restart the process every 3am like you can server-side.

That's one reason Rust is appealing: memory management is helped by the type system.



I seriously doubt that using Python bytecode rather than native code makes the application more lightweight. Here are the problems I see:

1. Code size: Bytecode instructions may be more compact than their native code counterparts, since the bytecode instructions are dealing with higher-level abstractions. But the dynamism of Python prevents some major optimizations. First, the bytecode has to include identifiers, since everything is late-bound. More importantly, a packaged Python application tends to include a lot of dead code, because the packaging tool can't accurately tell what modules, let alone individual functions, are actually used. And that's assuming you're using something like py2exe which at least tries to prune the set of distributed modules.

2. Startup time: Of course, even native code can vary here. If you have an executable that's statically linked except for OS libraries, the OS can just read that executable into memory and jump to the entry pooint; that's fast even for a cold start. But if your application consists of an executable plus several DLLs or shared libraries, the OS has to do a lot more work to get it loaded. Python modules tend to be more fine-grained than typical DLLs, and for each module, the interpreter has to find it on the search path, then open the file and read it.

3. Memory footprint: Native code is mapped into memory as shared read-only pages. The OS doesn't actually have to add any of those pages to the working set until they're executed, and it can page them out at any time without having to write to a swap file, which mobile platforms don't have. A few bytecode interpreters, most notably Dalvik, have managed to exploit this, but not Python. So all the Python bytecode has to be in memory all of the time. This is exacerbated by the limited ability to trim the code size as described above.

More generally, my thinking on performance, especially perceived performance, in end-user applications has been heavily influenced by this blog post:

http://chadaustin.me/2009/02/logic-vs-array-processing/

Here's the money quote:

> With disk and memory speeds improving so much more slowly than CPU speeds, the difference between a snappy desktop application and a sluggish application is a handful of page faults. When choosing a technology platform for a project, it’s worth considering the impact to overall responsiveness down the road. And I’m pretty sure I just recommended writing your entire application in C++, which sounds insane, even to me. I’ll leave it at that.

That post was several years old. These days I'd expect him to recommend Rust over C++.

Of course, you've been developing software for a long time, and you've actually used Python in a mobile app; I haven't. So maybe the issues I've raised just don't matter. I'm particularly curious if startup time has been an issue for Inky at all, especially on the mobile platforms.


Hard to say, really. Of course if you write something entirely in C or assembly language, you can make it really compact. In the article you cited, for example, Chad Austin mentions GOAL, which was what we used for the Crash Bandicoot games. That worked well because it was domain-specific and because the rest of the code was all highly-optimized C and hand-coded assembly (the latter mostly by me, in fact). And it allowed us to pack the (tiny) code for the critters in data pages that were very, very precious (the PS1 only had 2MB of RAM).

However, x86 assembly, at least, is pretty big, and -- as you point out -- Python instructions are quite high level so you need far fewer of them to do anything. In practice, .pyo files really are generally pretty compact. I'm not sure how it compares to ARM, which has a less crazy instruction set. But I'm very skeptical on the code size claims. It "feels" like Python code is much smaller, but that said I haven't done a real experiment.

Regarding startup times, you do have to be careful about imports and regexes: it's easy to write code where importing a module results in a ton of additional imports you don't really need yet, and regex compiles that are better deferred. But in any case, the big driver of startup times now isn't Python -- it's paging in and decrypting the database pages required to construct the first inbox view. That's all done by SQLite, so Python's largely irrelevant to the startup time these days.

I'm not sure how things come down with respect to memory footprint, but we fit comfortably even on a lowly iPhone 4, so I think it's becoming less of a practical issue.

The huge win for me personally using Python has been that it is extremely easy to come back to code I wrote several years ago (or that someone else wrote) and immediately understand it. For such a nasty domain as email, this really helps a lot. Anecdotally, I've been coding in C dialects since 1979 and I still haven't got the same ability to understand old C code as I do old Python code. For context, virtually all the code I wrote at ITA was C and C++, and Inky is the first "real" thing I've used Python for. It really is a nice language, despite having (in CPython, at least) some frustrating warts (and the whole Py2 vs Py3 issue).

Finally, it's worth pointing out that we do use C for the most heavyweight things (embedded database, TLS/crypto, parsing, image format conversion, etc.) So we're probably not typical of either a C or a Python app, really.


FWIW, you've been coding in C dialects since just before I was born. So your experience and judgment have a lot of weight for me. Thanks for taking the time to discuss these things.

It's typical for Python applications to depend on C libraries for databases, crypto/TLS, and image format conversion. For parsing, it depends; lxml is popular for XML and HTML parsing, but I don't know of any off-the-shelf equivalent for RFC 822 and MIME.


Supporting POP3 seems like an anti-feature in 2015.

It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?


Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.


From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.


Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.


What's great though is that at the end of it, he has a tool that works for him. That's a huge accomplishment.

All too often I obsess over tiny details in my personal projects and never get to anything useful at all. Meanwhile this guy built something that's probably not elegant, but totally does what he needed.


Perhaps Email serves its purpose perfectly: send/receive messages. For everything else, we could either extended it via plugins or create different tools SMS/Chat apps/etc...


This.

Email is pluggable, on client and on server side. The basic idea of email is done and finished; everything that extends it should be a plugin.


keeping up with email became a snap for me once I discovered mutt and procmail. This post definitely falls under "reinventing the wheel", but I can't say that's a bad thing. Unfortunately, most people's workflow is so different, that having one email workflow/interface just won't cut it, and sometimes it takes a custom solution.


I'd have a hard time switching to an email client other than mutt. I'm not sure what I would do without an actual grammar to deal with email.


What do you mean by

> grammar to deal with email.

?


The macro/command syntax for mutt. For example, tagging all the email from someone and saving it to a folder (Tsomeone, ;s=folder). Filter rules as well (e.g. sieve).


Oh, gotcha. Yeah. My favorite part is being able to customize/script those commands in the config.


I used to love mutt, but I found that the "scripting" support wasn't there.

So using mutt as my model I came up with https://github.com/lumail/lumail2/

A console client with _real_ scripting, in Lua. Sadly it is Maildir-only, because thats how I always used mutt, but it seems to make some people happy.


Why procmail, why not Sieve on the server side?


It's what I've always used. I've never really had a need to look for another tool. Sieve does look interesting though.

And I basically access all my mail from the server itself, so my procmail rules run locally anyway.


Perhaps they went over to mutt+procmail before Sieve was written? Perhaps they don't use Cyrus IMAP, or lack the relevant permissions on their host? Both of those apply to me; I have a mutt+procmail account and a Gmail account for different purposes.


Dovecot also supports Sieve. Nonetheless, I still use procmail, even though I could turn on Sieve on my mail servers. Habits die hard, and the tools I use to manage my procmail rules (Virtualmin and Usermin) don't yet support Sieve directly (though its on the roadmap for the Usermin update coming in a few months).


Which mail providers provide Sieve support ? I see Fastmail does, but Gmail doesn't.


> Which mail providers provide Sieve support ?

Nobody really does, which I find extremely frustrating since sieve is supposed to be the Internet standard for mail filtering.

Fastmail sort of does, in that they use sieve rules; but last time I looked, they didn't have a managesieve interface, which means I can't just upload the sieve rules I have spent a lot of time creating on my home email server. I have to create them from scratch using Fastmail's rather clunky (IMO) interface.

As far as I can tell, no other email provider uses sieve at all.


They do allow you to provide Sieve scripts as text, but you do have to use the Web interface to upload them, under Advanced → Rules, so changes aren't readily automatable AFAIK. Unfortunately, I don't remember exactly where the enable button is for people who aren't using it already, since I'm already using it.


> They do allow you to provide Sieve scripts as text, but you do have to use the Web interface to upload them, under Advanced → Rules, so changes aren't readily automatable AFAIK.

Not only that, but I can't upload them using my email client, which has managesieve support already built in.


We don't provide MANAGESIEVE because we don't trust the Cyrus implementation code to be safe to provide over the internet! No in-principle objection to doing it otherwise.

My ideal would be to manage sieve scripts via an IMAP annotation, but maybe that ship is well and truly sailed.


> We don't provide MANAGESIEVE because we don't trust the Cyrus implementation code to be safe to provide over the internet!

I'm not sure I understand. MANAGESIEVE means you send and receive text over a socket connection, which can be secured with TLS. If the text you get back is not valid per the managesieve protocol, you simply ignore it and send back an error; you don't need to trust any code on the client side and you don't need to provide any code over the internet. What is the implementation code issue?


mailbox.org does.


Liked the article but it produced a weird dissonance in me. On the one hand I totally empathized with his frustration/excitement at building a cool bit of software for himself. On the other hand I've never had problems managing my email inbox, mainly because I don't get that much relevant email.

In particular, since the Gmail tab redesign I've not had an issue keeping my Inbox unread count at 0(though with my Promotions/Updates/Forums tab unread count in the who-knows-how-manies). I guess this confirms his point about email protocols being dinosaurs, since the way Gmail implements its tabs is tied into their client implementations and doesn't work in other email clients.

The overall impression I ended up with was that Slate is simply using email for workflows it was never meant to support and is not very good at supporting. Email chains are not a good way to do project management/planning. It's good that they've started using Slack for chat-like communication. They should adopt some sort of project management app to manage projects. If you've got a user that's used to Gmail, they can learn Trello, for instance, in about an hour.

I tend to think about 3 levels of communication: chatting(text, audio, video); email; project related. Each level requiring additional structure and being a bit more cumbersome to use than the last when writing, but easier to use when reading/searching for information.

In that sense it makes sense to use 3 (sets of) tools for 3 different types of communication. The main downside is that it's somewhat complicated to transfer a conversation from one channel to another. However doing all 3 in one application leads to the risk of making a simple use case too complicated or a complicated use case impossible.

The ideal integrated communication app would need to provide IRC-like chat power, Hangouts/Skype like chat simplicity(+audio/video support), Gmail like email usability and auto filtering, and more than Trello-like project management/collaboration tools(for instance, having Google Doc-style shared documents properly integrated into it). All usable in interfaces designed for the particular level of communication you are engaging it at the time, but allowing you to move 'threads' of conversation up or down the hierarchy easily. I've not encountered anything that does this right. Heck just being able to properly integrate Gmail, hangouts chats and google docs/google doc comments between each other isn't possible and all those are made by the same company.


Having built several in multiple languages, I'd recommend for anyone to build their own email client.

It's quite easy to get the basics up and running but quickly becomes tricky getting all of the messages to display consistently. Then you can add embedded images and attachments to the mix. The list goes on and on. But, you really will learn a lot about protocols, parsing text & binary data, working with legacy formats and just all kinds of interesting stuff. Even if you don't wind up with an amazing client, it's still a lot of fun!


I'd second that. The basics are easy, but there will be constant niggles relating to RFC2047 header-parsing, MIME-issues, and issues relating to handling some crazy mails that people will send.

I've written my mail-clients in C++, but I suspect Perl, Python, or Ruby, would be fast enough with the massive caveat that you need to have decent libraries/interfaces for the low-level parts.


The most salient point he makes is when he says there's a difference in what email was designed and how we use it today. And it sounds like Slate needs better tools.

Do you even Confluence / JIRA (or insert your preference here)? If you'd stop trying to use email as the one tool for everything, you'd stop (or significantly reduce) the times you run into it's supposed shortcomings.


the problem with a lot of those tools is that they use email as a notification system. When they should run something more like the notification system for iOS or android. (you have seventeen new notices from jira in 3 issues).

You look at it once, then it is gone and you go to the site to check it out.


Isn't Nylas (https://www.nylas.com) supposed to fix this problem?


It's hard to say. That web site is so buzzword-driven and content-free that after reading a couple of the pages I still have literally no idea what it does or why I might need it.


Hey-- I work at Nylas.

Would love any feedback you have for the site! There's a bit more info on our "Features" page: https://nylas.com/features/ In particular:

> Nylas brings email into the 21st century with modern tools, RESTful developer APIs, and open source code. Integrating with Gmail or Exchange now takes minutes instead of months. Nylas is the easiest way to build an email app.

Pretty much this means we talk IMAP/SMTP/ActiveSync/MIME so your app doesn't have to. Plus there are some nice new features, like a transactional "delta" endpoint and webhooks, which makes writing mail processing code a breeze.

If you're a developer, the easiest place to dig in might be our docs: https://nylas.com/docs/platform or the open source sync engine: https://github.com/nylas/sync-engine

We also shipped a new desktop email app this fall, which started as a fork of Atom. You can see that code here: https://github.com/nylas/n1 Our website is sorely in need of a refresh, but right now our team is busy scaling and fixing bugs like crazy. (Relatedly, if any designers are looking for a job... feel free to email mg@nylas.com !)


Pretty much this means we talk IMAP/SMTP/ActiveSync/MIME so your app doesn't have to.

My first suggestion would be to try putting things like that statement front and centre, and then ruthlessly dump about 90% of the distractions you currently have. That one sentence told me more about what your system actually does than the home page, features page, and a few other areas I scanned combined. Following it with more specific information about how your system can be used and what concrete benefits it offers over what I'm probably doing instead today seems like a good idea.

From the pages I read before, I get that you have a tool that wants to be developer-friendly and fully buzzword-compliant. Of course, so does every other online service whose web site I visit. Unfortunately, aside from being something about e-mail, contacts and calendars, there is literally nothing on the pages I saw that tells me what your particular service is for or why I might want it, which is kind of a dead end.

Another point you might like to consider is that your docs repeatedly talk about things like integrating with some e-mail related service. However, "integration" is an empty word here. The back end code for various projects I work with already sends transactional mails via some service provider's mail servers just fine with the usual one-liners. It probably already integrates just fine with some well-known mailing list management services for larger-scale communications, too, again with close to one-liners. So we come back to what does the integration your service offers actually do with mail and calendars and contacts? Systematically process incoming mail? Manage a mailing list? Send lifecycle mails when someone new signs up? IMHO, something needs to say, very clearly and very early, what your USP is.


Completely agree, a bit frustrating guessing at lines like 'Nylas removes the pain of integrating your app with email ...'


They have an open source email sync engine/REST API https://github.com/nylas/sync-engine and client https://github.com/nylas/N1


Wow. That's site's awful.

I got a bit further than you --- I actually found the API docs, before giving up when a manual Javascript page-load hook broke the back button.

AFAICT it's a web service which talks the email providers for you, thus making writing email clients easier. I'm not quite sure how they do this without the end user having to give them their login credentials...


Another, much longer and more technical view of the complexities of email clients:

Why email is hard by Joshua Cranmer of Mozilla:

http://quetzalcoatal.blogspot.com/2013/09/why-email-is-hard-...


I couldn't help thinking of Google Wave while reading this. There was one missing feature that made a handful of my friends and I stop using it: regular GMail in the same client.

If we didn't have to switch back and forth, Wave would have slowly grown to dominate most electronic conversations I had with friends and colleagues.

Sigh.


That's exactly why I actually liked Buzz... It was integrated with Gmail.


We are working on this same problem providing unlimited, disposable, email for all.

We are getting close to something similar to an alpha version, feel free to subscribe: mailroad.co/


The CSS negative margin you've applied to `.row` is causing your page to add vertical scrollbars. You should remove the padding from `.container-fluid` and apply margins to the inner elements to stop it. It looks naff.


Can you explain 'naff'?

I looked it up (I've never encountered it) and it seems to be a (British) verb.



Aren't there already email solutions out there that treat email as a to-do list, divide it into discrete projects and so on? I believe there should be some, based on Allen's GTD method, if nothing else. (https://en.wikipedia.org/wiki/Getting_Things_Done)


This is an interesting read, when I saw a few of the comments here ask "how hard can it be?" I was reminded of this presentation by Rik Signes at YAPC::NA (2008)

Email Hates the Living

https://www.youtube.com/watch?v=4s9IjkMAmns


> I found the process frustrating, humbling, and strangely fun. I imagined I was on CSI—but instead of a person, it was my app dead on the floor, and I had to find out who killed it and how

What a great way to think about bug-fixing!


How is Slack different from IRC or using netcat on a LAN?


managed permissions, built in history (doesn't rely on you being connected to get a log of a conversation that happened when you were gone), great full text search, integrated rich content (posts, videos, images) which are also searchable in the same way, adding context to conversations, built-in system for notifying people, channels, and groups, with centralized options to tailor those notifications to yourself, a fully featured mobile application that synchronizes with the desktop app seamlessly.

Slack basically removes the requirement for you to have an IRC server, client, and a suite of IRC scripts and bots. It unifies the experience across the team, which, yes, does remove some levels of customization you would get from your personal IRC client, but in practice, it's been a highly productive work-chat tool.

I've basically replaced Dropbox as my source for important files, instead, putting them into a Slack channel. That way what ever made that file relevant is surrounded by the chat that necessitates the file.

It's not for everyone, and I don't think anyone would claim it should be, but if you work on a team, especially a distributed one, Slack brings a lot to the table "batteries included."


Yes, but what happens when Slack goes away?


We'll move to another tool.

Do you still have your IRC logs from 15 years ago? That's a loaded question. I know some people do, but most people don't and most people don't need them.

We put things all over the place that include "what ifs" – what about files in dropbox, or emails in gmail?

Everything includes some level of trust and some level of acceptance of ephemerality or potential loss.

Personally, everything in Slack is work related, not personal, so it can go away, just like I could get fired, or quit, and never have access to it again. If there's something I want to keep, I'll keep it somewhere I believe is safe and permanent, that's up to me, not Slack, or the tools I use.

I also have a reasonable expectation that Slack would provide some sort of export/dump of data, similar to what twitpic did, or any other business does when they sunset. This doesn't always happen, and it's not fair to assume it would, but I trust Slack because they've done a very good job at being responsive to their customers' needs thus far.


The same thing that happens every time you change a piece of software or technology?

You find a new one to replace it. Import your slack backups on it (someone who asks the kind of question you ask surely has backups, right) and get rolling.


It comes ready to run with tons of stuff you would have to add to an IRC setup. Each time the topic comes up there are tons of people insisting that they could do the same with an IRC software + X, and they are probably right, but that doesn't come nicely packaged and ready-to-use for non-techies. + it is a hosted service, which some people like.


More unicorn dust and SV hype-machine


I still believe that multi-part e-mail conversations can be cleaned up with a few "diffs" and presented in a chat-like view.


"When Gmail launched as a public beta in 2007" - Wasn't it April 1, 2004?


Discover Emacs with powerful Gnus.


I have stayed aways from Guns because I thought it was a newsreader, and I have no interest in Usenet. Is it a good email client too?



Addspam;DR




Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: