Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Switchboard – open-source email processing in Erlang (spatch.co)
92 points by mickhagen on June 17, 2014 | hide | past | favorite | 37 comments



I worked with Spatch to make Switchboard, and will be around to answer any questions.

Our motivation in building and continuing to build Switchboard is to make it easier for developers to process emails. Looking forward to more ideas!


Thanks for the nice product, and well done on a polished release - I've been playing around with it for a few minutes and I'm finding that its not only a great tool, but I'm learning erlang to boot. :)

Since I sent my first email in the 80's, I've wanted to build business-logic systems with mail-based interfaces - things like, have an address such as "calendar@mydomain.com" which can be cc'ed on emails which might contain future scheduling events, and have the calendar@ client automatically work out the details of reminding everyone of the date-target that was set .. Seems to me that Switchboard would be the ideal platform upon which to develop such services .. among other things too. See any gotcha's?

Its always amazed me that automated email is not used more and more for basic business processes - sure, we have mailing lists and autoresponders and all sorts of things like that, but the fact that most language/tech stacks don't really treat email as the human-maintained message queue, with first class priorities, seems a little lacking .. I guess Switchboard is an attempt to make up for that. Anyway, thanks again - I'll certainly be devoting some research time to this tool, and I hope I can build something useful on top of it ..


It should work very well for specifying the logic around the calendar example: Switchboard would accept the credentials of the accounts to monitor, e.g. calendar@mydomain.com, and then notify a "schedule notification" worker of new messages. The burden of scheduling reminders for participants would be on the worker, but there are a lot of tools for that. I like the idea!

One note: Switchboard is especially useful relative to a bare IMAP connection when you're monitoring many accounts. This is because it provides a lot of the boilerplate around restarting failed connections and issuing IMAP commands while an IMAP connection has an active IDLE continuation.

I agree that there's tons of room for more automation around emails. I think the lack of automation is due to not having high-level tools, which is where Switchboard comes in, and that emails are typically natural language. I'm personally excited to see Switchboard get used for email classification -- imagine if your business-logic could include the email "class", e.g. "sent by a human", or "marketing", ... etc. A bit like Gmail's categories, but open and accessible to developers.


So Switchboard will basically take the IMAP pain away so I can focus on the business logic? Great!

I'll play with it .. hope you won't mind if I contact you in a week or so in case I have questions or maybe even to give you feedback/demo of what I've come up with in the meantime..


I did a project like this a few years ago and here are my takeaways:

1. Building on top of IMAP is very difficult. It's not a technology that is really designed for the mobile world of today (e.g. streaming, sockets for chat)

2. Threading is a bitch. The old folder tree layout is formidable.

----------------

That said, it's so cool to see something like this in the wild. As an engineer, I would've used this as a boilerplate for my project years ago, and it would've saved me literally six months of efforts.

Kudos to this. It's gods work.

Free startup idea: Use this to help larger businesses collaborate more efficiently (much like if Yammer and GTasks had a baby).


1. IMAP is powerful and ubiquitous, but it does put a lot of work on the client: implementing a parser, caching values, correctly handling unsolicited messages. Switchboard tries to carry some of this boilerplate weight.

2. Yes! We still have to work this into Switchboard, but conversations in the JMAP (http://jmap.io) spec give an idea of how it will be implemented.


It seems to me that this is the startup idea, built on top of this tech.


Seemed to me that it wasn't particularly meant for internal business comms. Just random comms.


Would be a good idea to market it for those two verticals.


Interesting article about message threading:

http://www.jwz.org/doc/threading.html


That article is a classic, and is used by most people I've talked to implement threading if they can't rely on the IMAP server supporting the THREAD extension [1].

However, it also serves as an example of some of the complexity that clients are expected to handle to provide a modern email experience. That algorithm isn't trivial, and reimplementing it with every client is expensive in development time, and error-prone.

As mentioned in a sibling comment, the Switchboard server's roadmap includes making it thread aware, leaving Switchboard clients lean.

[1] http://tools.ietf.org/html/rfc5256


This is a really neat tool, and I look forward to using it. I can think of a number of uses, already - analyses of mailbox activity, statistics and metrics collection, business-processes being interfaced with email, and so on. Heck, I guess I could use this to replace my recent interest in PlanLeaf (email-based todo manager) with my own custom version.

On a side note - its funny how things flow on HN sometimes - yesterday I played with PlanLeaf, and had the thought 'nice, but it'd be even nicer to just have the code to process emails properly', and now .. along comes a stack that gives me a way to do just that ..


I worked on an email application for a year that was on top of imap. It was slow as hell. How does switchboard get around that?

On another note, I think context.io provides a decent paid solution. How is this project different?


For Switchboard's intended task of as-they-arrive email processing, speed is not a huge issue. That being said, there are plans for using more than one IMAP connection to parallelize some requests.

Switchboard is fairly different from context.io. Switchboard is open-source -- being able to keep in control of services accessing emails on your users' behalf is great. Also, Switchboard is intended for processing emails as they arrive using worker processes, as opposed to context.io which appears to be primarily a proxy.


Does anyone know how this compares to Lamson (which has been around for quite some time) ?

[1] http://lamsonproject.org/


Lamson is an SMTP server which acts as the gateway for emails from the outside world to be put in your mail store.

Switchboard doesn't have any direct connection to SMTP. It monitors changes in your mail store using IMAP IDLE and issues events to workers which can then do basically whatever your heart desires.

(btw, we host our switchboard mailing lists on librelist.com which is a showcase for Lamson. cool project)


Correct me if I'm wrong, but Lamson enables those hooks/workers as well - but integrates with an FSM and a queuing mechanism (Celery) to get this done.

Additionally, there is Apache James (Java) which does something similar.


You're correct that Lamson provides tools for processing emails, but it doesn't include a mechanism for feeding it emails from IMAP servers. Switchboard actually complements Lamson -- Switchboard could manage IMAP connections, collect incoming emails, and feed those emails to Lamson. You'd be able to use Lamson in front of Gmail/Yahoo/etc

I'm not super familiar with Apache James, but it appears to be a mail server with pluggable components. Switchboard fits in as a layer between any mail server that supports IMAP, and its version of pluggable components [processes connection over the interface].

Thanks, this really helps us see where the documentation needs to be improved :)


I've not checked this out properly just yet but I'm having a look. I can give you a heads up later.


I think I haven't clearly understood where to use Switchboard. In my normal mail stack (postfix, dovecot) where would Switchboard be placed?


It would sit along side your Dovecot server. The Switchboard core subscribes to events in a folder (using IMAP IDLE) and issues change events to connected workers.


so is this like zeromq for mails?


I can't give an unqualified yes, but Switchboard is similar to zeromq in that it fans out events [new emails] to subscribed processes [workers].

We actually thought about using zeromq, but went with a simpler solution for now, i.e. using websockets.


Can't wait to see what's built on this


Why is this based on IMAP? When I see "server-side email processing" I think SMTP.


The wording is a bit ambiguous — “server-side email processing” refers to processing emails which are stored with an email provider, e.g. Gmail. IMAP is used to interact with the email providers because it is a mail retrieval protocol, and is widely used.

If we were doing “server-side email processing” of emails as they were sent, SMTP would have fit in.


Looks nice so far, worked with IMAP for a while, not fun... So is it on top of IMAP or ... ?


Yup, IMAP is powerful, but it's gathered some baggage over the years.

Switchboard uses a simplified interface based on JMAP[1] for client's and workers, which it does proxy into IMAP commands. One of the conveniences of Switchboard is it manages the IMAP connections: restarting them as they fail and allowing concurrent access.

[1] http://jmap.io/


It seems to be, as far as I understand.

If that's actually the case, how do you handle the case where someone read a new mail (and thus, mark it as read) before app processed it ?


The application uses IMAP UID/UIDVALIDITY values, typically along with IDLE, to identify new emails in a mailbox. These don't change when an email is read, so marking as read doesn't introduce any race conditions in this case.


Looks great. Any future plans for implementations or API in other languages?


Currently, there are worker/client implementations in Python, Ruby, and JavaScript. New languages can be added in response to demand -- is there a language you'd be interested in using on the client-side in particular?

Also, because it uses JSON over websockets, it's fairly easy to implement a new worker/client that follows the Switchboard interface -- here's the Python implementation in 132 LOC: https://github.com/jtmoulia/switchboard-python/blob/master/s...


Could this be used for automatic GPG encryption before storage on disk?


I'm guessing that you want the emails to be encrypted before being saved to any disk? In that case, Switchboard isn't sufficient: it runs in front of email providers, and can't prevent them from saving an email to disk.

Switchboard could be used to fetch the email, encrypt it, and then replace the unencrypted email with the encrypted email on the server. However, replacing emails is invasive if the user doesn’t expect it, and other than the email provider's word you have no guarantee that the unencrypted email was actually deleted.


I fuckin love it. I have a huge hard on for Erlang lately.


will evaluate this thoroughly , needed something like this .


Great, I hope it's a good fit! 

If not, please let us know about any sticking points either through the mailing list, switchboarddev _at_ librelist.com, or the GitHub issues page.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: