
Show HN: Switchboard – open-source email processing in Erlang - mickhagen
http://switchboard.spatch.co
======
jtmoulia
I worked with Spatch to make Switchboard, and will be around to answer any
questions.

Our motivation in building and continuing to build Switchboard is to make it
easier for developers to process emails. Looking forward to more ideas!

~~~
fit2rule
Thanks for the nice product, and well done on a polished release - I've been
playing around with it for a few minutes and I'm finding that its not only a
great tool, but I'm learning erlang to boot. :)

Since I sent my first email in the 80's, I've wanted to build business-logic
systems with mail-based interfaces - things like, have an address such as
"calendar@mydomain.com" which can be cc'ed on emails which might contain
future scheduling events, and have the calendar@ client automatically work out
the details of reminding everyone of the date-target that was set .. Seems to
me that Switchboard would be the ideal platform upon which to develop such
services .. among other things too. See any gotcha's?

Its always amazed me that automated email is not used more and more for basic
business processes - sure, we have mailing lists and autoresponders and all
sorts of things like that, but the fact that most language/tech stacks don't
really treat email as the human-maintained message queue, with first class
priorities, seems a little lacking .. I guess Switchboard is an attempt to
make up for that. Anyway, thanks again - I'll certainly be devoting some
research time to this tool, and I hope I can build something useful on top of
it ..

~~~
makeshifthoop
It should work very well for specifying the logic around the calendar example:
Switchboard would accept the credentials of the accounts to monitor, e.g.
calendar@mydomain.com, and then notify a "schedule notification" worker of new
messages. The burden of scheduling reminders for participants would be on the
worker, but there are a lot of tools for that. I like the idea!

One note: Switchboard is especially useful relative to a bare IMAP connection
when you're monitoring many accounts. This is because it provides a lot of the
boilerplate around restarting failed connections and issuing IMAP commands
while an IMAP connection has an active IDLE continuation.

I agree that there's tons of room for more automation around emails. I think
the lack of automation is due to not having high-level tools, which is where
Switchboard comes in, and that emails are typically natural language. I'm
personally excited to see Switchboard get used for email classification --
imagine if your business-logic could include the email "class", e.g. "sent by
a human", or "marketing", ... etc. A bit like Gmail's categories, but open and
accessible to developers.

~~~
fit2rule
So Switchboard will basically take the IMAP pain away so I can focus on the
business logic? Great!

I'll play with it .. hope you won't mind if I contact you in a week or so in
case I have questions or maybe even to give you feedback/demo of what I've
come up with in the meantime..

------
buf
I did a project like this a few years ago and here are my takeaways:

1\. Building on top of IMAP is very difficult. It's not a technology that is
really designed for the mobile world of today (e.g. streaming, sockets for
chat)

2\. Threading is a bitch. The old folder tree layout is formidable.

\----------------

That said, it's so cool to see something like this in the wild. As an
engineer, I would've used this as a boilerplate for my project years ago, and
it would've saved me literally six months of efforts.

Kudos to this. It's gods work.

Free startup idea: Use this to help larger businesses collaborate more
efficiently (much like if Yammer and GTasks had a baby).

~~~
austenallred
It seems to me that this _is_ the startup idea, built on top of this tech.

~~~
buf
Seemed to me that it wasn't particularly meant for internal business comms.
Just random comms.

~~~
askura
Would be a good idea to market it for those two verticals.

------
fit2rule
This is a really neat tool, and I look forward to using it. I can think of a
number of uses, already - analyses of mailbox activity, statistics and metrics
collection, business-processes being interfaced with email, and so on. Heck, I
guess I could use this to replace my recent interest in PlanLeaf (email-based
todo manager) with my own custom version.

On a side note - its funny how things flow on HN sometimes - yesterday I
played with PlanLeaf, and had the thought 'nice, but it'd be even nicer to
just have the code to process emails properly', and now .. along comes a stack
that gives me a way to do just that ..

------
sherm8n
I worked on an email application for a year that was on top of imap. It was
slow as hell. How does switchboard get around that?

On another note, I think context.io provides a decent paid solution. How is
this project different?

~~~
jtmoulia
For Switchboard's intended task of as-they-arrive email processing, speed is
not a huge issue. That being said, there are plans for using more than one
IMAP connection to parallelize some requests.

Switchboard is fairly different from context.io. Switchboard is open-source --
being able to keep in control of services accessing emails on your users'
behalf is great. Also, Switchboard is intended for processing emails as they
arrive using worker processes, as opposed to context.io which appears to be
primarily a proxy.

------
sandGorgon
Does anyone know how this compares to Lamson (which has been around for quite
some time) ?

[1] [http://lamsonproject.org/](http://lamsonproject.org/)

~~~
mickhagen
Lamson is an SMTP server which acts as the gateway for emails from the outside
world to be put in your mail store.

Switchboard doesn't have any direct connection to SMTP. It monitors changes in
your mail store using IMAP IDLE and issues events to workers which can then do
basically whatever your heart desires.

(btw, we host our switchboard mailing lists on librelist.com which is a
showcase for Lamson. cool project)

~~~
sandGorgon
Correct me if I'm wrong, but Lamson enables those hooks/workers as well - but
integrates with an FSM and a queuing mechanism (Celery) to get this done.

Additionally, there is Apache James (Java) which does something similar.

~~~
jtmoulia
You're correct that Lamson provides tools for processing emails, but it
doesn't include a mechanism for feeding it emails from IMAP servers.
Switchboard actually complements Lamson -- Switchboard could manage IMAP
connections, collect incoming emails, and feed those emails to Lamson. You'd
be able to use Lamson in front of Gmail/Yahoo/etc

I'm not super familiar with Apache James, but it appears to be a mail server
with pluggable components. Switchboard fits in as a layer between any mail
server that supports IMAP, and its version of pluggable components [processes
connection over the interface].

Thanks, this really helps us see where the documentation needs to be improved
:)

------
soupdiver
I think I haven't clearly understood where to use Switchboard. In my normal
mail stack (postfix, dovecot) where would Switchboard be placed?

~~~
bismark
It would sit along side your Dovecot server. The Switchboard core subscribes
to events in a folder (using IMAP IDLE) and issues change events to connected
workers.

~~~
X4
so is this like zeromq for mails?

~~~
makeshifthoop
I can't give an unqualified yes, but Switchboard is similar to zeromq in that
it fans out events [new emails] to subscribed processes [workers].

We actually thought about using zeromq, but went with a simpler solution for
now, i.e. using websockets.

------
nwalter08
Can't wait to see what's built on this

------
wmf
Why is this based on IMAP? When I see "server-side email processing" I think
SMTP.

~~~
bismark
The wording is a bit ambiguous — “server-side email processing” refers to
processing emails which are stored with an email provider, e.g. Gmail. IMAP is
used to interact with the email providers because it is a mail retrieval
protocol, and is widely used.

If we were doing “server-side email processing” of emails as they were sent,
SMTP would have fit in.

------
christophe971
Looks nice so far, worked with IMAP for a while, not fun... So is it on top of
IMAP or ... ?

~~~
oelmekki
It seems to be, as far as I understand.

If that's actually the case, how do you handle the case where someone read a
new mail (and thus, mark it as read) before app processed it ?

~~~
jtmoulia
The application uses IMAP UID/UIDVALIDITY values, typically along with IDLE,
to identify new emails in a mailbox. These don't change when an email is read,
so marking as read doesn't introduce any race conditions in this case.

------
joshdance
Looks great. Any future plans for implementations or API in other languages?

~~~
makeshifthoop
Currently, there are worker/client implementations in Python, Ruby, and
JavaScript. New languages can be added in response to demand -- is there a
language you'd be interested in using on the client-side in particular?

Also, because it uses JSON over websockets, it's fairly easy to implement a
new worker/client that follows the Switchboard interface -- here's the Python
implementation in 132 LOC: [https://github.com/jtmoulia/switchboard-
python/blob/master/s...](https://github.com/jtmoulia/switchboard-
python/blob/master/switchboard/__init__.py)

------
dmix
Could this be used for automatic GPG encryption before storage on disk?

~~~
mickhagen
I'm guessing that you want the emails to be encrypted before being saved to
_any_ disk? In that case, Switchboard isn't sufficient: it runs in front of
email providers, and can't prevent them from saving an email to disk.

Switchboard could be used to fetch the email, encrypt it, and then replace the
unencrypted email with the encrypted email on the server. However, replacing
emails is invasive if the user doesn’t expect it, and other than the email
provider's word you have no guarantee that the unencrypted email was actually
deleted.

------
erlangod
I fuckin love it. I have a huge hard on for Erlang lately.

------
ing33k
will evaluate this thoroughly , needed something like this .

~~~
mickhagen
Great, I hope it's a good fit!

If not, please let us know about any sticking points either through the
mailing list, switchboarddev _at_ librelist.com, or the GitHub issues page.

