

Ask HN: Email handler - snprbob86

I am currently building a Django web application which involves listening for email. I looked high and low, but couldn't find an existing solution for implementing automated email handlers in Python. So I decided to build one. I'm still hoping there is a great library out there that I missed. Does anyone know of one?<p>Getting a simple handler up and running was easy, but I have run into some trouble with graceful handling of error conditions. For example, I don't always get SMTP error codes when sending mail to bad addresses or full mailboxes. Instead, I often get non-delivery receipts as emails a while later. Unfortunately, these NDRs appear to differ wildly by server and do not include structured or standardized meta-data such as the message ID that was not delivered.<p>I noticed some striking similarities in requirements between the system we are building and Anyvite. Anyvite automatically expanded a gmail address to a full name and parsed a non-delivery receipt (try sending an invite to a known bad email address). I am curious how they accomplished those two tasks. Since Anyvite was a Y Combinator company, I was hoping one of those guys could comment too :-)<p>Thanks!
======
aaronsw
Do what ezmlm does with bounce messages: use a different envelope sender for
each outgoing mail and encode the message ID in that address. If you get a
mail to the envelope sender you can typically assume it's a non-delivery
report of some kind.

For finer-grained information, you can search the resulting message for
HCMSSCs: <http://cr.yp.to/proto/hcmssc.txt>

Further reading: <http://cr.yp.to/proto/verp.txt>
[http://support.kavi.com/khelp/kmlm/user_help/html/bounces.ht...](http://support.kavi.com/khelp/kmlm/user_help/html/bounces.html#automated_bounce_handling)

~~~
snprbob86
This is certainly interesting. In the grand problem of making sense of bounce
messages, there seems to be three sub-problems:

1) Determining if the message is in fact a bounce. VERP doesn't seem to solve
this.

2) Correctly linking a bounce message to the original message. VERP seems to
solve that via encoding the message ID into the return path. I could implement
VERP with "plus addressing". Thanks!

3) Identifying the error that the bounce message represents. Unfortunately,
the presence of HCMSSCs in the hash format does not seem common. Even Gmail's
bounce messages do not provide them.

------
drm237
Hey, sorry we didn't respond earlier, just saw this post and the Anyvite
reference.

Our system is very basic right now and needs to be expanded on in the future.
We start by setting a return path on all emails we send. This directs any
bounce messages to a single mailbox that we can then process.

Parsing the bounce messages for us is fairly easy. We took some ideas from
David Rusenko's post at: [http://david.weebly.com/1/post/2008/06/startup-
email-monitor...](http://david.weebly.com/1/post/2008/06/startup-email-
monitor-your-bounce-messages.html)

You seem to be trying to tie a bounce back to a specific outgoing message,
which may or may not be the right thing to do depending on your application.
For us, we only care about the email address it was sent to so we know not to
send them anything else (if it's a hard bounce).

Best of luck.

~~~
snprbob86
Thanks! I sent a longer note via email :-)

------
jasonkester
You're looking for this:

<http://www.smtp2web.com/>

They listen for mail and hit your site with HTTP POSTs whenever they show up.

~~~
snprbob86
smtp2web solves the problem of getting the messages.

As far as I can tell, there are 3 ways to get access to the messages:

1) Control your mail server, use something like procmon to forward mail to a
script

2) Rely on smtp2web or some other mail server to forward mail to a script

3) Poll for messages via SMTP or IMAP

I chose #3 with IMAP over Gmail (configured for my domain).

The real problem I am having is identifying and categorizing non-delivery
responses / bounce mails.

~~~
aaronsw
I assume you mean procmail, not procmon. Decent MTAs (like qmail) make it
pretty easy to have mail delivered to a script.

~~~
snprbob86
You assume correctly. My day job involves a lot of Windows programming and a
healthy dose of procmon use :-)

------
apgwoz
I would use procmail (<http://www.procmail.org/>). You might also check out
the comments at this older post: <http://news.ycombinator.com/item?id=186875>

~~~
snprbob86
I looked briefly at procmail, but it seems to only have a primitive mechanism
for identify bounce messages. There does not appear to be a way to extract
meta data from them.

""" If the regular expression contains ‘^FROM_DAEMON’ it will be substituted
by ‘(^(Mailing-List : |Precedence :. _(junk |bulk |list) |To : Multiple
recipients of |(((Resent-)?(From |Sender) |X-Envelope-From) : | >?From
)([^>]_[^(.%@a-z0-9])?(Post(ma?(st(e?r)? |n) |office) |(send)?Mail(er)?
|daemon |m(mdf |ajordomo) |n?uucp |LIST(SERV |proc) |NETSERV |o(wner |ps)
|r(e(quest |sponse) |oot) |b(ounce |bs\\.smtp) |echo |mirror |s(erv(ices? |er)
|mtp(error)? |ystem) |A(dmin(istrator)? |MMGR |utoanswer))(([^).!
:a-z0-9][-_a-z0-9] _)?[%@ >\t ][^<)]_(\\(. _\\)._ )?)?$([^>] |$)))’, which
should catch mails coming from most daemons """

I still need a way to get meaningful information from the bounce messages.

For example, my system sometimes emails a file on behalf of a user. If I
receive a "mailbox does not exist" bounce, all I can do is notify the user
that they may have mistyped an email address. However, if I receive a "mailbox
full" bounce, I want to try again without the attachment and only with a web
download link.

------
brianm
Not python, but simple and works great: <http://subethasmtp.tigris.org/>

