
We just open sourced Flanker, our Python email address and Mime parsing library - twakefield
http://blog.mailgun.com/post/we-just-open-sourced-flanker-our-python-email-address-and-mime-parsing-library/
======
Erwin
Note that MIME is also used for multipart uploads. I'm currently still using
the builtin-parser.

Looking at a 100mb multipart upload I can see that email module takes 3.7s and
allocates 265 extra MB to parse it, while flanker.mime takes 0.22s and
allocates 0.2mb extra memory.

~~~
INTPenis
I would love confirmation on this statement.

I'm a corelibs nazi and refuse to add dependencies to the very last when I'm
writing Python.

But if what you're saying is true then I can see a very good argument for
replacing email with flanker.

~~~
daniel-levin
There seem to be a bunch of people who have the same opinion -- stick to the
corelibs until it hurts. Do you want to articulate why?

~~~
INTPenis
Less dependencies make for happy users.

Python has already simplified programming so much coming from languages like
C, so I guess I view it as a tolerance matter.

I'm willing to put up with the longer way of using corelibs in some cases, to
keep my dependencies low. And I'm willing to tolerate this longer way because
I feel that Python in it self is such a shortcut from other languages that I'm
already reaping huge benefits.

I know there's usually a huge community testing and supporting all
dependencies but having installed some Ruby and Node apps I've seen massive
lists of dependencies and I've had situations where it breaks down on some
obscure library used to do some really mundane task.

I could not have solved many of those cases without the years of Linux
experience I already have. I don't think users should have to go through that.

------
grinich
I'm working on a new email platform,[1] and we've been testing Flanker quite a
bit. It's awesome!

Dealing with MIME has always been an issue in Python, and the last move was
Zed Shaw's Lamson project a few years ago. The native email.parser module is
crazy slow once you start doing heavy processing.

Flanker's addresslib also is a no-brainer to use. These are the sorts of
things which should be bundled with the Python email module.

Thanks for open sourcing them!

\--

[1] This isn't announced yet, and will deserve a much longer post. You can
sign-up for news here: [https://www.inboxapp.com](https://www.inboxapp.com).

~~~
pekk
> These are the sorts of things which should be bundled with the Python email
> module.

But only after they don't need to be changed any more...

~~~
grinich
Agreed.

------
alexk
Hi All!

Mailgunners will be hanging around in comments, ready to answer any questions,
should you have any.

~~~
simonw
This looks really useful - thanks very much for releasing it!

------
lifeisstillgood
I am in two minds - eight years ago and more we were all desperate for a good
mail handling library. Now, I hardly seem to have a use for automated mailings
that is not handled by Mailchimp (sorry!) or I just want to fire off HTTP
calls to some API to mail custom mailings for me (which is I think what you
guys do)

In short email is probably more important to my business (even with my paltry
list numbers) but I no longer run a mail server, anywhere, I do not even have
a local mail client unless you count JavaScript.

I'm wondering if I am trading off vulnerability for ease at the wrong places.

... hmmm a little unfocused I am afraid but I really have not made any
explicit strategic decisions over email but even so, my whole mail handling
has altered.

I just am surprised I have not noticed how big a step it has been.

~~~
alexk
By making email processing easier, we hope that we can help others to innovate
in the email space, not waste time on the basic processing problems we've
solved a while ago

~~~
lifeisstillgood
I _think_ my point is that its still companies / providers "innovating", not
individuals, because many indviduals have given up running their own mail.

So, its a good thing, yes, but I think the unit of abstraction is not the
individual developer but the "organisation". WHich is oddly different from
most OSS

------
codegeek
Github repo for the lazies:
[https://github.com/mailgun/flanker](https://github.com/mailgun/flanker)

------
alexk
BTW, it took us a while to make sure we deploy right from pip on production,
that means that community will get all the latest hotfixes and the github repo
won't be stale as it's not just the mirror or one time copy.

------
j_s
Jeffrey Steadfast discusses the edge cases of MIME parsing and performance
optimizations in his MIT/X11 C# library here:
[http://jeffreystedfast.blogspot.com/2013/10/optimization-
tip...](http://jeffreystedfast.blogspot.com/2013/10/optimization-tips-tricks-
used-by.html)

I appreciate how the Flanker release includes benchmarks without thrashing on
alternative implementations.

------
bryanh
Wonderful, thank you for releasing this! I hope you guys continue to release
good bits for handling email, it really is a bear. The ongoing joke in Zapier
is a small library that I made called simple_imap. Well, as the story goes, it
quickly became anything but simple...

Where is the python-requests for email? I will seriously buy someone as much
beer they can consume in an evening if such a thing exists!

------
StavrosK
As an aside, I use MailGun with the Linux nullmailer package to send
notification emails from my home (and every other, really) server.

It's a solid service, very easy to configure, with generous free limits (for
the five emails a month my server sends) and with a very nice UI. Props to the
team.

~~~
ferrantim
Thanks! Makes our day to hear (I'm a mailgunner).

~~~
StavrosK
Well, you guys deserve it. I've had so many hassles sending email that
nowadays I just use you to do it. The centralized, one-page instructions on
setting up are especially helpful. "Do these things and your emails won't
bounce or get sent to spam".

------
sigil
This is great guys. It's surprising that you're able to get an 8x-20x speedup
over email.parser with another pure python parser. (What does this say about
email.parser, eh?)

Did you consider using a C extension and tools like lex or ragel for the
rfc822 scanning phase? Not worth it?

~~~
lifeisstillgood
I think it says that email.parser reads the whole message then parses. Flanker
can just read the headers and ignore the message, so I guess with the right
configuration and test data you get the 20x speed up.

It's pretty unlikely that many well-used and years old OSS language libraries
have orders of magnitude speed ups lying on the table, without there being a
design decision in there that can be argued many different ways.

~~~
sigil
But Flanker's full-message parser is still 8x faster than email.parser's full-
message parser, according to their benchmark.

 _It 's pretty unlikely that many well-used and years old OSS language
libraries have orders of magnitude speed ups lying on the table..._

Normally I'd agree with you, but this is the python stdlib we're talking
about.

~~~
lifeisstillgood
I missed that. But 20 down to 8 is still a headline vs a benchmark, and
benchmarks can be ... swayed based on who is benchmarking.

But I think my point elsewhere stands - I hardly want to parse email these
days, even though I consume more of it. Most of my mail is outsourced and it
seems to be a trend.

Maybe not a good one. I am not sure. I used to hate spamassassin though

------
state
I've been using Guardpost in production for the past few months, and it works
really well. I'm curious to see the internals. Thanks for this!

------
veesahni
Great work mailgunners!

Where does the project name (Flanker) come from?

