
How we send 22k emails every hour - jitbit
https://www.jitbit.com/news/email-architecture/
======
NKCSS
Haha, this made me chuckle :)

> _Here 's a cute completely irrelevant story, feel free to skip. The phrase
> "Because Australians!" has actually become an inside joke/catchphrase in our
> company. See, Aussies and Kiwis (New Zealanders) wake up before everyone
> else (even before Japan). Basically Australians are the reason SaaS
> businesses don't have an outage/maintenance window. And a couple of times
> when we were about to put a server down for maintenance on a late Sunday
> evening (EU time), someone would shout "Wait! Don't do that!" \- "What...
> Why?" \- "Because Australians!" So after 4 or 5 times this phrase is now
> used as an ultimate "42" answer why everything has to be up and running
> 24/7\. I even have a tiny map of Australia in my office. I (nor anyone from
> the team) have never been to Australia or New Zealand, but hey, we're always
> thinking about you mates [wink]_

~~~
sova
Australia is a good, almost-habitable place and the time difference is quite a
trip. Sometimes talking over video chat with my buddy it's daylight in both
the US and Australia and we both say "huh?"

~~~
lostsock
FYI there is generally 3 hours overlap of working hours between the East coast
of Australia and the West coast of USA. 8am our Melbourne time is 3pm San Fran
time (the day before).

[https://www.timeanddate.com/worldclock/meetingtime.html?iso=...](https://www.timeanddate.com/worldclock/meetingtime.html?iso=20190730&p1=152&p2=224)

------
zxcvbn4038
22k e-mails per hour averages just six per second — even the stated peak of 30
per second isn’t anything to write home about. If you go strictly by the book
you are writing email to disk and flushing the buffers when the smtp server
receives the email for relaying, and only acking reception when that has
completed - and that gets into iops, channels, spindles, and other fun
considerations that are often obscured by the cloud these days. But an easy
optimization for a bulk sender is to forego that and use a less resource
intensive method of tracking what hasn’t sent successfully and needs to be
retried. But whatever method is used I don’t think it would really get
interesting until you started sending thousands per second or higher.

~~~
symfoniq
I have to agree.

I built a custom email solution for a small business similar to the one
presented in the article, with a bespoke newsletter creation/mailing list
management/engagement reporting platform on top of it. It handles far more
than 22K emails per hour, which is a trivial volume for even very modest
modern hardware.

However, I do agree with the article that building a system like this can be
highly cost-effective. At one point, my aforementioned client was spending
nearly $75K per year to outsource this. They've been using the system I built
for more than half a decade now, with the only fixed monthly costs being a few
virtual private servers.

~~~
zxcvbn4038
I think the main value add with the outsourced solutions is not having to deal
with all the RBL issues, complaints, staying on top of the latest smtp
requirements from Google etc. That does seem to be a full time job and the 75k
is probably cheaper then staffing that directly. It’s not always about just
the raw infrastructure costs.

~~~
jorangreef
I think you would be surprised at how many outsourced solutions don't deal
with all the RBL issues, complaints, RFC compliance etc.

------
pjc50
This is interesting to compare with this 2004 paper on managing a large email
for the University of Cambridge:
[http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-uk...](http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-ukuug/paper.html)
managing 300k messages per day.

> The system’s back end Cyrus message store comprises 16 single-CPU PCs, each
> with 3GB RAM and 7 72GB SCSI disks providing 350GB of available storage
> after RAID and filesystem overheads. The front end redirector ppsw is 6
> dual-CPU PCs each with 1GB RAM and 4 18GB SCSI disks providing 50GB of
> available storage. The webmail system is another PC with the same
> configuration. The terminal-based Menu System is still running on the old
> Hermes systems at the moment, until migration to the new system is complete.
> It will then move onto a PC with plenty of RAM and CPU. The backup system is
> attached to a disk shelf containing 16 250GB SATA disks providing 3.2TB of
> usable storage, and a tape robot containing a 17 cartridge magazine and two
> LTO drives

These days each of those would be a "small" VPS.

Edit: just found an even better example,
[https://github.com/Exim/exim/wiki/Q1002](https://github.com/Exim/exim/wiki/Q1002)
"On a PII 400 with 128M of RAM running Linux 2.2.5, I have achieved 36656
messages per hour (outgoing unique messages and recipients)"

~~~
NKCSS
It's not really the same... You are talking about hosting big mailboxes; the
article talks about sending and receiving e-mail. Sent e-mails are probably
just discarded and incoming mails go into a processing pipeline, which is
handled externally, which removes the need for a web interface to browse
messages, search them and maintain backups.

------
wefarrell
_But we had to learn everything about email. Every damn bit._

 _SMTP, relays, MX-records, SPF verification, DKIM signatures, spam filtering,
reverse-DNS, blacklisting, "Internet Message Format", throttling, MIME-
encoding, email-antiviruses, bounces, headers, log-parsing, rate-limiting,
greylisting, RFC standards (and that no one follows them)... And of course -
about IP address reputation (more on that later)._

Are there any open source tools to tackle some of these? Given the maturity of
email marketing I would expect some pretty robust free tooling available.

~~~
atonse
I'd love to see if they did the math of the engineer time spent to learn all
this vs paying another service. (Although outsourcing this doesn't necessarily
mean you don't have to learn a lot of these things).

There must be an inflection point where it becomes more economical to host
your own, but I doubt 22k an hour is that point? Maybe it is.

~~~
allana
Sendgrid is pretty spendy, going by their public pricing and assuming the
aveage usage is half the peak usage, the Sendgrid bill could be around ~$2500
usage.

------
noego
> _We looked at paid PaaS providers like Mailgun, Sendgrid, Amazon SES... And
> all of them would cost several thousand a month at this scale. And since we
> 're not a sexy funded startup from California, but a boring customer-funded
> business, we can't afford this bill_

I'm a little curious about their reasoning for buy-vs-build. They say they
"can't afford several thousand a month", but their list of clients includes
Adobe, VMWare, GE and many other impressive names. I find it hard to believe
that with a list of major enterprise clients, they can't afford several
thousand dollars a month?

It seems more plausible that they can indeed afford it, but figured building
it in-house would be cheaper and produce more profits. Developer costs in USA
run about ~$5-10k/month, and that's assuming you're not trying to compete with
companies like Google. Would be curious to hear about the number of dev-weeks
they have spent learning all the stuff they mentioned, building it,
troubleshooting any issues, and keeping the system running and up-to-date. I'm
also curious about the effect building this project has had on their bus
factor.

~~~
jitbit
Hi, OP here.

Well, to be embarrassingly honest... We suck at pricing. We were offering
"unlimited" plans to everyone until recently. And the "impressive names" like
you mention, well, they mostly pay us around $250 a month - which used to be
our "Enterprise" pricing plan with unlimited everything (users, storage,
agents etc.)

So I guess the real answer is - we suck at positioning and we suck at
marketing. As the result - profits were REALLY low (Lesson learned - don't
compete on pricing).

P.S. Couple of years ago I met Thomas from "FE International" at some
conference, really experienced guy, who told me "dude, this is crazy, dump the
unlimited plan like right now" so we did. So I guess technically we can afford
a PaaS _now_...

~~~
eaenki
That's nuts. Millions and millions of dollars left on the table. You have to
A/B test the hell out of pricing (and methods of delivery). I've been a long
time user btw. Great product.

~~~
jitbit
Whoa, small world.

1st of all - thanks for being a customer. Just so you know - _we are terrified
to lose you_ since we're not VC-backed.

2nd of all - you're totally right. Pricing has to be A/B-tested like crazy.
But it's much more exciting to play with a Postfix server in your backyard,
aye? Evading the businessy/salesy part for as long as you can #sarcasm

------
polote
Very good article, that proves the common belief that self hosting emails is
not possible.

I do the same that the person in the article, sending 20k emails per day with
the same setup and same result: it works

~~~
koolba
s/proves/disproves/ right?

------
numlock86
Article summary: To send large amounts of emails get a VPS and use Postfix.
Watch out for IP reputation.

Seriously, not to rant or anything, but isn't this "email server 101"?

~~~
spiderfarmer
For you it is. For me it is. But for a lot of people the first thing that
would come to mind is: lets see what (insert: SAAS) can do for us. And that's
a good thing, because a lot of people just shouldn't be running their own
servers.

------
crispyporkbites
I am literally about to migrate from the mailgun API to my own email server,
so this is a really helpful overview. Thanks!

------
joekrill
> Setting it all up is a one-time expense...but this is knowledge that stays
> with you (and the company) forever.

That sort of depends on how well documented the entire system is, I'd think.
And even then, I'm not sure this is quite as good as it sounds.

Presumably this system is going to be humming along smoothly for quite some
time. But when something breaks or goes wrong, is it going to be easy to go
find the problem and fix it? Probably not. Whoever built it - if they are
still with the company - will likely have forgotten a lot of the little
details. Even if it's documented well, that documentation needs to be
reviewed, understood, and (re)learned.

So yeah, it's pretty low maintenance, sure - until something goes wrong. And
it inevitably will.

I'm not saying this is the wrong approach. Thousands of dollars a month for a
PaaS is nothing to shake a stick at. So this very well could be a great
approach under the circumstances. But there _will_ be hidden costs down the
road. It's just a matter of when and how much trouble they cause. (Will it
cause your systems to go down for days while a bug is investigated, for
example).

------
disiplus
how do you deal with spam ? do you have multiple "prepped" ip addresses that
you switch if some of yours end on the spam list. at your scale it probably
makes sense to take care of it, but i had problems with office365 simply
sending ours to spam folder, with mxmonitor showing everything in green.

for me it was important that every email is delivered (i have 99.9% now)

------
forinti
I once had to send out personalised emails for a large client in Portugal. We
couldn't handle sending them out one by one, but we found out that there was a
50% chance for at least two people to have the same name and surname. So we
bundled together people with the same names and then managed to overcome the
bottleneck.

------
jorangreef
"before that the inbound email is scanned for viruses (ClamAV)"

We moved away from ClamAV because it detected less than a tenth of virus
samples we tested in practice. Simple things like EXEs, ISOs, VBAs and JS
attachments.

VirusTotal uploads showed the same single digit detection rate for ClamAV.

I thought ClamAV was a better scanner, have things stagnated?

~~~
wielebny
Did you use anything else than stock rules?

~~~
jorangreef
Nothing other than stock settings and the latest signatures, same as
VirusTotal's ClamAV I think?

------
marsdepinski
Not sure if this article is serious or a joke. 22k/h is quite small and most
large enterprises handle this volume easily. Very very small scale. I
personally, built a system that routinely would send between 10-20 million
daily and doing about 3k/s. I know of systems that send 10x that ie. 100M/day.

~~~
aytekin
"tiny AWS instance running Postfix under Ubuntu. It has 2 Gigs of memory,
priced around $9/month and the CPU load rarely goes beyond 15%"

The title should have been "How we send 22000 emails every hour for $9/month"

16 million emails for $9 is actually pretty good.

~~~
asark
I'm really happy the answer ended up being "boring, old tools on a smallish
(virtual) machine" since my immediate reaction to the headline was "that'd be
slightly impressive I guess... with limited resources, 20 years ago".

------
CodeWriter23
@jitbit - you might want to add “Feedback Loops” (aka FBLs) to your list of
“Everything” you needed to learn.

