
Bulletproof email infrastructure setup guide - igor_a
http://iafonov.github.com/blog/hardcore-email-infrastructure-setup.html
======
dredmorbius
Great guide a couple of comments:

\- DKIM, aside from buying you a "non-spam" pass (or at least more generous
first treatment), can get you into "most favored sender" status with big email
providers. Notably Yahoo, who, despite corporate troubles is still a huge
player in hosted email.

\- Even if you're using a mail service provider, setting up an SPF record for
your domain, with your _MSP's_ IPs (or better, an "include:" record to their
sending SPF record) will help your mails go through.

\- If you're using postfix, then you'll want to use 'qshape' and 'pflogsumm'
to track your queues and delivery stats.

\- VERPs are great. Until you run into an idiot recipient who insists on
explicitly whitelisting all sender addresses. Including envelope sender. We at
Krell Power have encountered ... a large energy sector company relying on
Microsoft products who apparently favor this route. We've de-VERP'd their
mails.

\- Throttled delivery services with domain-based assignments are a great way
to manage queues. Strong endorsement of that here.

\- Track your mail reputation. Look up your domain and sending IPs on
Senderbase, SpamHaus, Spamcop, and other reputation systems. Address
complaints quickly.

~~~
ciupicri
Speaking of Yahoo, I and many others hate their rate limiting which is
ridiculously low by default, something like 1 email per day. If you want more
you need to fill in a form [1].

[1] <http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html> if I
remember correctly

~~~
dredmorbius
It took dropping an extract of our pflogsumm logs (showing very good delivery
rates to numerous Y! competitors) on their C-level execs (all I could find)
before I got to speak with a "concierge" who apparently expedited the process.

We've got excellent delivery stats now.

------
cullenking
Another option for bounce handling is to setup a client/server architecture.
Continuing with the rails example: since directly spawning a rake task from a
postfix hook will slaughter your machine if you get any decent amount of
emails due to loading the whole rails stack each time, what I have chosen to
do is this:

Postfix pipes the bounced email through command line to a simple client that
shoves the body over a socket to a server process which has my full stack
running. It then shoves the email into a resque queue to be handled by my
normal job queue.

There are many ways to skin this beast, however if you need/want realtime
response to emails, maybe to facilitate reply to notifications email to add a
comment, and you get any decent volume of traffic, you want your postfix hook
to be as fast as possible.

~~~
igor_a
I think this would be the next step for me. For now our performance isn't a
big issue and bounces are processed in background cron job once per several
minutes. For now volumes of bounces are not so high as you have, so we are
just fine with spawning application. Thanks for commenting.

~~~
cullenking
Cool - when we send out our newsletters to users, about 70,000 total active
emails, we get a solid 1000 bounces in the first hour or so. Bounces decrease
each time we newsletter, since we flag users with a bounce_count, and if it
reaches 2, or the bounce is a 5xx (permanent error), we don't email them
anymore. But, 1000 full stack loads in an hour on our mailserver would crush
it!

~~~
igor_a
Nope, you've get me wrong - we start application every 5 minutes to process
mailbox not for every email.

------
gregbair
I'd be careful about using the term "bulletproof" in this context. Before
reading the article, I assumed it was talking about sending spam without
getting blacklisted.

~~~
igor_a
Read the article's disclaimer:) I think even if you'll have all possible
things setup super-correctly and you'll put word `viagra` into the Subject
field - nothing will help you.

~~~
pyre
Would be interesting to get a perspective from whoever manages email for
Pfizer corporate. How many spam filters out there must they have to battle
against to get their legitimate email about Viagara through?

------
csytan
Recently there's also DMARC, which allows a domain owner to specify their
support of SPF and DKIM, and how to handle email that doesn't pass
authentication.

Another benefit is feedback reports from major email providers on messages
that have passed or failed authentication.

<http://dmarc.org/> <http://en.wikipedia.org/wiki/DMARC>

~~~
igor_a
Thanks, I have never heard about this thing. I think I would add it to the
article.

------
twakefield
This is a great resource for those that want to roll their own email
infrastructure.

We also have some tips about email content in our FAQs:
[http://documentation.mailgun.net/faqs.html#does-the-
content-...](http://documentation.mailgun.net/faqs.html#does-the-content-of-
my-email-matter-for-deliverability)

The Mailchimp resource mentioned in the OP is also highly recommended reading.

------
icebraining
How likely are servers to reject your emails simply because the PTR records
don't match? I'm using my ISP's SMTP server as a smarthost and I've configured
the SPF record to match their server, but making them add a PTR record is out
of the question, unfortunately.

(This is just for personal mail, that's why I'm avoiding paying for a proper
setup)

~~~
seanp2k2
Plenty. Protip from a sysadmin in the hosting industry: don't even try to run
your own MTA unless you're just curious as to how hard reliable mail delivery
is.

Use GMail or similar. Use iContact / StreamSend / MailChimp for newsletters.
Use Google Apps for any business you start.

~~~
icebraining
Well, the (sending) MTA is not exactly mine: I configured exim to send
everything through my ISP's. Of course, the domain is still managed by me, and
so is reception, although I keep Google Apps (free) for the backup MX.

------
edgesrazor
Thanks! This is a great walk-through for those using a solution with only 1 or
a few IP addresses associated with little chance of that IP changing. Do you
have any experience for those using services such as Amazon SES or Google
Apps?

Setting up the PTR records for reverse DNS lookups are very tricky since
Amazon and Google are always rotating the IP addresses for their mail servers.
We've been fighting with this in our application for a little while now (using
Python, not Ruby).

We're getting about 90% of our emails through, but some are getting denied for
no reason whatsoever. The worst part is, they are server-side denials, so the
emails aren't even going into the spam folders of the users.

Also, I second adding the DMARC section. DMARC is extremely simple, and it's
very easy to implement gradually.

~~~
igor_a
I haven't worked with external emails providers. For now we are sending around
100-150k emails/day and we are using postfix as MTA.

Regarding server side rejects - again reading postfix logs can save you from
such things. As an example recently we started seeing that we can't deliver a
lot of emails to orange.fr, and the worst - if send just one email it always
was accepted but when we sent bulk newsletter we had a lot of rejections - the
reason was quite simple we've got rejected because of high frequency so we
just had to throttle postfix on per domain basis.

~~~
edgesrazor
I think I might have to set up postfix and do some experimenting. One of the
strangest things I've seen is that some domains are rejecting emails with
links to our site in them, but will accept them if the links are removed
(plain text or HTML).

I swear, someone could make a good living handling this for companies looking
to handle transactional email internally.

------
mshafrir
If I'm using a service such as Mailgun or Sendgrid, what parts of the article
apply to me?

~~~
igor_a
None, but all of them apply to Mailgun or Sendgrid:)

~~~
krobertson
I wouldn't say none, you still need to setup your DNS settings with SPF/DKIM
to support Mailgun/Sendgrid. They usually have documentation how to do that,
but you need to be sure to do it. :)

Good article though... overall, if you aren't in the kind of email volume you
guys are, just use Sendgrid/Mailgun. Sending email is a PITA and just pay
someone else if you have the option. :)

~~~
seanp2k2
PTR too. rDNS is important.

------
daemon13
Igor, thank you, excellent guide/walk-through.

------
gregbair
it's down for me for some reason, but here's the article in the gh repo:
[https://github.com/iafonov/iafonov.github.com/blob/master/bl...](https://github.com/iafonov/iafonov.github.com/blob/master/blog/hardcore-
email-infrastructure-setup.markdown)

------
ajsharp
FWIW domain_info doesn't appear to be on rubygems.org.

~~~
igor_a
Thanks for pointing it out. I link directly
<https://github.com/iafonov/domain_info> in Gemfile. But anyway just pushed it
to rubygems - <https://rubygems.org/gems/domain_info>, so you can link it as
`gem "domain_info", "~> 0.0.1"`

