Hacker News new | comments | show | ask | jobs | submit login

Great discussion here, not necessarily about the OP's link, but still learned a lot. Would love to contribute my 2 cents...

Our app[1] sends/receives several million emails per month. Not an exaggeration, it's actually seven figures.

Meaning it's more than 100k a day. Meaning it's 5-6 emails every friggin second. On average. It, of course, peaks during US daytime, up to 30 per second.

We tried a looooh-ot of solutions (all priced at THOUSANDS a month at this volume) including Mailgun, Sendgrid, SES etc, but finally settled to a tiny Ubuntu micro-instance on EC2, running Postfix. It has 1 gb of memory, costs us $4 a month and the CPU load rarely goes higher than 4%.

Of course you would need to get yourself familiar with SMTP, postfix, SPF/DKIM, mx-validation, blacklists etc. And by "familiar" I mean "learn it tothe core" :))

Another thing - you need to build-up reputation for your IP, cause email providers like outlook/gmail/yahoo will simply reject your emails if you start sending a LOT out of the blue. You have to build it up gradually, takes months to get there. Makes it a huge PITA when you need to change your IP :((

PS. If you need incoming email to call some external REST-api - postfix can launch a local php-script that does that. Not sexy but - $4 a month, right.

[1] https://www.jitbit.com/hosted-helpdesk/


So firstly, I don't work for AWS or Amazon, or any Cloud provider. Just wanted to make sure that was clear.

After reading your very interesting comment I thought I'd do some maths on the costs SES should be charging you. Essentially at $0.10 per 1,000 messages, and sending "several million" or "seven figures" worth of messages per month, so a possible total of 9,999,999, you should be paying almost exactly $1,000 for that. That's not really "THOUSANDS", but it is substantially more than $4 haha :)

If you're sending 5,000,000, then the figured drops to $500/month.

Anyway your $4/month server is a very cool concept. Would you be willing to share the configuration? Perhaps you've written an Ansible Playbook to provision it for you?

EDIT: So essentially my point is this: it's not that expensive, compared to compute resources to actually run your application, to have someone else manage all of that for you.

SES is indeed much cheaper that the others (mailgun, sendgrid, postmark etc.)

But you have some things missing from your calculation:

1) Attachments (yes, SES charges separately)

2) Dedicated IP addresses (SES charges you separately for this)

3) S3 (where emails are stored

4) AWS Lambda (because you need a script/function, that processes incoming messages as they come in).

But yes, you're right, with SES its cheaper than others, couple of thousand tops

Developing and managing your own system, like the OP did, takes a lot of time and energy - all of that, when calculated, costs a lot more :)

> Developing and managing your own system, like the OP did, takes a lot of time and energy - all of that, when calculated, costs a lot more :)

> costs a lot more

> costs a lot more

> costs a lot more

The point was super clear, and yet you managed to miss it.

@jitbit clearly stated that he and his colleagues evaluated several possibilities, and the decided to set up their own system.

It's literally short-term decision: it's a make-or-buy problem.

The make options surely takes some time, but it is a one-time expense with pretty much low maintenance and super-low operating-cost ($4/month). It also requires some study but hey, that's know-how that is going to stay in the company.

The buy options is a lot more costly, but gives the gift of ignorance: you are not required to know or do anything.

And if you are wondering what the costs are: setting up a basic mail server for a domain takes as little as a couple of hours. A little-more complicated might take a day, and a complex setup not more than a week, for a skilled person.

Considering other options, it might just be cost-effective to hire a consultant to set it up.

Source: i've been running my mailserver for years, and I've done consulting in setting up and troubleshooting mail servers.

SES is on a couple of email black lists though, on the grounds that they're too laissez faire. On your own IP at least you own your own reputation.

Do you have a citation for this?

you would need to get yourself familiar with SMTP, postfix, SPF/DKIM, mx-validation, blacklists etc. And by "familiar" I mean "learn it tothe core"

Why? Wouldn't the "proper" or "best" way of configuring all these things be pretty much the same for everyone? Why could this just not be a receipe: do all these things, in this order, etc.

I don't know anything about this space, but I strongly suspect the generic answer applies: because you need to be able to debug the system when things go wrong. Things always go wrong, and no recipe can replace expertise when they do.

As OP said, there are also things to consider, such as IP reputation, etc... I've done higher volume mailing from a tiny server just like OP, but we also had a couple huge slices of a /24 and pile of /16 net blocks GRE tunneled and used different blocks of addresses for different types of mail. It's not as simple as just setting this up and forgetting it, there's a good bit of arcane knowledge required to do it right. If you're sending millions of messages per day, you cannot afford to burn a bunch of IP addresses because you triggered grey/blacklists for major ISPs or email providers. Shit, these days, you can't afford to burn IP addresses, period. They are officially exhausted. If you HAVE to hit a user's inbox, you can assuredly afford to spend a few grand setting it up right, and then reap the rewards for months/years afterwards for very little recurring cost. Just my two cents, as someone who built a single server to send millions of messages per day.

Well, there ARE recipes indeed, but sometimes people (legitimate people, not spammers) have their email settings misconfigured and you have to fix things...

Example: good practice is to reject (or at least defer) an email if the sending server is not listed in the domain's SPF record. But lots of people are sending email "on behalf" of gmail without even knowing it (when their mail-server forwards a gmail-message to another address, and this address never receives it).

There are tons of little gotchas like this that you need to look into :((

This is what I would do as well. Coming from a hosted mail solution (one of the earliest in the Industry as part a of the initial Dotcom bubble) that cost 10000x5+thensome as much to run, albeit with higher send rates and clusters, It totally makes sense to use ec2 for these needs today.

while it will not give your such configuration precision, i highly recommend to checkout sovereign project - https://github.com/sovereign/sovereign

In the beginning I used to setup just personal "cloud", but now actually using it to send emails for other businesses I run.

In theory it should be possible to run these playbooks without all extras and just keep bare minimum to send emails (never tried it, so not sure what it will take)

Sovereign will configure postfix + dkim + spf + blacklists for you. Plus provides your imap mailboxes to receive bounces and regular mail.

How about redundancy when your $4 a month VM goes down? If you're sending/receiving that much e-mail, I assume uptime is critically important.

Dumb question, can you reassign an already assigned public IP address to a different EC2 instance?

Let's say GP's micro instance goes down, or extra load brings the need for a larger instance, can they spin up a different EC2 instance and reassign the existing public IP address to it?

Absolutely, this is done on AWS with EIPs (elastic ip's). You can migrate an EIP between instances via the AWS API and the change takes effect nearly instantly.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact