
A serverless email server on AWS using S3 and SES - mzehrer
https://github.com/0x4447/0x4447_product_s3_email
======
arno1
> This stack was created out of frustration due to the fact that to this day
> there's no easy way to have a full email server without the overhead of
> installing and configuring all servers needed to handle incoming and
> outgoing messages.

Interesting approach, though I solved this frustration by the use of a Docker
and kept "my data is mine" \+ "no vendor lock-in" \+ "I control all the gears"
approach. (Though, it's not perfect since VPS is ran by "someone" else.. but
that place where you run this stack can be easily changed at your
convenience). Simple docker-compose.yml with 3 images and voila.

This AWS S3 SES setup looks far more complex than what I did using only 3
docker images: the postfix (for smtp), dovecot (for imap), opendkim (for email
sigining & verification). It's really easy to fire-up a VPS with a single
click nowadays.

If someone is interested in the images I am using:

\- [https://git.nixaid.com/arno/postfix](https://git.nixaid.com/arno/postfix)

\- [https://git.nixaid.com/arno/dovecot](https://git.nixaid.com/arno/dovecot)

\-
[https://git.nixaid.com/arno/opendkim](https://git.nixaid.com/arno/opendkim)

Then you just feed the images with the right configs (main.cf, master.cf, ..,
dovecot.conf, opendkim.conf).

It's also possible to template the configs and make the variable-based
configs. Make things scale friendly. I am also using Terraform to automate the
server deployment/DNS record updates so it is easy to get from 0 to 100.

The only drawback is that you are the one to maintain the OS/SW upgrades,
security, etc.. but that's something I really want to do by myself instead of
relying on someone else :-)

~~~
danenania
Another drawback is that while yes, you can scale up fairly easily with
terraform, your server can also fall over if you get a heavy burst of traffic,
and you'll return errors until you're able to provision more machines.
Depending on what you're doing, how fast you're growing, and how much
tolerance your users have for downtime, that might be a pretty big deal.

~~~
vidarh
You can set up autoscaling groups via terraform just fine, with a little bit
of care taken to ensure that you trigger on the right metrics.

If anything mail is pretty much the easiest thing you can possibly pick to
scale, because the inbound mail will be automatically retried. And haproxy in
front of SMTP servers works just fine (really, any load balancer that can load
balance raw TCP connections, but I've used haproxy to load balance thousands
of messages a second).

For your user-facing side you need a bit more resiliency, but nothing stops
you from using a service like SES to back a traditional setup for sending
either. Reliably scaling outbound mail is the easy bit - the hard part is the
reputation management that managed mail services provides, and no specific
deployment mechanism will solve that.

~~~
danenania
Sure, but for heavy/bursty traffic, you can still have downtime while new VMs
spin up. Retries might save you or they might make the problem worse,
depending on the size and pattern of the burst and how your auto-scaling
config interacts with the retry config of various hosts.

It may seem like a nitpick or something not worth worrying about, and for most
that's probably the case. But for some businesses it could be a crucial
difference. My point is simply that this is a legitimate benefit of serverless
that wasn't mentioned above--I didn't think that would be a controversial
point.

~~~
vidarh
That is no different for serverless. You don't magically escape startup times
- you need to carefully ensure that cold startup times are low enough, or that
you maintain excess capacity to compensate.

The precise extent is different between different platforms depending on
overheads, but that just means the point at which you need to trigger scaling
is different.

You can find lots of descriptions of approaches people have taken to keep
serverless instances running to avoid the cold start delays to work around
this... For autoscaling groups you'd instead configure the alarm points used
to trigger the scaling accordingly.

Serverless platforms tends to assume the startup will be fast enough to keep
the connection open rather than return an error, but that is a load balancer
config issue - you can set up retries and wait for other platforms too if it
makes sense.

(Though for email this really does not matter - retries for all commonly used
mail servers follow some form of exponential backoff up to many hours at
least; retries works just fine in practice)

~~~
speedplane
> That is no different for serverless. You don't magically escape startup
> times - you need to carefully ensure that cold startup times are low enough,
> or that you maintain excess capacity to compensate.

Serverless deployments are just another ladder step up the abstraction level,
continuing the tradition that hardware doesn't matter. Similar to code
compiled into assembly or a garbage collector managing memory. In the common
cases, these cases are harmless (otherwise they wouldn't be popular), but they
generally hide what's actually happening. Doing a garbage collection on a
200MB app is generally pretty snappy. But doing one on a 32GB server app can
take seconds or minutes.

Abstractions like these are fine, as long as the limits of the abstraction are
well understood. Sadly, that is rarely the case.

------
giu
Just a friendly reminder, since I've worked with SES in the past: Don't forget
about bounces when using SES [0].

From [0]:

> If your bounce rate is 5% or greater, we'll place your account under review.

To sum it up, try to keep track of bounced e-mails by using the SES
notifications [1].

[0]
[https://docs.aws.amazon.com/ses/latest/DeveloperGuide/e-faq....](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/e-faq.html#e-faq-
bn)

[1]
[https://docs.aws.amazon.com/ses/latest/DeveloperGuide/monito...](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/monitor-
sending-using-notifications.html)

~~~
statictype
We accidentally had ses credentials set up on our QA server and quickly got
banned for sending too many sdf@sdf.com emails. Took quite some time to get it
unbanned. Since then we switched to Mailgun for email delivery but ses is
still useful for processing incoming email via Lambda

~~~
cyberferret
We are careful to use throwaway email addresses like _randomusername_
@mailinator.com on our dev and staging servers so they still get delivered,
but we can just forget about them.

Really interested to learn about the SES simulator address though (posted as a
reply on this thread) - don't know how we missed that, and it would have
really helped with early testing when we were developing the email queueing
system on our app.

~~~
laurentl
We configure all our test addresses to success@ses.amazon.com (or whatever the
simulator address actually is) so we don’t spam or get banned by inadvertance

------
abjKT26nO8
I don't know anything about serverless --- to this day I fail to understand
what this word is even supposed to mean. And the deployment diagram[1] sure
looks complicated to me. I think I prefer old-school servers.

[1]:
<[https://raw.githubusercontent.com/0x4447/0x4447-product-s3-e...](https://raw.githubusercontent.com/0x4447/0x4447-product-s3-email/assets/diagram.png>)

~~~
throwGuardian
The benefits to using this approach over a traditional server are:

1\. Someone else maintains the software running these services, including OS
upgrades, security upgrades and patches, uptime monitoring, etc.

2\. Since every logical component is an independent service, each scaling
independently, any one single component is unlikely to become a bottle-neck
while scaling. In traditional monolithic servers, you'll have to have
contingency plans if you beat storage/network/CPU/RAM limits

3\. The closest thing to this is to break up a monolithic email server into
microservices and deploy them as independently scalable containers, which is a
considerable engineering effort.

Assuming this works as advertised, you can go from zero to a full blown email
service for organizations with thousands of people (assuming the stated AWS
limits are lifted), in record time

~~~
tuldia
> 1\. Someone else maintains the software running these services, including OS
> upgrades, security upgrades and patches, uptime monitoring, etc.

This is the standard marketing phrase echoed to promote serverless. By
experience, I don't think is valid. Packages like unattended-upgrades
automates all this stuff.

Also, not being able to verify what the software is doing is scary and looks
like a 10 steps backwards to me.

> 2\. Since every logical component is an independent service, each scaling
> independently, any one single component is unlikely to become a bottle-neck
> while scaling. In traditional monolithic servers, you'll have to have
> contingency plans if you beat storage/network/CPU/RAM limits

Except when it comes with a bottle-neck by default. Running mail servers
requires rather little resource.

> 3\. The closest thing to this is to break up a monolithic email server into
> microservices and deploy them as independently scalable containers, which is
> a considerable engineering effort.

Why in earth? Have you seen the postfix architecture?

> Assuming this works as advertised, you can go from zero to a full blown
> email service for organizations with thousands of people (assuming the
> stated AWS limits are lifted), in record time.

I'm pretty sure one can have a up and running mail server while the
"cloudformation" thingy will still be running :)

~~~
xienze
The long and short of it is that “serverless” is all done on a pay-per-use
basis. So is running a VM to host an email server — let’s say $5/month on the
low end. With this setup you’d be paying pennies a month assuming normal
personal usage. If you were running an email server for your Fortune 500
company, yeah this wouldn’t make sense. But for personal usage? Assuming SES
isn’t on the shit list of Google et al this is fire-and-forget, and dirt
cheap.

~~~
tastroder
This use case and logic seems really weird to sell the serverless buzzword. My
personal email has been somebody else's problem since the 90s and is done
through the provider of my personal domain at no additional cost.

Setting up a serverless email server seems like something I'd have to bother
with and maintain a few years down the line when the platform of choice
inevitably changes something. Some use cases of serverless applications just
shift maintenance efforts. Sure, I don't have to update OS packages. I still
have to wonder if the service I'm using for my serverless stuff will be there
(for startup vendors) in a few years or, more likely for vendors like AWS,
change their terms, pricing or aspects of their API. I can pay a student
intern to maintain a single VPS based $whatever, AWS consultants cost a
multiple of that. If that's something you can and want to do yourself, sure,
that's great - even for small use cases like this but then it becomes a
philosophical toy problem more than a technical challenge.

------
primitivesuave
Thanks for putting this together and documenting it so well. I’ve had to build
this solution twice now, and far less elegantly.

The S3 PUT charges caught me off guard the first time (receiving lots of
marketing/spam email will cost $1/1000 emails). I ended up putting small
emails up to 400 kB in dynamoDB and only using S3 for large emails and
attachments, which could be a means of cost reduction in this solution as
well.

~~~
mactunes
Honest question: how did you arrive at 1$/1000 mails? When I look at S3
peicing it says 0.005$/1000 PUTs.

Are there a lot of requests made or am I missing something else?

------
Cyph0n
I’ve been working on a small side project that involves processing incoming
email. In particular, it’s an app that needs to do something for each email it
receives from (hopefully paying!) users.

I am not interested in storing user mail, so SES is just too costly, at least
according to a quick worst-case calculation.

That leaves me with two options:

1\. Self-hosted Postfix

2\. Mail service like Mailgun

With (1), there is no need to worry about overages, but scaling the mail
server might be challenging.

The advantage of (2) over SES is that you are only charged a flat fee for each
email, regardless of size. Emails are then automatically deleted after some
period of time. Scaling up and down is easy.

For now, I am using Mailgun, but I am writing the mail processing daemon in a
way that will make it easy to transition to Postfix, if needed.

Also, I decided to write the mail processing backend in Rust, so I’ve been
learning the language as I go!

~~~
tuldia
> I’ve been working on a small side project that involves processing incoming
> email. In particular, it’s an app that needs to do something for each email
> it receives from (hopefully paying!) users.

I wish you all the best! Mind if I ask for the link?

> With (1), there is no need to worry about overages, but scaling the mail
> server might be challenging.

Honestly, quite the opposite.

1\. Duplicate your MX box.

2\. Duplicate your MX record.

That is it :)

> I am writing the mail processing daemon ... in Rust...

You might like to take a look in
[https://github.com/mailman/mailman](https://github.com/mailman/mailman) for
ideas/inspiration. It's a great tool for processing emails too, but cannot
deny I'm now curious to see how one in rust will look like.

~~~
Cyph0n
Thanks! No link yet, but there is a messy GH repo:
[https://github.com/aksiksi/vaulty](https://github.com/aksiksi/vaulty)

Yes, I only learned about MX record priorities last night haha. With Postfix,
the most straightforward way to run code on receiving an email seems to be
through a pipe filter. Running multiple filter processes probably requires a
beefy server.

Thanks for that link! I might just use a similar approach to allow users to
configure how to receive emails (HTTP or stdin, etc.).

~~~
unilynx
Then limit the number of filters... you can have postfix run a fixed number of
smtpd processes, and each process handles only one message at a time.

When they're all full, your server will just stop handling messages, but SMTP
will retry anyway, giving you plenty of time to scale up if the load is
consistently too high

------
coding123
Seems like a crazy amount of architecture. Does AWS keep all this stuff
organized in some way, or will my personal experiments in Lamba accidentally
break this because it's all merged together?

Say I've installed this.

I now want to write my own lamba service to handle contact form POSTs or
something. Then I decide to delete it, but I accidentally delete one of these
crazy email things. What happens?

~~~
drwiggly
You can make "new" sub accounts from your main account. If you wanted to
segregate it completely.

------
theqult
>his day there's no easy way to have a full email server without the overhead
of installing and configuring all servers needed to handle incoming and
outgoing messages.

[https://mailinabox.email/](https://mailinabox.email/)

------
anonu
Now someone just needs to create a serverless (aka client side or browser
only) Gmail like interface you can host on S3.

And the shackles will be broken...

~~~
arno1
[https://www.rainloop.net](https://www.rainloop.net) ?

Anyway, what is the point in moving from one Giant to another Giant? :-)
Unless it saves the cost, I don't see the benefit of such hassle.

~~~
Normal_gaussian
Most serverless offerings are somewhat compatible with a pitcher of greasing.
It would be a doable exercise to duplicate this codebase for a few other
platforms.

------
insomniacity
I'm missing something - how are people reading this in an email client if it
doesn't have IMAP support?

~~~
geek_at
I think the bigger problem is that you can up their bill by spamming them

------
bsder
Please don't use '+' for special purposes in email addresses without making it
changeable (I recommend '_' instead).

Yes, it is "nominally" accepted--in reality there are too many website that
"validate" email addresses and barf on '+'.

------
newman8r
If you only need to read the emails from S3, take a look at this project
[https://github.com/mewa/s3abird](https://github.com/mewa/s3abird)

------
lowdose
Yes this is legit! Got it to work on a throwaway domain under 30 minutes.

------
whatsmyusername
You can get receive at any address on a domain forwarded to whatever email you
want if you're registered with Monicker.

I imagine most other registrars offer the same thing.

------
glandium
Note that SES only retries delivery for a fixed 840 minutes (not
configurable), which is an annoyingly too short amount of time.

------
Boulth
Wow, this is cool! JSON structures resemble JMAP. I wonder what'd be the
effort to add JMAP endpoint to this?

~~~
chrismorgan
JMAP is a radically different beast. The similarities between this thing’s
JSON format for sending and JMAP’s Email data type are superficial only:
they’re both JSON and are representing the same thing, so it should be no
surprise that they look similar. But that’s a quite tiny part of what JMAP is:
JMAP is an object synchronisation protocol. (And this is why JMAP so much more
complex than the typical REST API. And why I prefer it so much.)

I also think the JSON here is only for sending, not for receiving—I presume
that you’ll receive the MIME message, because otherwise you’d be throwing away
all kinds of essential information.

All of this gets you basically nowhere along the path to JMAP, and achieving a
JMAP endpoint would be a _lot_ of effort. This project doesn’t look to be at
all suitable as a base for such an endeavour. Things like sorting (e.g. newest
first), querying (e.g. emails from so-and-so) and JMAP’s state management (so
the server can tell you “something changed” and you can ask the server to tell
you what changed since _x_ , rather than needing to throw everything away and
start again) don’t work well within the design of this system—you need to
store lots of extra details along the way, maintaining indexes and other such
things.

For such an endeavour, I would instead recommend either wrapping an existing
mail server in serverless voodoo (much of which I expect to be not too hard:
you’re essentially just replacing ingress and egress and not running it as a
daemon; but there will be architecturally difficult parts like getting push
channels working probably), or starting new mail server software from scratch
designed to be able to work serverless.

(I work for Fastmail on our webmail. I have general knowledge of how mail
servers work internally, but little specific knowledge; for example, I have no
idea how amenable Cyrus, which we use and develop, would be to serverless
packaging.)

~~~
Boulth
Thank you for your detailed in depth response!

For the record it seems this seems to do some basic processing:

> The Inbox or Sent folder triggers another Lambda function that loads the raw
> email, converts it to a .html and .txt file, and stores it alongside the
> original message, while storing any attachments in the attachments.

This looks OK for me having clients do indexing and processing. From your
description it seems JMAP choose a different tradeoffs and puts more services
on the server side.

~~~
chrismorgan
This project seems to be designed more for a process-and-delete workflow,
whereas JMAP is designed as a general-purpose object synchronisation protocol
on which you can build arbitrary email clients.

------
coder1977
How does this compare to setting up something like sendy (sendy.co) on
lightsail and connecting to SES?

------
raoulbhatia
... and I thought about a similar thing (Email server using Lambdas) just
yesterday ... ^^

------
z3t4
What about email signing?

~~~
eeZah7Ux
After surrendering all your data to a cloud provider?

~~~
vidarh
Signing e-mail servers two different purposes:

Authenticating the actual sender vs. authenticating the server or service
controlling the domain for reputation purposes.

If you trust Amazon enough to use their cloud services, there's little reason
not to trust them for the latter. Doesn't mean trusting them with respect to
the former.

Things like DKIM makes no assertion that the content was sent by the right
person; just that whomever controls the domain has trusted the service in
question to send e-mail on their behalf.

------
eivindga
Cool project! I'll make sure to test it someday.

Thanks for sharing!

