Having recently adopted Resend and skimmed a bunch of different email APIs, I'm still waiting to find a single provider whose API supports Stripe-style idempotency [1] so that I can guarantee I don't send the same email through their API multiple times. I'd like to confidently avoid accidentally spamming a user if i.e. a background job retries multiple times due to an unrelated error, or merely from failing to receive the API response that an email was sent/created successfully.
I don't even expect them to keep a list of IDs forever, just some best effort like "we don't send anything with the same id twice in a one-hour window"
Gmail's API did support this I believe, and then ofc my org transitioned to Microsoft so my little homemade email service quit working
While you are here, I have been asking for years to have better access control on API keys so they can only use assigned servers. So my staging cant send prod emails...
I understand this change might be a large undertaking to implement across the entire API, and although it would be good to do everywhere, it’s primarily just the send email API that needs it.
Thank you! I'm not sure how this didn't come up in what I felt was a pretty comprehensive round of Google searching, but it looks like a fantastic option. If only they had free allotment or something <$20/mo for a small number of emails :) But otherwise this looks a the solution I was looking for.
This is exactly what I've been wrestling with recently.. it's so disappointingly absent in all the offerings out there.
The solution I ended up with was to build my own pseudo-idempotency around Postmark. It helps that Postmark at least has proper persistence so you can query their API to check if you've already sent a certain email. I had to move away from Mailchimp because, if an email gets queued for some reason, it isn't reflected as sent in the API so there's no way of knowing whether an email is hiding in "queued limbo". Postmark doesn't have this problem.
The only caveat is that there is a delay between sending an email and it being reflected in the API (seems pretty standard across the different services). This means I had to implement a simple database backed lock to make sure I never try to send the same email more than once in quick succession.
It's kind of ridiculous, but.. I couldn't find any alternative.
Edit:
Wonder if it would be worth wrapping something like this up into some kind of a self-hostable proxy service. You'd provide the idempotency key in a header and it would do the rest
Self-hosting the mail-composing, mail-merging and tracking functions while using third-party SMTP is well-trod territory. The former can be done on a pretty basic box and the latter, at scale, is a heavy lift.
My guess is Plunk is for people who need a Mailchimp-like offering at a lower-than-Mailchimp price point.
The last time I tried to use Sendy with SES I received a NNN from AWS after I stating I would be using it for email marketing (MailChimp alternative). It seems that my "self-hosting" idea was cancelled by a 3rd party.
Most likely you did not answer the questions that are most important to them.
- How will you make sure that only subscribed recipients will get your emails
- How will you handle bounces and complaints
- How will you monitor your sending
Don't beat around the bush and you will be approved in less than 1 day.
Resend is also built on top of SES, and yet it's an extra layer of hosted services on top of it. So this is actually a self-hosted alternative to i.e. the functionality that Resend offers, even though there are still other underlying dependencies. Seems like a reasonable way to describe it IMO.
So if an application uses a 3rd party provider for authentication (Google, Github, etc) is this no longer considered "self-hosted", despite it running on user's hardware?
I'm curious to know where you draw your imaginary line.
SES is an email platform. So Plunk is a self-hosted email platform - which uses a cloud platform to actually send email. If it's not sending email by itself, it's not a self-hosted email platform.
Those apps using 3rd party auth providers almost universally have them as an optional plugin, allowing you to choose one of a dozen auth providers or your own self-hosted one. You're not tied to any specific 3rd party platform, and rarely tied to any 3rd party platform at all.
if we look at the philosophy behind self-hosting - to be able to host it ourselves, aka not rely on an external party for hosting, that means that if an external party is down or otherwise unavailable, then that line becomes fairly apparent. If 3rd party auth is required, then it's not really self-hosting then is it, because if that third party is down and the software can't be used, then that philosophy is violated. if it's optional, then that's fine though.
For something to be considered self-hosted, If the bombs drop and civilization is over and me, and my home lab are the only survivors, can I still use that software? I'd probably have other concerns in that case, like how am I going to survive nuclear winter, but that's where the imaginary line is.
I think you are mistaking interoperability with a network with dependency with a propietary service from a specific vendor. These are different things.
how so? ircd will keep running just fine on my server, as well as exim and dovecot for email. external Name resolution would be an issue, but I can run internal DNS for the other survivors that join my network.
edit: prmoustache understood your comment better than i did.
the question was if it is self hosting or not, and eg slack and discord are not. so regardless of how many humans are still alive to use it, on the only server still running, the survivors could use something self hosting whereas they could not use one that required a system that was down.
That ircd servers in the example above can still be used with a bot as an automation tool (at that point as the only survivor the security would be the least of your concerns), as a way to invent imaginary friends to pretend your life is less miserable, whatever.
Or say you have a mastodon or pixelfed instance. Sure no other instance will ever connect and no other user will ever be able to see your posts...but you. Which could include some fond memories you want to keep.
It's a pointless intermediary when you have the bots directly there on your last remaining server though.
These semantic games are obfuscating the real issue here; tying the definition of "self hosting" to the "If the bombs drop and civilization is over and me, and my home lab are the only survivors" scenario is less useful than a two handed pit saw to a sole survivor.
I appreciate your assist to what I consider to be fragmede's "foot in keyboard" moment of poor definition, I suspect we're all better served with a better definition.
This is self hosted. I think we'd all accept that a "self hosted Stripe" that obviously had to offload payment processing to a third party gateway could still be self hosted because the platform is, not every dependency. Volume email is not something you should be looking to 'self host' and would leave the product dead on arrival if it did.
It feels like pedantry to critique that aspect so heavily over spirit v letter.
This is one thing to have a self hostable platform you can plug to any email infrastructure of you choice, another one to have it dependent on one specific vendor.
Where's the line, then? You're always going to have external dependencies, so in your usage it seems like an unattainable ideal more-so than a useful phrase anyone could use to convey ideas.
In my utility room I've got a box running TrueNAS with a pile of hard-drives serving data over SMB and NFS. I've got a bunch of tiny business desktop PCs all set up in a k3s cluster hosting a variety of services for myself, family, and friends. Is this self-hosting?
The entire thing's half-useless without a tiny VPS that provides ingress since I'm behind CGNAT. Speaking of, I've got no way to get bits to and from the internet without my ISPs. I also rely on external services for off-site backups and some other storage. While I run my own IMAP/webmail services, I rely on AWS SES for sending email because managing email deliverability (especially from a shared residential address) is something I just don't have the time and patience for.
I think it's generally more useful to take it to mean "taking more control over your dependencies, data and privacy" without drawing a hard line. If someone migrates their social group off of Facebook and on to their own mastadon/something instance running on a VPS somewhere, I don't see any reason to gatekeep the term "self hosting". Making the goal unattainable just discourages people from making positive steps.
Words have meaning. Insisting on proper usage is not gatekeeping. It’s like this with every new word; everybody wants to appropriate it to mean whatever they themselves are doing, in order to unfairly, and dishonestly, capitalize on the value the word signals. See also, for example, “ecological”, or “open source”. And they all complain about gatekeeping when they are called out, too.
The meaning of words is defined by the people that use them and how they're understood by others.
"Full control over the whole stack, full stop, no exceptions" is not how people generally use or understand the term "self hosting", and if we were to prescriptively define it as that it would make self hosting unattainable and useless as a term.
I proposed a useful way to draw the line. How would you propose we define it?
IMHO, unless you own the hardware, it’s not “self-hosting”.
IMHO, unless you have said hardware running in a location under your personal control (i.e. a room, building or house you either own or rent), it’s not “self-hosting”.
Both of these are not unattainable, or were not so in the relatively recent past.
But I will concede that it’s not always easy to distinguish in the gray zones. It’s like where you sleep. The differences are gradual from “owning”, “renting”, “apartment by the week”, “dorms”, “hotel room”, “airbnb”, “hostel”, “flophouse”, “tent”, “homeless”. Which of these constitutes a “home”? Opinions vary.
That said, however, resorting to “The meaning of words is defined by the people that use them” is also something which people arguing in bad faith almost always do.
You said self hosting means you own the entire stack yourself.
What makes "the hardware and the physical location" anything but arbitrary in terms of that? Why is that "the whole stack"? Why is the hardware and location more important than how I, or anyone else in the world, actually accesses it, when basically all of us are relying on some external party for internet access? What justifies a line that prevents me from deriding anything as "not self hosting" if someone hasn't set up their own tier 1 network?
Why does putting a server in my house, but relying on an ISP, an external box for access, and a bunch of other stuff count as "self hosting" but sticking an old rack mount server at a friend's place because they have symmetric gigabit fibre and my acreage has 5/0.3 DSL not self hosting?
Why would purchasing a server and colocating it at a data centre with a fully encrypted drive be less "self hosting" than putting a raspberry pi in my utility room behind tailscale to bypass the CGNAT? How is putting a server in a data centre different than putting it in a rented apartment?
If I run a MUD on my laptop that's online whenever I check in to a hostel and have internet, am I self-hosting the MUD? If I'm not, who would you say is hosting it?
Circling back to my original comment--what is the actual benefit to "your own hardware on property you own"? Is it "taking control over your dependencies, data, and privacy"? Because then anything that accomplishes that should probably be included on a spectrum of self hosting. Somebody hosting a matrix server on an Digital Ocean VPS as a social network for their friends is still closer to that goal than somebody setting up a Discord server. Somebody hosting a Kubernetes cluster in their closet on a residental ISP is closer. Somebody racking a bunch of gear at a DC is closer.
There's no reason to tell any of these people "you're doing it wrong".
On a separate note: I feel I've got a fair point here, one that you've practically admitted in your own comment. Going on to strongly imply I'm arguing in bad faith really came across as a dick move.
Trying to redefine “self-hosted” to suit your purposes does not benefit anybody but people trying to make “self-hosted” a feel-good meaningless marketing term.
> There's no reason to tell any of these people "you're doing it wrong".
I’m saying they’re not self-hosting. Everything else is you hearing things.
I would normally obey the rules of debate and answer your litany of questions, but I think that in this case it would not help anybody, as you already know the answers.
To most shops, using SES is considered "doing it ourselves" so I somewhat get the characterization. It's very common for self-hosters to use DO, Lightsail, Hetzner because the thing you're hosting is the software. With SES the line between it and SG can be huge if you're using all of SG's high-level features or razor thin if you use SG as SMTP over API.
Furthermore, the thing that complicates the current discussion is with email services in particular, so much depends on reputation and avoiding getting on block-lists. There's also the header wizardry that has become a requirement for interoperating with a lot of large email servers that isn't required for using the protocol but is practically required to ensure delivery.
On the flip-side of that, there are those who want the extra amount of customization that can only truly come from owning the entire stack. Or perhaps they already have a high-reputation IP address they have been sending from for years, and a self-hosted solution is necessary to continue with that (barring some transfer of that IP address to the operation of another company). In that case, calling this self-hosted is being disingenuous.
In some ways I see the dependency on SES (or other third party provider) as being a benefit for email hosting, especially for a new venture/property. It's why there is some value in depending on other examples of managed services. But I would be clear about that and not call it self-hosted.
You host it yourself. It's an email marketing platform (not an email server), but you use AWS SES as an email infrastructure because it is cheap and reliable. You cannot build or host your own infrastructure since emails are hard these days, you use AWS SES or alternative to send emails like most of these services and apps do. It's so easy to get blocked or blacklisted by other email providers.
The wording is awkward and suboptimal. If you don't know what AWS SES is, you'd easily get the wrong impression. In fact, I'd even say it's factually incorrect. It's not built on top of AWS SES. It's built to utilize AWS SES for its functionality. Slightly different, pedantic really.
“Platform” normally implies that it includes the infrastructure that implements the intended functionality, or what is named in the product/project name (“email platform”). In that sense the name is misleading.
(Disclaimer: I run Buttondown, which is an email-inflected product. I also have a comparison guide of various MTAs here — https://buttondown.email/comparison-guides/esps — and will add Plunk!)
Nuances of self-hosting aside, I'd recommend caution looking at that pricing pane.
The disclaimer is:
> Calculated based on the plan that best matches the features of Plunk at 2500 emails per month.
And the author then cites prices of $0.001/email for Plunk and $0.013/email for Sendgrid.
I am neither a user or fan of Sendgrid but that feels like disingenuous anchoring. Sendgrid's free plan is 100/day; if you want to eliminate the free plan entirely and just focus on their cheapest paid option, it's $19.95 for 50K emails — or $0.000399/email, cheaper than Plunk's sticker price.
It's not entirely obvious to me how that $0.013 was even calculated ($0.013 * 2500 = $32.50). Either way, be sure to spot-check against live sticker rates before making a decision there.
My understanding was that the main interest of SendGrid, Mailgun et al was precisely that they manage to (more or less) reliably mass send email through the great MS/Google/Proofpoint, where hitting the recipient's mailbox is less than guaranteed if you self-host.
Maintaining sender reputation is tough and I don't think anyone in their right mind should try this in house. If you're not a M3AAWG member, you'll probably hit a brick wall when trying to explain to Google why your whole IPV4 block is erroneously on a blocklist.
Founder of Plunk here.
Yes, Plunk is designed to work with AWS SES. I was not the one that made it hard for folks to self-host their own email infrastructure. SES is simply the cheapest and most reliable option out there. Reminder, we are not talking about a private "let's send 10 emails a day" solution. This needs to deliver marketing emails at scale.
Does that mean you have to use SES, no. The code is open-source, swap it out for another provider if you truly believe that you can do it better. I'm open for contributions.
To this day I don't understand why email providers don't offer first-party Mail Merge facilities themselves.
AFAIK Microsoft is the only one that offers the feature with Word and Outlook in its Office suite, and even that is very restrictive (besides being downright inaccessible from the newer apps).
Because delivery rates through self-hosted SMTP are lower than through commercial e-mail delivery services; the latter being close to unity[1] and the former ranging from near-zero to about 7/8 depending on how much work and time you put into doing things.
1: My "close to unity" number comes from my experience with the mailgun APIs, I assume that SES is similar.
I absolutely hate how it feels like the e-mail delivery services and Google/Microsoft/whatever are working together to form a protection racket: sure would be a shame if your email were to get lost, better pay us to guarantee it makes it to its destination...
Imagine the outrage if your national snail mail carrier were to shred 50% of all mail which didn't have a Premium Protection stamp. Why do we let the big cloud hosters get away with it?
Because if you send marketing emails through SMTP, your delivery rate will be approximately 0.00%, or if you're leveraging a third party SMTP, they will ban you.
> or if you're leveraging a third party SMTP, they will ban you.
Not if you are paying them for the service.
There are ommercial email services that aren't google/microsoft/aws. My former company used to use one (can't remember the name) that provided you with smtp servers and a set of dedicated IPs per customers that were preused by them to build a good reputation beforehand.
I think the point is not necessary that you have to selfhost email (although it can be done, some people do), but depending on a single vendor through a proprietary service/API.
Email platforms are actually two platforms, composition, and delivery.
In the before times, I helped build and was a lead engineer on a team building a delivery platform from scratch, with very minor composition bolted on. (For a time reference, this was before HTML email was even common.)
Some of the lessons from then are very relevant still today. A big one that comes to my mind is 'idempotent' email isn't really possible, because SMTP has no 'send iff message-id has not been received' primitive. You can get to try-send-only-until-2.5.0-code, but even that can be non-trivial engineering. We went to a rather large degree of effort to do that ourselves, since repeat emails are a terrible signal, but I think thats only a marginal benefit to the larger operators.
Idempotent delivery even isn't a direct benefit to advertisers or most sending actors: Primarily, for the majority of the market, they don't really care about duplicated deliveries below some statistical model, since the costs are heavily externalized. Secondarily, since those parts of the industry depend on out-of-band read-indicators, engineering to improve delivery idempotentcy doesn't reflect into a market advantage.
Email's gotten prettier in the two decades since then, but its quality at every point in the stack I think has gotten worse. I routinely have to firewall whole ASNs because the bigger bulk operators (salesforce marketing cloud im looking at you) have no functional mechanism to call out particularly spammy customers, and thats after aggressive greylisting. Sure, the greylisting cuts out >95% of my spam, but the noise level on that last 5% is still horrendous. Forwarding with headers to abuse@ should be a universal paradigm but thats 99% a noop these days.
Given the lesser upside and relative difficulty of the delivery problem, it seems only natural that most of the effort in email platforms these days is in composition. Even if you want to work on the delivery half, with IP reputation, aggressive antispam, and the general enshittification of all things email, the barrier to entry is orders of magnitude higher than it was in 2000. I don't want to say its impossible, but I think you'll have a bad time if you're trying to do it anywhere that doesnt have enough size/inertia or money that email receiving operators choose to negotiate with you rather than preemptively block you.
Plunk's API does not appear to offer any such feature: https://docs.useplunk.com/api-reference/transactional/send
Unfortunately neither does Resend, Sendgrid, Postmark, etc.
[1]: https://docs.stripe.com/api/idempotent_requests