Hacker News new | past | comments | ask | show | jobs | submit login
Mailgun doesn't validate the email headers of email sent through their system (twitter.com/radiotoolbox)
147 points by kxrm 66 days ago | hide | past | favorite | 79 comments

Mailgun used to be a HN favourite, great service, great customer support, and quickly resolving issues / providing solutions to problems like this (and many others). Email is hard, and mailgun made it less hard, helping you solve problems along the way.

Several years ago, the customer service got worse, and the fixes / solutions stopped coming. I don't hold my breath that this (or any other issue) will be solved any time soon sadly.

What service do HN recommend these days?

Postmark, for sure. Run by a small dev shop called Wildbit that is bootstrapped, treats their employees fairly, and grows in small sustainable ways.



What's the advantage here over SendGrid? SendGrid has an unlimited free tier of 100 emails per day, and for $15/mo you can send up to 40k emails/mo.

Postmark does have a lower $10/mo starting price for 10k emails per month. But the next jump is $50/mo for 50k emails.

I'm currently working on a project and intended to use SendGrid, so I'm wondering if there's any benefits of Postmark.

Recently opened a SendGrid account on their $14.95 tier for a new project and found the shared IP blocked by MS domains (outlook.com, live.com, hotmail.com). MS servers confirmed the reason in the SMTP negotiaton as owed to the IP/provider.

We reported it to SendGrid and their only option was to upgrade to their $89.95 plan to get a dedicated IP. That plan comes with 100K monthly sends and we are nowhere near that.

So, the choice was to have a significant portion of important transactional emails, like registration, not go through or overpay for a plan that is wildly overmatched for us.

Email is hard, but this borders on unethical. Customers pay for and integrate a service that simply doesn't work as advertised. They make no offer to mitigate (e.g. change to a new shared IP). It's just "oh yeah, if you want the service to actually work reliably, you need to pay us 6X more".

You often pay for "good company" when you choose a more expensive provider. So the cheaper the provider the lower quality of their shared IP pool due to the type of customers they tend to have on the platform. The free tier lowers the customer quality even more. IF you are sending mission critical email then dedicated IP makes a lot of sense, but probably not at a huge price multiple mentioned for Sendgrid.

Amazon SES shared IP pool isn't better than Sendgrid or Mailgun but their dedicated IPs are only $25/month. If your volume is low that might be the only cost since they offer 62k/month completely free and after that it's $0.1 per 1000 emails.

Unfortunately not a lot of providers offer dedicated IP option on lower level plans.

>Amazon SES shared IP pool isn't better than Sendgrid or Mailgun...Unfortunately not a lot of providers offer dedicated IP option on lower level plans.

I think the main problem I have with SendGrid here is that they knowingly offer a paid product with such abysmal shared IP deliverability out of the gate, then offer no meaningful mitigation except to upgrade to a plan that includes the dedicated IP at a 6X premium, along with features many customers don't need (primarily send volume). Many customers will say "that's overkill, I don't need that send volume".

Very much a bait-and-switch feel to it. If they're going to offer the lower cost plans, then they need to be forthcoming about the disparity in deliverability. Instead, they say vague things like "own your reputation with a dedicated IP", when the truth is actually "12%-15% of your emails may suddenly stop being delivered unless you choose a dedicated IP plan".

Publish metrics and let people choose.

Same story here, but being a Sendgrid customer for 6+ years, suddenly the shared IP is blocked as well. After opening a lot of support tickets and getting no response, I had to nag someone here in HN who mentioned working for Sendgrid, he escalated but the response was the same: pay a lot more to get what you used to have.

Migrated to Postmark right away and we've been a happy customer for 2+ years now.

Yeah, I'd used SendGrid's higher tier on other projects for years, as the send volume justified it. So, generally had the dedicated IP address.

They need to do a better job of managing their shared IP pool. As it is, they are offering paid plans that are unsuitable for many common use cases. Really, unless you have control of all possible receiving domains (e.g you're using it for an internal app), you're rolling the dice.

Else, at a minimum, they should disclose deliverability metrics on their various plans so customers can make informed choices. As it is, their marketing is deliberately misleading.

Thanks for the feedback on Postmark. I'll have another look at them.

EDIT: Just glanced at Postmark and they're already looking much stronger than SendGrid, and with much better pricing. The "deliverability without a dedicated IP" language seems to be directly aimed at providers like SendGrid. Are they able to live up to that promise?

Also like their policy around content retention.

Will be exploring switching costs.

> Are they able to live up to that promise?

Postmark product manager here, so I'm a little biased but... yes we are! You can see our Time To Inbox stats on our status page: https://status.postmarkapp.com/

We also have a handy migration guide here: https://postmarkapp.com/migration-guides/sendgrid

Thanks for the heads up. You guys do a good job on documentation. A little redundant on the marketing side, but overall very clear. That migration guide is very well-done/thorough.

Off topic: What does "he scaled" mean in this context?

I'd guess "escalated", meaning engaged someone higher up in the company with more authority.

To me, there is one big difference with postmark. Their focus is on transactional emails. Not marketing emails. I believe they have a separate service now for marketing emails, but the big reason I chose them is that they are very focused on keeping their IP addresses clean. One way they do this is by only allowing users of their service to send transactional emails like invoices, password resets, or thing that have been specifically requested by users. I assume they keep all that on IP addresses reserved only for that. Most people won't mark a password reset email as spam. But they will frequently mark recurring marketing emails as such.

I use Postmark for everything that is very important that it gets through. I actually use mailgun for mass emails, marketing, announcements, etc.

I opened an account with Sendgrid for a new project the other week. Account got banned right away, didn't even log in one time and they said to contact support to get it unblocked. Only issue is to contact support you have to be logged in and their alternative form also wants you to log in.

Sendgrid is not friendly.

Are you using linux, by chance? Recently I've noticed Amazon, Google, and others reporting my logins from linux as "suspicious activity". I've never logged-in from another OS, so I'm wondering if they're all relying on some third-party service that automatically equates linux with "suspicious".

Yet Amazon, Google, and others all use, produce and offer Linux themselves. How can logging into a server running Linux from a desktop/laptop/handheld computer running Linux be "suspicious" activity. Is Android not Linux.

I am on Mac OS on a residential IP ¯\_(ツ)_/¯

It could be that Linux drive spam bots are using their system.

Spoofing the user agent is probably the first thing these spam bots would do.

Heavy SG user here and while it's still pretty functional and not problematic enough to move away from, it's not as good as it used to be.

Before being acquired, SG's support was fantastic. There are also various oddities that have cropped up, such as how they're currently hitting our webhook endpoint with the same event every 20 seconds for the past 8 days (a similar problem occurred last year too). They also seem to have suffered some dings to their IP reputation in numerous places due to, I assume, this: https://krebsonsecurity.com/2020/08/sendgrid-under-siege-fro... .. I continue to encounter numerous systems that flat out refuse any email from any IP I can muster at Sendgrid and we have a bunch on different subnets. So anyone who works at Red Hat, Packt, Akqa, Zendesk, etc.. they're not getting our mail.

We use Postmark as a fallback for users when we get the inevitable error messages back, and they have been pretty good, although I find their API a little too slow to move everything over to them and SG has a fantastic "delayed send" feature which is a must for deliverability to iCloud addresses.

Pragmatically I prefer Sendgrid, but Postmark is very good and feels somewhat more wholesome to use, particularly if your levels are low. I'd use Postmark for the same reason I'd rather use a local bakery than buy presliced at a supermarket.

> although I find their API a little too slow to move everything over to them

Postmark product manager here... I'd love to know more about this. We publish our API response times on our status page (https://status.postmarkapp.com/), so this is a surprising comment. Can you email support@postmarkapp.com with your account details and we'll look into this for you?

To clarify for the benefit of any onlookers, I do not mean to say that Postmark's API is slow absolutely, but that it is slow(er) for how we use an email sending API relative to SG.

Our email system is custom and does all of its own merging on a per subscriber basis, so we send a distinct request for every email, so even an otherwise minor difference between a 30ms and 85ms round trip "feels" slow at scale. I believe PM has a "batch" call available, but this requires some extra dev time at our end if it would resolve our issues (and it may!)

I will get in touch, though, in case there are ways to help mitigate this problem or if the batch mechanism will solve it all. We have previously encountered similar performance issues with SG from time to time due to DNS sending us to unsuitable endpoints, etc. so it can be a universal problem to some extent based on how we've approached the problem.

(Edit: Actually, that status page maybe demonstrates what I'm talking about, since the average API response time shown is 437ms. Even with 5 concurrent threads that would take 2.42 hrs to send 100,000 emails - so I am guessing the batch API is the way to go with PM :-))

Yep /batch is definitely the way to go! But feel free to reach out to support and we can take it from there.

> What's the advantage here over SendGrid?

I am not familiar with products mentioned above but SendGrid is often discussed as a source of spoofed messages. They allow sending as 3rd party domains, or at least allowed it until recently. I suspect that this may be affecting their deliverability.

Postmark has way better visibility into what is being sent. You get per-email history/status, so you can easily tell if something is queued or bouncing. Trying to debug errors on Sendgrid is less fun.

On the other hand, postmark is a bit more flaky - I track delivery RTT and had had a few cases in my first year with them where Postmark had > 30m delays (and no outage reported). Sendgrid has always been bombproof on delivery times over many years of usage.

I still prefer Postmark on balance.

> On the other hand, postmark is a bit more flaky - I track delivery RTT and had had a few cases in my first year with them where Postmark had > 30m delays

Postmark product manager here... I'd love to know more about this. We publish our Time To Inbox on our status page: https://status.postmarkapp.com/

The only time emails might sit for longer than a few seconds is if there is some issue with the account (such a accidental spam being sent due to form abuse). If you have specific examples, can you please email support@postmarkapp.com and we'll look into it for you?

i use sendgrid for a small project because it's free and has template support, but the site is unbelievably unresponsive. it takes 30+ seconds for anything to load. we have a paid account at work and it's exactly the same, so it's not my connection or the free tier. my assumption is that twilio has left it on autopilot.

Second this 100%. Postmark is my go-to for every project

Why not use AWS ses? It's so much cheaper.

One important decision argument against AWS SES is their policy to keep bounce rate below 5% (account put under review, if unresolevd until end of month, will be suspended, with hard limit of 10%) [1] compared to least strict Postmark's bounce rate of 10% [2].

Sometimes for SAAS products with a huge userbase or freemium pricing model is super difficult to keep the bounce rate so low for transactional emails.

[1] https://docs.aws.amazon.com/pinpoint/latest/userguide/channe... [2] https://postmarkapp.com/support/article/1137-servers-faq

Policy like this is meant to protect sender reputation, not just provider IP pool. High hard bounce rate lowers your inboxing in the long term. If your bounce rate is consistently above 5% consider verifying all new emails before sending them an email > https://www.bigmailer.io/blog/email-verification-service-pro...

Natural email decay rate is 2-3% per month so the goal is to stay close to this range, but the lower the better.

UX and support?

Their customer support is really top-notch!

Mailgun was acquired, which sometimes has a side-effect of product quality decreasing.

I wish Cloudflare would launch an email API service, seems a good fit for them.

A private-equity company bought them couple years ago, and felt like the quality took a hit after that.

Looks like they were recently aquired by another company trying to compete with Twilio. So maybe things will improve?


What is it about private equity that seems to systematically rot businesses from the inside-out?

I can answer this question from experience as a grunt, and opinions that may not be exactly correct as I've never been on the PE company side myself.

The company I work for was bought by PE. Within weeks, most our hireable people left. Perhaps they were scared, perhaps they've seen it before, who knows. Some good people stay, but usually the comfortable ones, not the ones changing the world. How do I know, because I consider myself one of the competent comfortable ones, just being honest. But I'm leaving this year, it weighs on your soul. I barely do anything. At first it sounds like a dream, then you realize you're bored, not learning, and not working your brain.

Then, those useless asskissers you assumed wouldn't last long rise like cream to the top. They have no idea what they are doing, but are good at placating, and take positions of power.

From my perspective, it's easy to think PE people are dumb to let that happen. Maybe they are. Or maybe they don't care. Or maybe they are smart because they know these placaters will help 'sell' the company when it comes time to dump it. Make no mistake. PE exists to dump your company on some sucker.

I built and launched my own provider: https://ohmysmtp.com

I've been using Amazon SES for a few years with my larger clients. No real complaints about deliverability and since we're aggressively not spam, haven't run afoul of any of their automated tools.

Another +1 for Postmark. Reliable and insightful.

What really sold me personally as a bootstrapper was how they offer initial credit for their services to help get going.

Would be interesting to map at which point in time customer service comments turned from positive to negative. Not that I’m suggesting PE buyouts typically lead to this…


Mailgun CTO here - I'm sorry that we've let you down. I'd love to better understand any issues you're having. My email address is in my profile.

If accurate this means any mailgun user can pretend to be another one when sending email out- that's pretty damn bad. Since companies add mailgun to their SPF/DKIM records it means those spoofed emails will be hard to distinguish as fake.

I have been trying to tell people that this is happening for years. More about that spoofers were doing it, not what tools they were using

I didn’t know I needed proof because people often resorted to victim blaming even though I never fell for any of the emails

I think it's difficult to prove, and the only people who care will be the domains owners that are impacted. If you send enough mail, you won't notice a few extra emails going out from another account.

Mailgun should have a way to monitor and block this. We used SMTP to interface with Mailgun and frankly, I didn't even think of this as being a vulnerability until I left the service. The DMARC reports just prove it was happening.

Josh/CTO of Mailgun here - I took a look at this case and these messages were being sent from the sandbox feature of our service. The sandbox only allows messages to be sent to "authorized recipients" through an opt-in mechanism, which makes it impractical for spamming and phishing. The purpose of this feature is to allow customers to get comfortable with the interfaces and test the product in a safe way without having to add DKIM/SPF records. I reviewed these messages and confirmed there is no illicit behavior, likely just a misconfiguration from a user of radiotoolbox.

I know it's accurate, I had another domain on my machine that was sending email through my account, on accident, and Mailgun did nothing to stop it.

Judging by the reputation of the new Mailgun owner, I wouldn’t be surprised to find out this is a feature for scammers, not a bug.

I wish Cloudflare provided an outgoing email API service. Seems like a good fit for their customers, and I bet they would take security more seriously than Mailgun.

In UniOne.io, Mailgun rival, we ask each our customer to add separate DNS entry for a new domain proving that he/she owns the domain, and don't let another customer send using this domain.

Do you have a product that competes with MJML? Building email templates is hard - email client markup sucks.

If the point is getting beautiful responsive emails without low-level HTML coding - well, yes, we have responsive email template builder integrated. Just with drag-n-drop you can get a nice template and then use it for your transactional or marketing emails.

I was recently looking into Mailgun's EU-hosted offering. Does anyone have recommendations for EU-based alternatives?

Mailjet is a French company with a lot of compliance parlance in their website. They are now under the same company as Mailgun (Pathwire) and there are some weird mentions of Mailgun in the Mailjet docs, so I don't know where they're heading.

I was considering using Mailjet, but from the docs it looks like the inbound processing is not as sophisticated as Mailgun: we can't set inbound routes dynamically according to the destination email address.

In my initial investigation, I actually concluded that "Mailgun" and "Mailjet" were two product names from the same organization. I didn't realize they could be the result of a company merger. It's actually Mailjet with its outbound email templating service that I was considering using.

I‘m looking for an alternative to send mail and receive inbound mail. I‘d prefer to connect my domain so I can send from an API+interface (like gmail). Google does only allow 300 or so mails per day through SMTP. I‘d like to send more like 1000+ through an API (I’d like a combination of mailgun and gmail) Is there a service like this out there?

Mailgun does this, however I wouldn't recommend any of these services unless you are willing to fork out the cash for a static IP.

The free tier or shared IP side is where these kinds of shenanigans can be played out.

Yes, but they don‘t provide an interface for inbound mail, only redirecting, right?

They have inbound on their paid tiers.

Have you looked at Office 365/Outlook? A $100 a year subscription gets you a lot of firepower.

Mailjet ?

Mailgun doesn't even validate their own headers. I get daily emails from mailgun, which pass every check (spf, dkim, etc).

Reported it a bunch of times, but gave up. No response, no fix... unfrotunate, because mailgun was a reaaally cool product

Another anecdote of mailgun issues: we recently had an issue where a staging environment of ours started sending mails to real people.

We use this staging environment to do as-much-as-possible end-to-end dry-runs of real work, and thought that mailgun's sandbox functionality would be an ideal fit for this - we get to test everything up to and including the mailgun api to find regressions or other issues.

...until it started sending real mails; and when we talked to their support staff the response was the frankly astonishing "yeah, that's by design, sandboxes will send real mail out of the sandbox when under unusually high load" (paraphrased).

Seriously, W.T.F.

Now, of course we can and do do normal testing with entirely fake data, but data-distribution dependent failures happen; that's kind of the whole point of a staging environment. I'm not sure what the point of mailgun's sandbox even is if it's useless for staging.

Josh/CTO of Mailgun here - Can you point me to any support tickets referencing this issue?

Well.... thanks for ruining my weekend...

Gonna be fun on Monday looking into this, talking with leads, and looking for Mailgun alternates.

If you pay for a static IP and tighten up your SPF records to just that IP rather than using Mailgun's include, you should be fine.

It's bad that their service allows this at all, but it isn't the end of the world for all of their customers.

What trust do I have that a static IP would fix this?

If the bug is before any routing to which server to send from takes place... then it'll just end going out on the static IP all the same..

I was on the free tier so I don't know for sure, but I believe the whole point of the dedicated IP offering is you are the only customer who can use it. So long as your SPF records reflect just that IP SPF will remain in force if someone tries sending through one of the shared IPs as you.

But if moving isn't a huge deal, there are plenty of recommendations in these comments of alternatives.

The Mailgun support also says DMARC records prevent this, when the reject policy is set to deny.

Any Mail Server I encountered allowed you to put anything you want as the From header. With Microsoft Exchange being the exception.

The problem is that those basic fakes are normally rejected by the spam filter because they fail the SPF/DKIM checks on the receiving site.

If you actually use mailgun yourself then you add mailgun to the list of permitted senders and probably add a DKIM key as well. When someone else then fakes your email address without any validation, mailgun might sign the message and bypass every anti spam measure there is.

This could be solved without validation (for example by using dedicated IP addresses and DKIM keys per domain and attaching those to your specific account and API keys) but that'd take some significant engineering effort (and address space, loads of servers are still configured IPv4 only for some reason).

From the screenshot, I gather that the DKIM checks failed already. That still makes mailgun an open relay, though, so they should be added to the necessary IP blacklists if they can't fix this problem.

True, but do you just allow them to borrow the reputation of your other customers? If a scammer knows example.com is a customer (DNS records will tell me this quickly) and a scammer decides to open an account and send as example.com through the service. Does Mailgun have no responsibility or ability to stop this behavior?

Seems easy enough to stop from what I know of email. Otherwise DKIM and SPF are worth nothing in systems like this.

Mailjet stops this by simply requiring domains to be registered to accounts before an account can send from that domain. Registering a domain requires adding a DNS record or file on that domain with a random unique guid

Yep, and Mailgun requires a domain as well, they just don't do anything with it for filtering this behavior.

Josh CTO of Mailgun here - In this example, there is no actual sharing of domain reputation with other customers. Most inbox providers base reputation on the DKIM domain, which in this case is "sandbox.mgsend.net" not the header from address.

Each message is uniquely signed by the sending domain belonging to the account owner, meaning that the domain reputation cannot be borrowed/shared between customers.

Hm, strato, 1and1 and t-online only have one mx and spf record for all customers. Allowing you to send as anyone they have in their system and make it look authentic.

Surely they can cut off emails before it reaches the SMTP server though? Don't they have a milter that drops email sent by accounts that don't have the domain registered? They don't really forward spoofed mail do they??

They do, they only check if you are allowed to log in.

Exchange even has a feature built in for exactly this, their send connectors can’t store more than one account.

If you're worried someone might spoof your domain, then use DMARC.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact