
Mailgun Security Incident and Important Customer Information - hodgesmr
http://blog.mailgun.com/mailgun-security-incident-and-important-customer-information/
======
ad_hominem
When I get spam email, I usually check the headers and if it's coming from a
reputable service (Postmark, Sendgrid, etc.) they usually have a web form or
an abuse@ email to send the headers to so that they can shut down the account.

Months ago I received spam from a Mailgun server and tried to use their web
form[1] to report it, but it was broken. I reported both that bug and the spam
email to their support, which acknowledged it. Weeks later I got another spam
email from that same domain, popped open that report form and it was still
broken (FWIW as of today it seems to be working again). So I followed up on my
initial support request with that info but got no response. Just a few days
ago I received another spam message from that domain.

I personally consider all that a very bad sign in an email service provider
and wouldn't use Mailgun myself. In contrast, I've been very happy with
Postmark.

[1]: [https://www.mailgun.com/receiving-spam-from-
mailgun](https://www.mailgun.com/receiving-spam-from-mailgun)

~~~
aceoflala
Postmark costs money, Mailgun does not.

~~~
ad_hominem
Postmark is free up to 100 mails / month. But mail deliverability issues are a
hell that I'm happy to pay a small fee to avoid.

~~~
eikenberry
Do they have an overage charge for the free 100/month, like the 1.25/1000 they
list for their non-free use? So you could be free most of the time with the
occasional 1.25 charge if you have a busy month?

~~~
ad_hominem
Unfortunately I don't think so. :(

> _We offer a Free Trial plan for testing purposes only. The Trial is limited
> to 100 emails a month with no overages allowed._

[https://postmarkapp.com/support/article/1107-how-does-
monthl...](https://postmarkapp.com/support/article/1107-how-does-monthly-
pricing-work)

------
r1ch
This was used to steal bitcoin cash tips on Reddit by hijacking password reset
emails
([https://www.reddit.com/r/bugs/comments/7obxkb/mailgun_securi...](https://www.reddit.com/r/bugs/comments/7obxkb/mailgun_security_incident_an_update_on_the_state/))

I find it amusing they still have a "trusted by Reddit" blurb on their
homepage after this!

~~~
codelitt
I don't believe this would even be an issue if they offered the option to not
log sensitive data. I had requested that they provide something like this and
someone quite senior reached out to me. He was very polite and professional.
He explained that they had to keep this data for operational and compliance
reasons and that all email providers are required to. However, that didn't
resolve my security concern.

We ended up going with Mandrill which does offer the option to not log
sensitive data ^1. Whether they log it somewhere else for the compliance
reasons that Mailgun mentioned isn't mentioned anywhere in their docs or
privacy policy, but doesn't seem to be accessible from everything I could
find. You should never log or allow others to log password reset urls or other
sensitive details.

1: See documentation here:
[https://mandrillapp.com/api/docs/messages.JSON.html#method-s...](https://mandrillapp.com/api/docs/messages.JSON.html#method-
send) and search view_content_link

~~~
reaperducer
More and more "compliance" is an IT industry excuse for "because we want to."

~~~
sgrove
Honest question, why would you "want to" adhere to compliance? It's almost
always more work and more cost, _I think_.

~~~
nuggien
The cost of paying fines for non compliance would be more.

~~~
sgrove
Exactly, that's very different from "because we want to," it's, "because
there's a very big stick over our heads if we don't."

I just thought the attitude/assertion was in discord with my own
experience/understanding.

------
clon
Why would employees need access to client API keys, as opposed to just client
ID?

Furthermore, this seems to indicate that the API keys are not hashed. I would
expect some bits of the API key to work as an identifier and the rest of the
bits treated as secret material (properly hashed).

As a Mailgun customer, this is concerning..

~~~
somedickhead
As a former Rackspace employee, I had access to every customer secret IN PLAIN
TEXT through multiple web-based systems with a click of a button (IE: business
as usual).

~~~
twunde
Thanks for confirming this. I had my suspicions, especially after the last few
years of using them and just seeing massive problems that seemed to be caused
by the software at Rackspace.

------
OJFord
Er, can we expect more information to follow?

1\. How was the employee's account accessed? No 2FA?

2\. Do employees ordinarily have access to customer secrets (e.g. API keys) or
was there some further exploit?

3\. The advice in OP for affected customers is to roll keys and SMTP logins.
Couldn't/shouldn't you do that for them? Surely security should trump up-
time/deliverability?

~~~
somedickhead
All Rackspace employees are issued hardware or software RSA tokens and a VPN
client.

I seriously suspect this was the job of an insider, not a compromised employee
laptop.

~~~
ralphm
MailGun has been spun out of Rackspace almost a year ago.

------
rcMgD2BwE72F
Does this only affect Mailgun's customers? If these customers hold data of
third-party – let's call them "end-users" – in Mailgun accounts, Mailgun
could/should communicate the total number of individuals affected. "1% of our
customers/users" can affect millions of individuals.

------
gouggoug
In those security disclosures, I often read what I see as contradictory
language.

For example, I'm confused by this kind of statement:

> Mailgun has now completed its diagnostic of accounts that were affected and
> has notified each of the affected users. At this time, we believe less than
> 1% of our customer base was potentially affected. If you were not directly
> notified by Mailgun regarding this incident, then your account was not
> affected.

If you _believe_ that _less than_ 1% of users were affected, it means you
don't know for sure how many accounts were affected.

From there, how can you state that "If you were not directly notified by
Mailgun regarding this incident, then your account was not affected"?

Doesn't this last statement mean you know for sure my account was not
affected? Isn't it in direct contradiction with the previous statement?

~~~
Goopplesoft
Foremost, it was written by a human and unintended language contradictions are
common. With that said, what you're suggesting isn't necessarily true -- the
language can also indicate potential false positives, again because of the
nuances of language.

~~~
gouggoug
> unintended language contradictions are common

Yes, definitely true. Although some contexts, like a security disclosure,
might warrant a very carefully non-contradictory worded statement that leaves
no doubts of interpretation.

> the language can also indicate potential false positives, again because of
> the nuances of language.

Yes, but in this context, false-positive aren't important to the audience of
the disclosure. Nobody really cares if their account was "identified as
affected, but in the end wasn't".

If you announce that 1% of your user base was affected, and it turns out that
50% of this 1% were false-positive, great! You were still right in announcing
that 1% of your user base was affected. You can always correct this later and
announce that things panned out better and only 0.5% of your users were
impacted.

------
devicenull
No 2FA on staff accounts?

------
rajeemcariazo
I like Mailgun so much because of its simplicity but last November 2017 the
default postmaster account of one of our domain in Mailgun was hacked. (I
don't know where it was hacked but i suspect it was on the Mailgun server
because I kept the secret key in my server very well). We moved to Sendgrid
because my account in Mailgun got a very bad reputation. One of the hacked
smtp credentials was used to send spam.

------
Gys
> At this time, we believe less than 1% of our customer base was potentially
> affected. If you were not directly notified by Mailgun regarding this
> incident, then your account was not affected.

------
ppierald
> Finally, we’d like to assure our customers and partners that we take
> security at Mailgun very seriously.

So very seriously that they don't even use https for their blog...

~~~
a_imho
Former mailgun customer. Asked them to delete my personal data a couple of
weeks ago (I was not able to do it myself... ) because I would rather they
don't leak it in a security hiccup. They kindly refused to do so (as I don't
believe any tech support can be that incompetent) and kept spamming my inbox
instead. While the severity of this incident is not clear, never imagined
curses can act on such a short notice.

~~~
DarronWyke
This is because Mailgun is in the practice of spam. The number of spam
campaigns I've seen with Mailgun as the conduit is high, second only to
Mailchimp.

~~~
bwag
To be fair, Mailgun is in the practice of sending email. It just happens to be
that email is one of the main conduits of spam.

~~~
DarronWyke
No, that's just correlation. Email can be spam, but not all email is spam.

------
ram_rar
2FA, 2FA, 2FA!

------
MechEStudent
Only 1%? My eye. This has smell of Yahoo to it. I bet within 6 months, this
goes up toward 10%. I bet they lost their entire data.

