

Security Hole in Sendgrid - ndaiger
https://chunkhost.com/blog/15/huge_security_hole_in_sendgrid

======
jtchang
Social engineering will almost always work. I don't really fault Sendgrid for
this (though I could see this not working as well if you were using Amazon
SES...no support to even talk to!). It sucks that they got caught with their
pants down but I bet a good social engineering attempt on ChunkHost might have
yielded similar results.

The lesson here is to have multiple defenses. 2 factor auth is a great start
and it worked in this case.

~~~
jyap
You should fault Sendgrid as they specifically have a policy NOT to perform
this change of email request (from the article).

Sendgrid can also change their systems so that phone support personnel can NOT
perform this change or perform this change with approval from a supervisor.

Sendgrid being in the business they are in should also know that they are
susceptible to these types of attacks and what they can lead to (many, many
systems which can have password requests sent to email addresses).

~~~
sillysaurus3
I don't know... Mistakes happen. There seems to be little to gain from
faulting SendGrid, but faulting SendGrid would force them to take some kind of
action such as terminating the rep. I think I'd prefer the rep remain
employed, because I trust they'd never make this mistake again. Also, now all
the other reps know to avoid it.

EDIT: May I ask what can be gained from faulting SendGrid in this case?

~~~
cjbprime
I agree with you that terminating the rep is not interesting, and I think
you're mistaken if you feel like that's what anyone thinks will solve this
problem.

Actions SendGrid could take:

* Make it impossible for their front-line support staff to change the email address on file. If you want that -- which should be _extremely_ rare! -- you talk to a high-level manager who is competent at authenticating you.

* Send the email that says "hey, we're going to change your email address now" with a lead time to allow for the possibility that, even after your authentication, you've been conned.

* Make a phone call to the phone number on record, too.

You ask what's gained by faulting SendGrid, because you take it as a given
that they will make these changes. But that's not how blame works. The blame
serves a _function_ of ensuring those changes by holding them accountable for
their current problems.

------
staunch
Another title for this submission could have been:

"Massive Security Hole in ChunkHost. Non-2FA accounts can be owned."

Because it turns out anyone with a Sendgrid Support account also effectively
had _potential_ access to any account at ChunkHost not using two-factor
authentication. Which is also true of thousands of other companies that are
relaying their password reset emails through third party SMTP services.

SendGrid seems lame, for allowing this and for their response promising to
yell more loudly at their support people, but they're an SMTP relay service
not an authentication service.

~~~
np422
That would be because sendgrid is lame.

I've recently tried to use their service to send emails, club member
newsletters. I've never thought much about bulk e-mails before and I thought
it was a "solved problem" by now. Sendgrid are well known so they were my
first choice.

During initial testing I found both bugs in the API and missing functionality.
I've worked over 20 years with IT and yet sendgrid support is easily on the
top-3 list of my worst support experiences, a total waste of time.

We ended up using mailgun instead and so far it looks much better.

~~~
alanning
We looked into sendgrid and also went with mailgun. Been with them for over a
year now. Originally started with their mailing list API but had enough
service delay problems that we were considering switching entirely. Don't
remember exactly but may have had some outages as well. Their support
suggested we change to using their batch sending API instead and I'm glad we
stuck with them because it's been solid for over 5 months now. (Well they did
just have two issues recently, ssl cert and duplicated email sending but it
didn't affect us too much.)

Overall they've been a good service provider and their tech support has always
been helpful and pleasant to deal with. (I know! Crazy, right?).

------
MasterScrat
The problem is that it was technically possible, for a representative, to make
this change without the proper verification. You just can't rely on humans for
that.

~~~
dangrossman
SendGrid once went over my head and changed my account settings on behalf of a
customer of mine, without even consulting me beforehand. I had a customer with
a history of reporting mails as spam (activity reports he requested in a
webapp). When you report a mail as spam, the address goes into a blacklist to
avoid causing sender reputation issues. After the third or so time of having
him ask to get the mails again, then mark one of them as spam, I told him I
wouldn't be offering that feature to him any longer. He contacted SendGrid
about it directly, and the SendGrid rep actually went into my account and
added his address to a whitelist to bypass the blacklist, where I intended it
to remain.

I thought that was just the strangest thing, and it didn't sit well with me.
Accessing a customer's account when they request service is one thing, but
making changes on behalf of a stranger you _know_ has no authority over that
account? That was when I moved the last of my apps over to Mandrill.

------
cjbprime
Looks like a deeply unsatisfactory response from SendGrid. They don't even
know for sure ("it appears .. pretty much confirms") that their own support
staff changed the email address?

~~~
arjie
This shows why so-called PR-speak is necessary. Email like this should not be
in a conversational style because it can easily be misinterpreted.

~~~
rnovak
are you serious? How was it misinterpreted? It point blank says that that's
what happened? Please illustrate how exactly it was manipulated to say
something else?

------
hadoukenio
So what's the answer? Here's two very legitimate scenarios:

1) You sign up, enable two-factor auth, then lock yourself out (lost password
and your second-factor). How do you prove to the service provider that you are
you?

2) You sign up, enable two-factor auth, then Mallory claims that they locked
themselves out. How does the service provider prove that Mallory is not you?

~~~
lstamour
Text message or phone verification of details only you should know about the
account history, payment methods, etc. As others mentioned, a waiting period
during which they try to contact using any means s previously authorised for a
response. Compare IP addresses and deny logins from strange countries or
origins without further verification, etc. Of course, every measure and
countermeasure needs to be justified, since there's an implementation and
upkeep cost, but... It's possible to be "more certain," that something is
legit or not.

~~~
hadoukenio
Yep, so text messages and phone verification could really be considered a
"third factor". I guess anything information you have already provided to your
service provider is considered an X-factor.

------
toomuchtodo
Use Amazon SES to generate your outbound emails; ensure proper IAM policies,
and that you're using 2 factor auth to login to your AWS account.

~~~
balls187
Is AWS any less susceptible to Social Engineering attack of this type?

Specifically--can AWS support staff grant access to AWS accounts, and if so:

what are their criteria for doing so, and what are the policies in place to
ensure those criteria are met, and how are those policies audited?

As a TechStars alum, my company was granted $50k in AWS credits, which were
tied to my AWS account[1]. When I left the Company, the CEO was able to get
the credits moved to a different AWS account that was company owned, without
my intervention at all, even though I was the only account owner.

The fact that he could have credits moved out of the account without any kind
of verification from me[2], should be cause for concern.

[1] I should have created a new Amazon account for a group email [2] Obviously
the credits belong to the company; they weren't mine to use, so I would have
authorized the migration.

~~~
driverdan
Amazon (non-AWS) is infamous for being vulnerable to social engineering
attacks. I don't know if they've changed their polices more recently but they
are (or were) often the first attack vector for social engineering. If you can
get access to an Amazon account you can get the last 4 digits of the user's
credit card number(s). You can then use that info to reset accounts over the
phone with other companies.

------
jusben1369
What I found most interesting was they were targeted due to Bitcoin. Today
_most_ services would never store credit cards themselves but rather have them
stored at a secure payment gateway. So they're unlikely to be attacked to get
to those cards. T

Trust me I know that CC's are getting sniffed in transit too often so I'm not
saying they're 'safe'. I'm just wondering if there is something unique to
Bitcoin that suddenly makes you a target as though you were known to be
storing CC's data at rest onsite.

~~~
driverdan
Bitcoin is like cash. It's far more valuable than credit card data.

------
ef47d35620c1
Email accounts are the weak link for many things... DNS registrations, web
hosting, etc. Get the email account and you have it all.

~~~
danielweber
I'm coming around to thinking you should get a secret email address for your
sensitive accounts. Treat the string of that email address as almost as
important as the string of your password. The problem is that you do have to
tell it to people who probably don't share that philosophy to the same
extreme.

------
steven2012
We use Sendgrid and have hundreds of thousands of customers that might be
phished by this social engineering trick. It's absolutely unacceptable that
such a crucial piece of infrastructure is vulnerable to such a simple trick.

I'm going to bring this up with our team and see if there's another vendor
that can more reliably protect our customers.

~~~
mantrax3
Your logic is a bit weird.

Sendgrid just experienced this _major_ embarrassment and are currently re-
training their staff to avoid it again at all costs.

And you're going to move away from them _now_?

~~~
steven2012
I'll talk it over with my group. Retraining doesn't guarantee anything. It
could be that this problem is endemic to the company itself.

For example, why should 1st level support have the ability to make major
changes like this? It sounds like only 2nd level support, a smaller group of
more highly trained support staff, should have the ability to do this. Does
SendGrid have enough money/resources to split their team into 1st and 2nd
level support? Would a larger company have those type of resources that would
better protect my customers?

These are questions I will talk over with my team.

------
wojcikstefan
It's an important lesson for all of us. I've seen a lot of privacy ignorance
when it comes to support (e.g. folks handing over sensitive data after an
anonymous request on Olark). We should all go an extra mile and verify the
identity of the requestor.

1) If your support chat doesn't enforce authorization, always ask the
requestor to send you an email. 2) Make sure the domain is correct (that's
where Sendgrid screwed up). 3) Never agree on replying to a different email
address than the one of the sender.

------
gmansoor
BCC every message is evil, as it can be misused as in this case. SendGrid
should never allow that, or at least should flag such behavior. At the
minimum, they should notified account owners of this change.

~~~
icebraining
The attacker got SG to change the email on file, so the notification would
just be sent to him.

------
rootuid
It's not a security hole, misleading title.

Should read "social engineering resulted in security breach" and guess what,
this happens all the time whether sendGrid or not.

------
foxylad
My first thought was to whois chunkhost.info, which returns a clear name,
email address and phone number.

Or is it that easy to register a domain with a fictitious persona?

~~~
johns
It's that easy. You can use anything you want.

------
IgorPartola
So what they are saying is that SendGrid should have had two-factor auth and
this would have never happened.

~~~
Meekro
SendGrid's policy (as stated in their first email) is that the support people
shouldn't be changing account emails in this fashion. Even if SendGrid had 2
factor auth, who's to say the support guy wouldn't have just disabled that?

~~~
jrochkind1
If their policy is that support staff should _never_ be able to change an
accounts email address... why does the system let them do it?

~~~
IgorPartola
At some point someone has direct DB access and can do this. Sure, normal
support people don't but this just makes the social engineering aspect a bit
more complex, not impossible.

------
platz
[http://twofactorauth.org/](http://twofactorauth.org/)

------
hoodoof
SMS verifications should be sent out to allow over the phone changes.

------
davesque
Alas, the weakest part of the system is...

------
ogdenogly
Sendgrid seems to be its own worst enemy (Adria Richards debacle, now this).

I suggest we dub the act or state of being one's own worst enemy, being
"sendgriddled."

------
callesgg
Send your emails yourself.

It's not like it is hard. At least not harder than integrating to a third
party email sender.

~~~
je42
Sending email is hard and for most people not a thing that provide them with a
proper ROI, because among other things...

\- You need to make sure that your email servers IPs are not on black lists.

\- That you use DKIM properly

\- your multipart mime encoding is correct

\- bouncing e-mails are handled...

\- take care of scaling & operating the servers

\- and so on....

You can spent ( and waste ) a lot of time on this... especially if you are new
to email sending..

~~~
gtaylor
> \- You need to make sure that your email servers IPs are not on black lists.

This alone is a huge, huge chore. Especially if you run a hosted service that
allows some user-specified content in outbound email bodies. It's a lot
cheaper for us to pay Mandrill to handle all of that for us, provide us
excellent metrics and diagnostics, and let us know if one of our users is
sending junk mail before it gets out of hand.

~~~
danielweber
Even if you do _everything right_ you can still find yourself on a blacklist.
Then you get to pound sand.

Blacklists suck. Not as much as spam sucks, though.

~~~
vidarh
Frankly, with the amount of bullshit blacklists out there that real mail
providers actually trust, I've come to detest blacklists more than spam.

~~~
danielweber
Has anyone set up a blacklist full of random IPs as a joke to see how many
people follow it?

~~~
vidarh
Well, we _have_ been put on blacklists that the operator wanted money to
"expedite" removal from. I'm assuming that blacklist was pretty much random
IPs... And it did impact deliverability.

