
Reading Uber’s Internal Emails: Bug Bounty report worth $10K - vinnyglennon
http://blog.pentestnepal.tech/post/149985438982/reading-ubers-internal-emails-uber-bug-bounty
======
zaroth
I found this write-up a bit confusing and hard to follow.

The vulnerability is that _any_ SendGrid user could configure a webhook
callback which would POST back all received emails for _any_ domain which had
its MX set to 'mx.sendgrid.net'. OP exploited this against Uber to receive
copies of their emails.

Presumably there was no way to tell from one account that another account is
web-hooking your email out from under you. So you have to wonder, if it's as
easy as just typing the domain you want to listen in on.... who else was
getting all their emails tapped this way?

From Sendgrid's documentation;

    
    
      Setup
      The following steps are required to begin
      parsing email:
    
      Point the MX Record of a Domain/Hostname or 
      Subdomain to mx.sendgrid.net
    
      Associate the Domain/Hostname and the URL in 
      the Parse API settings page.
    

Shocking omission by Sendgrid, where's their write-up and apology?

~~~
uraniumhacker
At the time of the exploit, there was only one check, the MX record. Also once
I claimed the domain, another user cannot claim it from their account. So this
could be exploited silently as well because the victimized company won't know
about it for a while.

------
ndaiger
Sendgrid allowed attackers to social engineer control of my company's account
and intercept password resets, despite an explicit warning from us a week
prior (we received a chat transcript of the failed attempt and let them know
that it was not us and someone was actively trying to social engineer access
to our account).

Then they had the gall to try to convince me on the phone that it must have
been my fault (after our blog post about it blew up).

Needless to say, I think they are terrible.

Previous HN discussion:
[https://news.ycombinator.com/item?id=7476836](https://news.ycombinator.com/item?id=7476836)

~~~
andrewvc
As a former customer (it has admittedly been a few years) I'm not surprised.
Sendgrid's entire business is based on price. Emails are one of the few costs
that don't really scale that well. They cost a lot more than people think they
should. When I list did a cost analysis they beat out a lot of other
providers.

I haven't used them in years, so I'm not sure if they are still the low cost
leader, but they are definitely the kind of thing where you get what you pay
for. Their lack of investment in product and infrastructure showed for the
years I used them, and their slowness in delivering updates was incredibly
frustrating.

~~~
philliphaydon
Who would you recommend then?

~~~
jc4p
I'm not the person you asked but I too have had bad experiences with SendGrid.

It really depends on which aspect of SendGrid we're talking about.

Transactional e-mail? Mailgun is a easy to use API on top of Amazon's SES.

You can even set up incoming e-mail hooks e.g. "When a new e-mail arrives,
POST the contents to this address and attach any attachments on the e-mail as
file uploads."

Newsletters? Drip campaigns? I have less experience on this side of the realm,
but HubSpot has been the best experience I've seen. Their web management UI is
also powered by their own open API, if I remember correctly.

~~~
jjeaff
I like mailgun, but I am having a devil of a time with them sending to MSN and
aol addresses. Basically 0 deliverability. And their tech support is going on
4 days without looking at my ticket. But maybe it's not their fault.

------
ComputerGuru
The last time there was a SendGrid article on here, the feedback from the
community was far from kind [0]. I again re-iterate that SendGrid has no
business sending emails [1].

[0]:
[https://news.ycombinator.com/item?id=12142728](https://news.ycombinator.com/item?id=12142728)
[1]:
[https://news.ycombinator.com/item?id=12145019](https://news.ycombinator.com/item?id=12145019)

~~~
dinersanddrives
This! I actually started using them when they helped me out a hackathon but
switched to a more dev friendly service.

------
tyingq
Ouch. No domain verification required by Sendgrid before allowing you to
inject a hook that dumps email contents.

That's much broader than just Uber.

Edit: Yes, it's been fixed, but the fact that it existed for quite some time
is still troubling. I'm also curious if the fix retroactively disabled any
existing unverified hooks.

~~~
innoying
I believe they did retroactively search for accounts.

Source: I had a number of accounts banned when testing different iterations of
this bug.

------
ransom1538
Totally just curious. The law for "exceeds authorized access" is 20 years in
prison. I think uber has a bug fixing program. Maybe sendgrid does? Do the DNS
carriers? Does his ISP? I read the law. If one of these companies refutes
access - this guy is facing 20 years [1]? W(why)TF do people do this? The US
government is notoriously creative in these prosecutions [3]. Companies refute
access all the time to not look like idiots EVEN WITH a bounty program. [2].
If I came across this I wouldn't be blogging about it for 10k?

[1] "(2) intentionally accesses a computer without authorization or exceeds
authorized access, and thereby obtains—(C) information from any protected
computer;" (C) except as provided in subparagraphs (E) and (F), a fine under
this title, imprisonment for not more than 20 years, or both, in the case of—"

[https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act](https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act)

[2] [http://motherboard.vice.com/blog/facebook-is-refusing-to-
pay...](http://motherboard.vice.com/blog/facebook-is-refusing-to-pay-a-bug-
bounty-to-the-researcher-who-hacked-zuck)

[3]
[https://en.wikipedia.org/wiki/Aaron_Swartz](https://en.wikipedia.org/wiki/Aaron_Swartz)

~~~
tyingq
Uber's bug bounty program seems like explicit permission to me. The twist
here, as you mention, is that it is sendgrid's infrastructure. Chasing bug
bounties does seem a bit risky in this "cloud" era where it's not just one
entity running the targeted service.

~~~
beejiu
It is not listed as an in-scope domain on Uber's bug bounty policy. You would
have to argue that the access was unintentional - that you believed you had
authorization to access the server from Uber and that Uber had the ability to
assign that authorization on behalf of SendGrid.

~~~
tyingq
That is a bit tricky. You could argue that the MX records for a domain that is
"in scope" (www.uber.com) are in play. They don't specifically say you can
only follow CNAME and A records...and you do have to follow CNAME for some of
the listed domains.

~~~
otakucode
Unfortunately, the law simply does not care about technical details. If you
were being prosecuted for this sort of thing and you brought and expert
witness or testified yourself about a technical justification such as this,
you would be accused of attempting to bamboozle the court with fancy technical
wording and it would not go well for you. You might even get found to be in
contempt of court. Even the very simple fact that 1% of DNA tests have been
shown to be false due to sample contamination and other factors is not
permitted to be mentioned in court when DNA matches are being used to
prosecute someone as this is too technical. Cases which involve computers fare
even worse.

~~~
tyingq
I'm sure a lawyer would find some non technical analogy to help explain the
concept.

It isn't a trick...Some of the approved domains have no IP addresses directly
associated with them. You have to traverse a reference to another domain name,
not on the list, to get to an actual IP address.

Now if that traversal is by something called a CNAME or something called an
MX, does it matter?

------
mikesea
Someone reported this same vulnerability to us via HackerOne _months_ ago. We
worked with Sendgrid support to re-claim the domain and they said they were
urgently working to fix the issue, or not.

Edit: just saw this post was from September. Author probably made thousands in
rewards circulating this vulnerability.

~~~
innoying
I do not believe the author circulated this report to multiple companies,
however once it was made public a number of other reporters in the community
did and continued to iterate on it until SendGrid fixed the issues.

Source: I am a member of said community: [https://bugcrowd.com/bored-
engineer](https://bugcrowd.com/bored-engineer), [https://hackerone.com/bored-
engineer](https://hackerone.com/bored-engineer), etc

------
bdittmer
This is a lot like a bug I found in Heroku's system a few years ago.
Basically, if someone doesn't claim the wildcard subdomain for their primary
domain and has a wildcard SSL cert anyone could (can?) claim subdomains. A
quick google search yielded hundreds of exploitable domains. At the time it
seemed like a pretty big vector for phishing.

I have no idea if they fixed this and they gave me a t-shirt.

~~~
brandur
I can't recall all the exact details, but there is some validation logic in
place along the lines of "if there's a wildcard domain installed then newly
added subdomains must be on the same account as the wildcard's owner". You
could give it a shot, but I don't think this attack would work.

(I used to help maintain the system responsible for this, but don't work there
anymore.)

~~~
bdittmer
This was probably 2012 or 2013. I'll try to find the write up I sent them, but
this check definitely did not exist then

edit: here's what we sent to heroku
[https://gist.github.com/bdittmer/6461b7a5093acd7d6263](https://gist.github.com/bdittmer/6461b7a5093acd7d6263)

------
gizmo
This looks like a massive security flaw on sendgrid's side. They should use
DNS validation like everybody else to prove ownership of the subdomain.

~~~
wahnfrieden
From the article:

>Also at the moment of writing this bug, it has come to my attention that
SendGrid has added extra verification which forces you to have a verified
domain before adding a inbound parse webhook.

------
uraniumhacker
Hi thank you for the post. If anyone has question, feel free to reach out to
me here or at uranium@uraniumhacker.tech. As a writer of this blog, I will be
able to provide the feedback necessary and clear any misunderstanding/false
positives regarding this.

------
abalone
Looks like they found a similar "3rd party email service / subdomain MX
record" exploit with Slack.[1] Although not so severe in Slack's case because
it's a lesser used feature. Looks like Slack is using Mailgun.[2]

[1] [http://blog.pentestnepal.tech/post/150381068912/how-i-
snoope...](http://blog.pentestnepal.tech/post/150381068912/how-i-snooped-into-
your-private-slack-messages)

[2]
[http://mxtoolbox.com/SuperTool.aspx?action=mx%3aslack.com&ru...](http://mxtoolbox.com/SuperTool.aspx?action=mx%3aslack.com&run=toolpage)

------
wongarsu
Honest question: Do bug bounties with such low bounties really do more good
than harm?

A bug bounty incentivizes people to look for bugs. But when you find an
interesting bug like this you have the option of either having the possibility
of making millions from the social engineering possibilities alone, or
claiming the bug bounty and get $10k. Of course claiming the bounty is the
moral thing to do, but some people with both the skill and motivation to do
bug bounties are bound to have lower moral standards.

~~~
arkem
I think it's hard to monetize most bug bounty bugs.

There isn't really a market for most XSS, CSRF or even RCEs bugs for web
properties. Getting the bounty payout for a bug from the owner is often the
best deal available. I think the only exception to the no-market situation is
browser RCEs and smart phone OS jailbreaks.

~~~
Hydraulix989
Right, and this one could have easily been monetized because it allows
attackers to intercept and read any one of SendGrid's customer's private
internal emails!

~~~
arkem
I don't agree that this could have been easily monetized.

This vulnerability only allows you to intercept future incoming emails that
are delivered to sendgrid domains. Even if it did allow access to all of their
customers internal emails there aren't many buyers (and no obvious
marketplaces) out there for that kind of access.

The risk is simply too high, if you're Lyft (for example) being caught with
that access (even if it was never used) would be a possible terminal event for
the company. Purchasers would be buying a giant liability and a small
competitive advantage.

Maybe there are shady companies or criminal groups who would be interested in
the access but even then I feel $10k would be roughly how much money they'd be
willing to spend (considering the same amount of money could also buy
thousands of verified credit cards or a million email addresses at legitimate
providers).

~~~
Hydraulix989
Corporate clients aren't the ones purchasing black market exploits, the
marketplaces aren't obvious. Just confirmed with a friend who does not have
ties (disclaimer) that this one would be worth ~100k.

~~~
arkem
Maybe things have changed in recent years but here's some information on what
I considered benchmark prices when I was more involved in this kind of thing.
I have some background in vulnerability selling and I used to do this for a
living (first government, then later tech company bug bounty programs).

The "grey market" prices I've seen (~5 years ago now) from independent
researchers, small firms specializing in vulndev for law enforcement /
intelligence and the bigger famous security firms go something like this
("black market" criminal markets are similar I hear but I have no first hand
knowledge there):

250k would get you a iOS jailbreak (for a recent iOS version)

100k would get you an IE/Firefox/Chrome zero day (though not with a sanbox
escape, you'd have to pay separately for a privesc)

~25k for a Linux privesc

For an RCEs in medium popularity services (databases, ftp servers, etc) you'd
be looking at something like 10k (or often thrown in free with a bigger
purchase or support contract as a good will gesture).

The prices would also change based on the quality of the exploit, how recent
the version the exploit targeted was, the amount of exclusivity you want (more
for full exclusivity) and whether the research was specially commissioned (if
you had asked a firm to look into a specific piece of software the price would
be higher). The larger providers also encouraged organizations to sign
subscription deals and longer term service contracts that could include free
exploits or reduced costs.

That said, there's a lot of information asymmetry in the market so I'd expect
a lot of variance in prices.

------
joshmn
Would have been worth more to SendGrid, don't you think?

------
foota
Several months ago I reported a bug to a popular service which was allowing
password reset & other emails to be retrieved with just a serial id from a
service like sendgrid.

------
hartator
It's just me, or $10k is far from being generous?

~~~
innoying
Why does this comment appear on every bug bounty HN thread? Straight from the
horse's mouth [0]:

    
    
      The black market is very unlikely to be a place you could sell a bug in a specific 
      website or service. It is not “worth millions”. Please stop repeating this.
    

[0] - [https://medium.com/@collingreene/to-the-bounty-
hunters-9259b...](https://medium.com/@collingreene/to-the-bounty-
hunters-9259b1544325)

~~~
Hydraulix989
What a cute strawman (and completely incorrect strawman from the wrong horse
at that, this is Trumpist drivel from the guy who runs the actual bug bounty
who of course has some acute rationalizations for underpaying for some of the
most intricate technical work in the industry).

Nobody said it was "worth millions" but I have second-degree connections in
Scandinavia that would pay 10x, like I said ($100,000).

@collingreene doesn't sound too familiar with these rather-illicit
organizations, he strikes me as a product manager type person with a loud
voice, not someone who actually has found and sold zero-days before. Maybe he
doesn't have the technical acumen to do so, but hey, I'm not one to judge.

It's hard to establish proof that the market value on the black market is, in
fact, much higher given that it is the black market (you're not going to find
these people on Medium); However, one public example of this is the leaked
Stuxnet details showing similarly high 5-digit prices for zero-days.

This isn't a specific bug either (it wasn't "oh, the log files for that one
UberEATS micro-service were visible"), this flaw allows you to intercept the
emails of pretty much any single one of SendGrid's clients. Imagine the damage
someone could do with that, had it gotten into the wrong hands. Only $10k,
what an insult.

EDIT: Upon further examination, it turns out that said author also contradicts
himself and corroborates my own argument:

[https://medium.com/@collingreene/why-product-security-is-
har...](https://medium.com/@collingreene/why-product-security-is-
hard-52e3f73178)

Primary source: [https://www.wired.com/2016/09/top-shelf-iphone-hack-now-
goes...](https://www.wired.com/2016/09/top-shelf-iphone-hack-now-
goes-1-5-million/)

~~~
Hydraulix989
Wow, to further add injury to the insult, United pays bounty hunters with
miles:

[https://www.united.com/web/en-
US/content/Contact/bugbounty.a...](https://www.united.com/web/en-
US/content/Contact/bugbounty.aspx)

Don't these companies get that researchers need to live?!

