
MongoHQ Security breach - steveklabnik
http://security.mongohq.com/notice
======
joelgascoigne
Hi everyone. Joel here from Buffer.

Some of you might have heard about our security breach on Saturday. I just
wanted to leave a quick note and clarify that this MongoHQ security breach was
the method used to obtain our users' access tokens, which led to the wave of
spam on Saturday.

This is a key final piece of our investigation which brings it back full
circle. I'm very happy that we have been able to gain the full understanding
and can be confident there is no backdoor.

I want to be clear that this is still our fault. If access tokens were
encrypted (which they are now) then this would have been avoided. In addition,
MongoHQ have provided great insights and have much more logging in place than
we have ourselves. We’re also increasing logging significantly as a result.

I've updated our security breach blog post with this information. If you want
to see the full set of events, take a read here:
[http://open.bufferapp.com/buffer-has-been-hacked-here-is-
wha...](http://open.bufferapp.com/buffer-has-been-hacked-here-is-whats-going-
on/#update9)

Let me know if you have any questions about this. I'll keep an eye on this
thread.

~~~
computer
Have you considered not outsourcing critical parts of your company to startups
that are iterating fast and potentially breaking things? Could you share the
reasoning behind Buffer choosing to store their customers' private data with
MongoHQ, instead of your own secure infrastructure?

~~~
eli
That's sort of a loaded question, no? A database is, in one way or another,
critical to almost every business. I don't think that means every business
should build one in house.

~~~
akkartik
He's not talking about building the database, he's talking about hosting the
database. And he's not claiming you shouldn't use third-party hosts, he's
suggesting restraint in _who_ you choose, in waiting for a track record to
accumulate.

------
aroman
As much as the actual cause of the concern (apparently serious employee error)
was preventable, I was still impressed by their detailed disclosure, the steps
that have been taken to mitigate it, and actual details about what was
compromised other than the generic "non-credit card personal data".

I don't use MongoHQ for any projects right now (I evaluated their offering a
year or so ago and opted to go with MongoLab, which I am still suing), but
their response impressed me. There is indeed such a thing as screwing up
properly, imho.

Kudos to the MongoHQ team, and best of luck in dealing with this mess.

~~~
jdp23
> opted to go with MongoLab, which I am still suing

this is either an unfortunate typo or a great backstory ...

~~~
aroman
Hah! Oh god, yep, that was a typo. Part of me wishes I had a cool backstory to
tell though... (not that I think MongoLab is in the least deserves to be the
object of such action, of course!)

------
biot
Long story short: MongoHQ is securing access to critical administration
functionality behind a VPN and will be requiring two-factor authentication
instead of having that functionality available to anyone on the internet via a
simple password.

What's interesting is that I think we've all been there at one point. You
start building something, it gains momentum, then you get so bogged down with
adding functionality and fixing bugs that you never get around to implementing
the security features you know you should. At what point do you say "Okay, we
have too much at risk to not correct this immediately"?

~~~
ams6110
_that functionality [critical administation functionality] available to anyone
on the internet via a simple password_

I mean COME ON, MAN.... there are mistakes, and then there's incompetence.

~~~
olefoo
It depends on the simplicity of the password...

I mean yeah if the password was 'letme1n' that's one thing. Whereas if it's a
"at least 16 characters, mixed cased, punctuation and no english words"; maybe
that's another.

But, I would have thought at the very least, ssh with keys-only on the
external-facing bastion host.

~~~
nthj
What with keyboard loggers, unsecured wifi, video cameras, stolen computers,
stolen iPhones (with email access, now you're vulnerable to password resets),
there are just too many attack vectors for even a 16 character password to
suffice. You need to be protected by both something you have and something you
know. (My iPhone has a 16+ character password. It's a pain. It's worth it.)

Heroku's lack of 2-factor auth has literally given me nightmares.

------
JshWright
I'm not a MongoHQ customer (and as I don't use either AWS or MongoDB... it's
unlikely I ever will be), but this is one heck of a disclosure.

Other companies (looking at you Linode...) could stand to learn a thing or two
from how MongoHQ is handling this. Extremely transparent, and explaining in
very clear terms the steps they'll be taking to mitigate the breach and
prevent future incidents.

All that being said... Ouch.

------
flyt
Especially bad for people that were using MongoHQ to manage their DB backups
to S3:

As a precaution, we took additional steps on behalf of our customers to
invalidate the Amazon Web Services credentials we were storing for you (for
the purposes of backups to S3). While this prevents the abuse of your AWS
credentials by any malicious party, it may have resulted in additional
unintended consequences for your AWS environment if you were utilizing the
same AWS credentials for other purposes.

If you had S3 backups configured with MongoHQ, when possible, we have disabled
the IAM access keys via AWS. In any case, please go to the AWS Management
Console and regenerate any keys given to MongoHQ .

~~~
joshchaney
I was one of those people bitten by this last night. My client called and told
me he was getting access denied when trying to upload files through his CMS.
After some digging I found the S3 key had been revoked. This was concerning,
as I hadn't touched the CMS code I wrote in like 3 years and I've had issues
deploying old stuff to Heroku in the past. I really wish MongoHQ had contacted
me first about revoking the keys.

~~~
muns
Best practice would be to create an IAM user for each purpose rather than
sharing the same AWS key across all of your apps, for this exact reason

~~~
joshchaney
At the time this project was put together, IAM didn't exist. But I agree that
this would be the best approach going forward.

------
theboss
One thing I hate seeing is startups and security. Not that mongo is bad or
anything this is just a good time to say it.

Build your product correctly from the start and have a security policy early.
Don't leave XSS in your website, don't just hack code together and throw it on
a server. As a security guy I love seeing new startups products because so
often the code is so freshly written and immature that anyone it's asking for
trouble. Measure twice cut once

~~~
alexandros
Some times taking more time (and/or better coders) to do something 'right' vs.
just hacking it together is the difference between failure and success. So
from an individual startup's POV this may not be worth doing. Having enough
users/success to warrant a security breach is a luxury problem when viewed
from the eyes of an entrepreneur with no product.

On the other hand, collectively, startups losing everyone's credentials/data
are burning the commons wrt people's willingness to use new products.

But then on hand #3, it's not as if BigCo's have a much better record, so I
guess this gets chalked up to the general 'why in the world do we release
software this bad?' and the general answer is 'because people have been
trained to accept it'. In the same way that if the car was invented today it'd
never surpass our current health&safety considerations.

So I guess my point is, software is terrible in general, and security is just
one aspect of many. As long as we don't have a better way to write it, it'll
keep being bad. It's not fair to expect startups to set a higher standard,
when they're optimising along several other dimensions at the same time.

~~~
theboss
You're right that it is a luxury but you are wrong that it is unfair to expect
startups to set a higher standard.

Startups have fewer assets than big companies. If you can't secure 1 webserver
or you have a security breach with only 3 employees, there is something
seriously wrong.

If you expect to one day have 10,000 servers you better believe I expect you
to be able to handle 3 or 5 or 10 by yourself.

~~~
memset
How, or where, does one learn how to implement a secure application (say,
nginx + uwsgi + python + mongodb)?

This is a serious question - the best that I'm aware of are blog posts, but
every time someone writes a post about security, HN comments (et al) give
contradictory advice.

------
roin
If I were running a company, I'd buy everyone a license of 1Password and make
its use mandatory. Sure there are still plenty of attack vectors, but it's
just too easy, and 1Password is such a cheap way to mitigate risk (not to
mention you're doing your employees a huge favor by reducing their
vulnerability outside of work).

~~~
jumby
Better yet, how about making sure your employees can only connect to your
internal tools when on a properly secured VPN and not "externally" from
home/coffee shop/airport?

~~~
DannyBee
Can you explain why this is better than allowing access using 2FA over HTTPS
(with a non-crappy set of cipher choices)?

IE What does the VPN buy you, specifically, on the employee side?

(I understand entirely what it buys you on the other side of the equation,
such as a smaller attack surface, i'm just trying to understand why you would
think having a VPN would have made this particular case more secure)

~~~
tptacek
The value of a VPN over individually-secured HTTPS/TLS+2FA connections is that
you can configure the VPN once, use very standard networking tools to
continuously ensure that your internal services are only available over the
VPN, and not have to worry about individually securing different internal
services.

Another benefit is that as your internal userbase changes, you can revoke
access from a single point and be reasonably assured that you've mitigated
risk, which is something you only get with individually-secured services if
you have a reliable directory system.

A problem with individually-secured ops/support systems is that most 3rd party
code is not ready to be securely deployed Internet-facing.

Both approaches are totally workable, but the VPN approach is easier.

~~~
DannyBee
I guess, in a lot of ways, i'm not sure about " and not have to worry about
individually securing different internal services."

This is essentially something you need to worry about anyway, for other attack
reasons.

~~~
tptacek
That's true and a point worth making, but as a practical matter, breaching
almost anyone's perimeter is game-over. To not have that be the case, you need
to design from the ground up so that internal services don't trust their own
deployment network; it's difficult, time consuming, and in many cases
confining (ie, it makes some services prohibitively difficult to deploy).

It's for this reason that pentesters learn quickly that the "make an arbitrary
HTTP query from the target's own server" bug is usually sev:critical; for
instance, in virtually any Fortune 500 network, that pivot gets you (with a
little effort and 50 lines of code) to a JMX console somewhere, and from there
code execution.

There's no good reason not to do both (ensuring that your internal services
are authenticated reasonably and don't expose functionality or information
pre-auth, AND setting up a VPN). But the VPN is the most valuable step.

------
bdcravens
_detected unauthorized access to an internal support application using a
password that was shared with a compromised personal account_

Someone's updating their resume tonight.

~~~
richadams
Seems a bit harsh to fire someone for that. People make mistakes. If anything,
you know that this person isn't going to make the same mistake ever again.

Of course, if they do, then firing might be the way to go. First time mistake,
second time incompetence.

~~~
bdcravens
Perhaps something like this should be in a policy handbook?

"Employees may not use passwords that are used with any service outside of the
company."

Zero enforceability (but some companies would probably ask for a list of
outside passwords I'm sure), but in the face of a direct policy violation,
100% fireable, and can help a company in terms of liability.

------
jumby
And this is why you use 2FA and put this behind a corporate VPN in RFC 1918
space. Why they are just planning for that now is amusing.

"Our support tool includes an 'impersonate' feature that enables MongoHQ
employees to access our primary web UI as if they were a logged in customer"

~~~
iloveponies
2FA and VPNs are not exclusively the only way to secure things. X.509, bastion
servers, airgaps that require physical access to a secure facility etc are
also valid options, dependent on your systems and their configuration.

~~~
jumby
Granted, but an airgap would make working with some internal support tool a
bit cumbersome :)

Bastion servers if properly firewalled might be OK for a short term solution.
The concern there is if you allow unfettered ssh (for example) is someone
watching for the inevitable brute-forcing that will ensue?

~~~
iloveponies
Mandatory SSH keys mitigates the brute forcing risk, and turns it into a
nuisance. My employer presently has this arrangement and has done so for a
while. Bastions only get you in the door: different entrances for different
environments, users keys are only propagated to the machines they need.

~~~
jumby
Roger that. I keep thinking of my customer support people as non-technical and
for whom ssh keys, port forwarding & bastion hosts are way over their heads
but your point is taken. There are other (cheaper!) ways to secure an internal
network.

------
lgleason
These guys have great support. A client of mine was using them and had a
cluster get wonky. The performance of their website got really slow during an
in-opportune time. MongoHQ got right on it, fixed the issue and gave them 6
months of service and doubled their instance size free or charge to make
things right..... Sure it would have been better if the servers had not gone
down, but that can happen on any system. They also followed up for the next
couple of weeks to make sure that things were running smoothly.

I was very impressed with their customer service and would highly recommend
them.

~~~
jumby
That's nice. Their application security setup, processes & procedures are
apparently severely lacking. So, you might want to think about recommending
them to your other clients.

------
brandoncarl
It would seem to me that New Relic license keys would also be available to the
hacker(s), as they are required to link MongoHQ to New Relic.

In speaking with New Relic, these keys cannot be regenerated.

Wondering if anybody could advise as to what security issues that might
create? I presume it could potentially give access to some infrastructure
information. Have emailed their support but waiting on a reply.

I'd like to chime in that, this circumstance notwithstanding, MongoHQ has been
great during the time we've been a customer of theirs.

~~~
brandoncarl
From New Relic support:

With the license key someone could configure an application to report to the
account affiliated with the key. They would not be able to access the account
or use the key to retrieve any personal information from the account.

------
aferreira
While I commend the team at MongoHQ for detailing the attack, the strategy
used to restrict any further damage and implementing new processes and
additional security, I've heard this story one too many times.

When are companies going to realize that security is a _critical_ aspect of
their business? Why are people _only_ acting when there's some big security
breach, using trivial systems before this?

------
jimmytidey
So is the story here that someone targeted and broke into a mongoDB employee's
email account, found a password for an admin panel, then used that to extract
details stored by Buffer to access people's FB & Twitter accounts? I wonder if
Buffer was the target, or if it was just the first valuable thing the attacker
happened across?

------
tsenkov
Disclaimer: I haven't used MongoHQ.

Yesterday, I decided to go with MongoHQ for my current project and this
morning I saw they got hacked. Irony aside, couple of months back I decided to
check AppFog (PaaS) and the next day they were acquired. :)

I believe this is a great time to become a new user of MongoHQ - security has
been tightened, experience has been built. The "storm is over" and the chances
of a new "storm" in near future (with the measures these guys have taken) have
decreased substantially.

I think they reacted pretty impressively, once the breach was detected.

As much as I understood, the biggest flaw in their operations was the fact
that many/all internal/support systems were exposed publicly and not within a
VPN.

Still, they were around for just about 2 years, which isn't that much time,
after all.

~~~
dotmanish
Can you please choose Microsoft and decide to use Windows? I want to see what
happens tomorrow. :)

~~~
tsenkov
I can be used as a weapon. :)

------
leokun
So I am wondering if any service I'm using is using MongoHQ to store my
sensitive information in plain text and was breached. Like, it is a bigger
deal than just "change your password" when an attacker can "god mode" into
customer databases.

~~~
jturolla
The question remains: Will they disable god mode? I'm truly not comfortable
with this feature, I should be able to block total access from my database. I
deal with extremely sensitive personal information of thousands of users, and
I would never give access to it for support people from a company I hired -
Not even MY support team has access to it!

~~~
reidrac
I think in those cases you may not be legally allowed to use a 3rd party
managed service to handle the data, although it probably depends on the
country where your company is operating.

Also that "god mode" only makes things easier, if you're hiring a managed
service by definition it means someone must have access to manage the service.

The answer is: manage it yourself effectively locking access to any 3rd party.

~~~
nolok
There is a difference between "someone must have access" and "a random support
account at that company can have full access".

I agree with parent that customer should be able to disable god mode on their
account.

------
benmccann
Putting support functionality on the VPN means folks from outside the
organization can't get to it without VPN access. It also means that you must
give more people within your organization access to your VPN which raises the
chances of having a compromised account be able to access other valuable
assets on the VPN such as your production hardware. How do you make a trade-
off between these?

~~~
pfg
VPN access doesn't have to be all-or-nothing. They could (and should) lock
each employees VPN access down as much as possible, i.e. support personnel has
access to their support tool and nothing else, etc.

Production hardware should be on a separate network/VLAN/whatever anyway.

------
whbk
Pretty poor communication -- we were also among those who had to get new AWS
credentials with no idea as to why ours were no longer working. Luckily we
jumped on it pretty quickly, but I hope this kind of thing is handled much
differently by providers in the future. Very inconsiderate of the impact on
users.

------
Kiro
I've never understood what people use MongoHQ for. I mean, the data is
completely open and it doesn't seem to have any support for authentication.

~~~
mhitza
What do you mean the data is completely open? Do they generate just a
tokenized endpoint to which everyone can have access to unauthorized?

------
ing33k
"impersonate" feature in suppot system - I hate this feature, I always refuse
when some one asks me to build a feature like this .

------
meowface
At least they used bcrypt to hash user passwords.

~~~
s_kilk
For the non-experts among us, does this imply that having the bcrypted
passwords in hand is almost worthless from an attackers point of view? Would
it be feasible at all to derive the plain-text passwords from that encrypted
data?

~~~
mLewisLogic
With sane defaults, bcrypted passwords are "worthless" in the hands of a
casual attacker.

Somebody would have to REALLY want to access your account and have SIGNIFICANT
financial/computational resources to crack it. And that only gives them one
password. Significant and independent work is required for each individual
password.

On the other hand, there's nothing stopping a developer from doing something
really dumb like setting bcrypt iterations to 1.

~~~
s_kilk
>> On the other hand, there's nothing stopping a developer from doing
something really dumb like setting bcrypt iterations to 1.

I _really_ hope that doesn't turn out to be the case here.

------
gmjosack
The name of this company is unfortunate. It doesn't appear that this service
has any affiliation with MongoDB, Inc.

~~~
petercooper
Mongo is a trademark of MongoDB so I suspect they got permission because
naming your company after a trademark is a quick way to get into trouble.

------
joevandyk
Anyone got any advice on setting up a VPN and 2 factor authentication? We use
Ubuntu, AWS, and Rails.

~~~
danielpal
Send me an email to dpalacio@authy.com and I'll give you access to our Authy
OpenVPN plugin free.

~~~
joevandyk
Sent!

