He didn't do this because he's an amazing sysadmin. He did it because he's competent, knows how to put a service on the Internet, and RTFM.
I mean. If you can connect to something and use it without having to authenticate yourself, wouldn't it naturally cross your mind to check that others can't do the same? It's just common sense.
This is simply first-time developers with no server experience, deploying their first project, who don't understand the first thing about having a server on the internet. Publicly exposing an open port for software that is innately designed to run on a private network has nothing to do with... whatever you want to call all these "exposé" pieces about "attacks".
You could write an article to explain the introductory concepts of system administration to newcomers to the industry. But to even mention the word "attack" when the only thing involved is an open port is... sigh.
I feel for newcomers who need to learn. I really do; the amount of information one needs to absorb to be even remotely competent is vast, and takes at least a few years to pick up (and then another decade or two to fine tune that knowledge). But this is not a situation where the maintainers of software packages are to blame for not educating their users, or insinuating that their products are not "secure out of the box". It has nothing to do with individual software packages, and everything to do with the very core aspects of having a computer connected to a public network.
Real companies with real data make some pretty elementary mistakes with regards to security. I'm a security tester and the number of times I've got access to systems deployed by real companies who've really paid money for an external security reviewer using things like default creds is quite high.
It's tempting to think that this is just an education issue and that once people know how to do security well things will get better but personally, my opinion after 16 years in security is that this isn't the case.
Effort spent on security is a trade-off with other things and in many cases people make the choice (either unconsciously or deliberately) not to prioritise it.
The insecure defaults were an issue sure, but anyone installing a piece of software in production without at least reading up on config options needs to find another job.
We need good and consistent rules about this, and "well I was giving it away for free" isn't as clear a boundary as people will think it is.
I would imagine that if someone were to sue MongoDB Inc. today about the issue in the article, their first defence would be clear documentation that explained/recommended production guidelines.
I don't know if D-Link had similar, but then again legal systems sometimes produce weird results that don't seem rational.
I agree that we need good and consistent rules, in addition to that, we need DevOps/SysAdmins/SRE's that are responsible enough to know what they are doing.
Carefully read documentation instead of "quickly deploy" only to come back a year later writing soppy "don't use MongoDB or XYZ because we didn't read the manual". :)
MongoDB is on 3.4 now, so I would rhetorically wonder why some people/companies are still on <=2.6.
If the data that is being ransomed is that important, it'll be a good lesson to those DB maintainers to upgrade and secure their stack.
This part seems to be glossed over but is a HUGE issue.
It sounds like several companies have tried to pay the ransom with varying levels of success  ... why are they not just restoring from backup? I can only assume they don't have backups. (!)
What is their DR plan if the server dies? Or someone accidentally pushes code that messes up the contents of the DB? Or someone tries to drop the development database but oops: they didn't notice they were connected to the production server?
Even if you're using a hosted service, what if they go down? Get hacked? Lock you out because of a billing dispute/TOS violation/DMCA takedown/accident? Hired bad sysadmins that didn't do backups (correctly)?
Not having backups of your data is inexcusable and just reeks of utter incompetence, and has nothing to do with configuration defaults or documentation.
You still need to do external backups. You may have a lot of trust in the provider and these less frequently, but you should still do them.
 Had this happen to me once in the early 2000's: company I worked for had a dedicated server at a colo facility. After several days of them not responding to phone/email/etc, their answering machine was changed to a message saying the SEC had seized all assets and had all the owners 'under investigation' or something like that. We had external backups, but immediately took the latest stuff and got everything migrated to a new system in a new facility. Server stayed up for a few weeks after that, but then suddenly their whole IP space went offline. We never did get our server back.
Other than the cost, I recommend it for people who can afford it. Wonderful service that I was happy with for a long time.
I'm obviously not going to defend companies that don't have current backups (though this is practically everyone), and the importance of backups is always a great thing to emphasize, but in this case, the best option is to pay the ransom and get your stuff back.
Now I'm sorry, but MongoDB inc IS at fault as well, for not forcing developers to create credentials upon installation from the beginning. Any vendor that doesn't do that with its product isn't serious about security. Let's take Wordpress, imagine it didn't force its users to create an admin account upon deployment, everybody would be mad about it. But somehow MongoDB got a pass all this years? bullshit.
I hope this hack will permanently damage MongoDB brand.
There's a moral distinction between culpability and impact. There are profoundly stupid things that people do, yet still need protection from.
Those DB admins were incompetent by lots of measures, but their data still has value and its seizure is a public harm. It's the job of the rest of us (in this case, MongoDB's developers) to take reasonable steps to minimize the chance of that happening.
Secure defaults are a very reasonable precaution. MongoDB fucked up.
We help each other out in this society. So in this case if you're a database developer with a good handle on deployment security, you don't put a insecure-by-default product in the hands of people who aren't. I genuinely can't understand why people are arguing to the contrary.
Even knifes are sold with some package that prevents them from cutting before the package is removed.
I agree, I don't think your job is done just because you wrote somewhere "pay attention to this".
But that's fine. At our place, our mysql cookbook is maturing. This mysql cookbook makes it really easy to say "hey, we need a mysql master with these 2 databases". The only security overhead consist of generating 4 passwords in that case (and we're planning to automate that).
Once you've done that, you get a mysql master with passwords, restricted root access, a local firewall, backups and all kinds of good practices. It's secure because our cookbooks are good, and people use our cookbooks because they are easier than handling mysql directly.
And that's a kind of cooperation I'm really happy about. Devs want services, and we provide easy ways to setup services in decently secure ways.
Do the community chef, ansible and puppet cookbooks use secure defaults for MongoDB? Asking because I have never used MongoDB.
Yes, but I've grown to dislike that term, though it's my current job title. There's a number of people who're titled with DevOps and they're yelling about Docker this and CD that, and AWS/RDS those.
That crowd is way to excited about some tools and some solutions in specific use cases, and they sometimes tend to drown out the real value of config management and close cooperation of devs, ops and other involved people with their own specific focus - for a very large amount of use cases.
> Do the community chef, ansible and puppet cookbooks use secure defaults for MongoDB? Asking because I have never used MongoDB.
I'm a chef guy. First google result for "mongodb cookbook" doesn't set authentication, but makes it easy to enable required authentication. Second cookbook result doesn't enable authorization by default.
This makes sense though. If a community cookbook manages mongodb, that cookbook is supposed to support all use cases of mongodb and it usually tries to mimic the default use case of the application in question. To maintain that, the chef cookbooks for mongodb don't enforce authorization.
However, if I was supposed to implement a mongodb cookbook to use in my place, I'd intentionally fail the chef run if authentication is disabled and stop mongodb in that case. This would be trivial in both cookbooks I looked at.
This is not exclusive to DevOps, though DevOps is a hot buzzword right now so lots of posers are flooding in. Most people, no matter their station or the prestige of their company, have no idea what they're doing. Knowing this is one of the most important things for dealing with the corporate world (I think I'm missing a couple of the other most important things, though...).
Different people try to cover this up in different ways. One popular way is acting like they always know about new cutting-edge tools sponsored by AppAmaGooBookSoft and that the next thing will finally be the Silver Bullet we've all been waiting for.
This impulse has brought us the prominence of node.js, MongoDB, the proliferation of "big data", and many other vastly-overdeployed niche products that have been ravenously and shamelessly misused by incompetent people trying to fake it through their careers. Our standards for this are, frankly, sad. It must have to do with a combination of a cultural bias against non-junior engineers outside of Java/.NET-land and putting completely unqualified MBAs in charge of tech (and yes, this is also applicable in startup world via VC proxies) -- but I digress.
Been hearing that song and dance re: docker and k8s for the last year at least, and boy is it ever tiring. Docker and k8s are both very niche, very immature products that greatly complicate system administration. They are missing features that you rely on and that you will have implement horrifying hacks to work around. Why are you doing this to yourself, again? Oh, because AWS is too expensive and you want to consolidate (fake reason btw, real reason "because it makes people think I'm smart")? Yeah, about that...
It's fun to throw together a lab for experimentation, maybe hook up a weekend project, but no sane Real Company is going to be moving its stuff to all-containerized k8s any time in the next 2 years.
My current project at work? Converting all of our environments to docker/k8s...
There are devs who are unaware of the concept of authenticating access to an application?
Yeah, well, we're in the middle of a trend to act as though dedicated systems people are bad, pains in the arse, you shouldn't need or want them.
Come on, man, this is 2017. Get with the program.
you can't just demand the user be smarter, you put sane defaults on there so they actually have to go to at least a small amount of effort to shoot themselves in the foot.
Besides, you should be using containers for everything anyways. If something happens to it, just throw it out and spin up a new one.
I wouldn't call this reinforcing behavior...I'd call it protecting your users. Common sense is something that comes with experience, and MongoDB is popular enough that it's likely to be in many web devs' first projects in production. Is it their fault they haven't developed this "common sense"? Yes and no. On the other hand, is vulnerable-by-default ever a good idea?
I am a former Sr. Systems Administrator, and Sr. Systems Engineer. My current title is Sr. Devops Engineer.
Pretty much everybody who knows their way around a compiler and administers systems could probably take this title and be fine, and then you're not suffering under the yoke of devops or whatever. Because when I hear "developer," I can't help but think a feature engineer on the product side for some reason. Same with that word creeping into "devops," and that leads to things like "full stack" engineering -- creating the impression that only a select few work with or consider "the whole stack," which is a scary subliminal message. Even if you're a developer simply changing the colors on the Facebook feed or something, you should always be thinking about the full scope of things and how your changes will be operated. (Your operations nerds will love you if you think about us ahead of time.)
As an aside, systems administrator with no software skills is a totally and completely fine career, with a number of very smart people (and friends, still proudly announcing every year of uptime in texts), and in a lot of these discussions I see them denigrated. Just remember that, not saying anyone is doing it.
I would expect anyone responsible for something to have competency in it or find someone who is to help. Titles are irrelevant.
Refusing to deploy a process into production without setting a password is more like refusing to drive over a bridge that's obviously missing chunks and crumbling -- that is, it should be obvious, even to the untrained eye, that there is a massive structural issue that makes it unsafe.
I will grant that some junior people may assume that local connections are coming in over a trusted socket and that TCP connections are required to hit an authentication process, but even this is kind of a stretch, and no one should be leaving production deployments for real companies in the hands of someone that inexperienced.
I will also grant that guilt-by-association is probably not appropriate here. There are a lot of good people at companies that have grossly incompetent and/or disorganized managerial superstructures that may have caused something like this to end up exposed to the internet. I have personally worked at multiple places where the daily comedy of errors we called "work" could've resulted in an unpassworded Mongo ending up on the public network.
Any way you slice it, driving over what is a clearly destroyed, ruined bridge is someone's fault, somewhere (and per regular management practice, the person least responsible for the fault will probably take the fall :) ). Let's just not pretend that this is an ordinary complication or oversight in the course of sysadmin.
Furthermore, running "apt-get install" usually isn't enough to get something exposed onto the public network, even for home users, whose routers block all incoming ports by default. Someone has to go in and explicitly open the traffic before something like Mongo gets exposed; there's no reason that labs, developer machines, etc., would be publicly accessible.
Mongo is hardly alone here; comparable services like MySQL, PostgreSQL, and Redis also install without a password by default. For MySQL, this is perhaps less noticeable because Debian/Ubuntu (and probably some other distros) include a prompt to allow the user to set a new root password as a post-install script. For PostgreSQL, the default mode uses IDENT authentication, i.e., it's accessed by dropping to the "postgres" user ... and, like all other system users, including root, this user is passwordless by default.
For Redis, it's passwordless by default and I'm sure there are many installations that have this same "vulnerability" in redis-land (there are also many installations that have real vulnerabilities, because Ubuntu's redis-server package in 14.04 hasn't been patched), because people rarely think about redis auth either. Just since last year, there is a feature called "protected mode" that requires the user to explicitly disable a flag if they intend to run a server that is a) bound to a non-loopback interface and b) not passworded (but it doesn't actually require someone to set a password). That's a cool feature, but as users don't RTFM, I'm sure they either flip protected mode to false without understanding what it does or just give up on redis and install MongoDB instead. :)
The point here is that pretty much everything you can install on a server is passwordless by default. I don't think the system of someone who never thinks about passwords would last very long at all. This is NOT a MongoDB-specific anomaly. Passwordless is the typical case for most services.
A feature like Redis's protected mode is a nice bonus, but it's not fair to lay responsibility for this at the feet of Mongo or to call it a "vulnerability", as the OP does. The responsibility lies with people who say "I'm going to put all of my company's data in here and push it live without thinking even a tiny bit about security."
Do a lot of people do that? Yes. Should MongoDB take more steps to protect such people from themselves? Maybe. But this is a very basic, very routine thing, which is easy to look up and rectify -- not some obscure anomaly buried under 600 pages of documentation or requiring a special compile-time flag. Mongo doesn't deserve the heat for people who never set up a password.
Having something listening on a public IP without any password protection is irresponsible whether it's a database or security camera. Last time I installed mySQL I had to both set a root password and specify socket to listen on, and it took five seconds and no documentation reading was needed, so it is possible to both be user friendly and have good security. With mySQL you also have to explicitly set from what IP a user can connect from.
Secure should always be the default! Making it not secure should require messing with config files, not the other way around. Most people will just stick to the defaults. There are plenty of examples where you do not need a password, for example opening your fridge or turning on your TV, both being connected to the Internet. Being able to connect to it from your friends place should ring a bell, but I don't think most people try that. Just like people don't regularly port-scan their networks.
This is not the case in the United States at all. In fact, it's the opposite. ISPs used to only offer modems that were intended to plug directly into one's computer, which would bridge to the computer's ethernet interface and expose it directly to the internet. Third-party routers would be placed in the middle to allow multiple computers to connect and they almost always include a firewall that has everything blocked by default.
Recently, ISPs have been bundling their modems into routers and providing customers with a single device that facilitates multiple computer access, wifi, etc. Every time I've encountered such a thing, it has had a firewall that blocked everything (except maybe a port for remote tech support) by default. This is by far the most common.
IPv6, as always, is a pipe dream, but when it becomes real, yes, we will not have the assumption of NAT to fall back on -- though it's pretty safe to assume that consumer-level hardware will continue to block all ports by default.
>Last time I installed mySQL I had to both set a root password and specify socket to listen on, and it took five seconds and no documentation reading was needed, so it is possible to both be user friendly and have good security. With mySQL you also have to explicitly set from what IP a user can connect from.
This is not a feature of MySQL, but a feature of the distribution you installed it on. It is true that MySQL's configs don't listen on all interfaces by default. IP-based security generally means that every user is wildcarded, and MySQL auto-generates a wildcard root account without a password iirc.
>There are plenty of examples where you do not need a password, for example opening your fridge or turning on your TV, both being connected to the Internet.
MongoDB is not the equivalent of a fridge or TV. MongoDB is the equivalent of a power saw. If you can't be arsed to take basic precautions while using it, you really shouldn't be anywhere near it.
And while many TVs and fridges may have an annoying built-in wifi feature now, most people do _not_ think of these in the same context as server software. :)
Especially if it's a software that's targeting people who wants things easy & fast, like Mongo did at the beginning.
See reasons for WordPress needing to be PHP 5.2 compatible, which had been out of support for 6 years.
That's why so many other systems are insecure. Security nearly always increases friction.
I once had a conversation with someone who started an open source project about the "10K meter" view. I strongly encouraged secure defaults. He asserted that when getting started, grabbing mindshare requires insecure-out-of-the-box. They could fix that later.
Later never came; the project died (Not due to security). But I'm pretty sure something like this would have happened if it had taken off, because later doesn't come until something like this happens.
I don't know how wide-spread this thinking is, but it slots in well with a lot of the decisions I see made by other startup types. There seems to be a lot of conventional wisdom about the slimy-ish shortcuts and deceptions, bad design decisions, and so on that are "OK" to make to get off the ground. I'm not going to comment on the necessity of that sort of bad behavior from a business context; I think the answer is complex, and kind of appalling. But on the engineering side, it seems to usually lead to nasty things coming back to bite you, and it is notable that it does so after you've gotten attention and people depend on you.
Instead you write it in a way that it has the potential to be great.
Customers that have been using your software for a year will have different problems when they have been using it for four.
On one project we essentially had a goal of doubling our data capacity of our (on premises) software every year to stay ahead of our customers' data collection rate. So my goal was 10% per month. We kept on that treadmill for over two years. Yes a couple of those months involves some pretty deep changes to get to a better info architecture. But most of it was constant application of the campsite rule.
That said, in that same project we added security and permissions after the fact, and it was one of the most frustrating things I've ever done. A constant and unending stream of bad news from QA about some other spot we missed.
But for now, if I use this technique in the code, and you business guys want to start calling files 'assets', or clients 'customers' (or vice versa), then you guys can go fight it out in a room and just tell me what you want afterward.
Security, concurrency, and localization are three things you can't just tack on later. Anyone 'test infected' would add automated testing to that list too. They all involve a long game of whack-a-mole and they're IME bad for morale and productivity.
It's not a perfect comparison, but there are some similarities to the tragic cases of children being left in hot cars in the summer. Dozens each year, largely by parents who aren't abusive, neglectful, or what most would consider to be "bad parents." Instead, it's what's called a prospective memory failure: habitual memory processes override prospective memory, which by its very nature involves additional cognitive mechanisms. There are a couple of PM theories, but the multi-process model suggests that PM cues can trigger a prospective intention even without an active monitoring process. Miss or interrupt the cue, and the intention isn't retrieved and followed through on. Failures such as airplane crashes when pre-flight tasks are interrupted and tasks are missed, fatal anesthesia incidents,, fires because gas appliances were left on, etc. A number of industries have been putting more effort into hedging against PM failures over the past few years through better checklists and other aids and precautions precisely because the consequences can be so tragic.
Getting back to the topic at hand, the issue with dumb shortcuts like insecure defaults isn't that they're bad on their own but that the way our brains work makes it less likely that we'll go back and fix all aspects of those shortcuts later before a problem arises. Documenting your shortcuts, setting reminders, and other actions can help minimize that likelihood, but how likely is it that you'll do so when you're in the middle of taking an ill-advised shortcut in the first place? At that point, you might as well spend the time to do it right.
It's the same danger that comes with hotfixes. Even when you have no other choice, you've put yourself in a position that could have unseen danger later on. Having a policy in place (checklists?) for dealing with something like that can help you avoid PM failures by forcing you to go back and retroactively apply your normal procedures to the patch in question. Style guides and other guidelines help as well in everyday circumstances, even though they can be a bit annoying at times. Because no matter how you look at it, mistakes will be made.
Insecure defaults and stupid shortcuts make it more likely by stacking the deck against the people working on a project.
But most of us chose not to do that for a living.
Internally everything is screwed, from the architecture to the security settings, to the reliability of data storage (or lack thereof).
MongoDB is the school case of 1) Worse/bad software can succeed despite being bad. 2) Don't trust hype. 3) Developers in general have low standards.
-  https://www.elastic.co/guide/en/elasticsearch/reference/2.0/...
-  https://www.elastic.co/guide/en/elasticsearch/plugins/2.2/di...
-  https://www.elastic.co/guide/en/elasticsearch/plugins/5.1/di...
Move fast, and break things - you're not responsible for the damage your product does. Our industry both optimizes for, and rewards this kind of behaviour.
This issue, along with its related issues with permissions on persistent storage, is one of the major reasons I'm hoping to move away from Docker as soon as possible.
Re root permisions I guess if you're running a typical network daemon (Apache, say), then requiring Docker to run as root is no worse than running the service on the host; root would be a requisite anyway for the daemon to open privileged ports/gain logging permissions etc. before dropping to "nobody" or what have you.
If you are not using docker, you can manually add the specific privilege to open low-numbered ports without running the program in question as root.
I think the main guilty parties at mongodb were/are in marketing.
technically sophisticated users understood the immaturity of the product and the tradeoffs that came with its architecture.
however it was sometimes marketed as a general purpose data store, or as an alternative to much more mature relational data stores, which was and still is an unfair comparison.
We could expect every programmer to have domain knowledge of databases, even when it's not normally needed to operate one.
Or we could just make databases not acknowledge writes that haven't happened, not listen on all ports, not hide information about data loss in non-clustered setups deep inside the documentation.
Counterargument: many people who are good at a particular topic enjoy feeling superior to people who don't have their knowledge.
unfortunately, I don't think this was always communicated effectively by the explosion of mongodb marketing and hype.
What do you mean? If you you mean it took knowledge to run per se: it didn't: unzip and run the service, it's up. It wasn't any more complex than any other piece of software.
Or do you mean it did require considerable knowledge to avoid data loss on software marked as 'stable'? Yes, that's the problem, and yes, that's unreasonable.
But Mongo pretty much rejected that world view (web scale!) so it's a little odd that they designed a system that only works in it.
No. Absolutely not
Unsecured local connections yes, anything from outside 127.0.0.1 should be explicitly allowed and secured (like port 80/443). Rate-limited, fail2banned, fuzzed during testing, etc
"I changed my password on one system and then on the other, but somehow my account got locked up in that brief time between changeover, how the hell do you unlock an account? This system is stupid."
"We're scaling up and we're running into outdated rate-limiting settings on every service we use, some of them even having multiple layers of rate-limiting within a single service, and each setting requires hours of debugging to figure out which setting is taking down the whole project. This system is stupid."
There's a reason Mongo got adoption while "secure" systems did not.
If you only have a VLAN separating you from harm, it only takes 1 bad switch configuration or any other device on the VLAN to accidentally be running a proxy or router.
ElasticSearch => Dropped authentication (i.e. an important aspect for security) to get a chance at monetization.
MongoDB => Dropped everything, including your data. YOLO!
Elasticsearch got all the way to version 1.4 with no authentication as part of the design. Between 1.4.1 and 1.5 release candidates is when they announced Shield as part of their enterprise offering.
Here are the release dates and announcements:
This is how this goes, and I've seen the pattern followed with several infrastructure services, even here in this HN thread right now:
1) A bug report comes in to AwesomeDB. "You do not have authentication support. This is bad. I'm going to yell about it on social media and make fun of you until you can accept a username and password, because everything in existence should write its own authentication code. Even MySQL takes passwords, dude, get with 2017."
2) A rough version is implemented by an intern, because authorization is way less sexy than hyperdimensional bloom filter quantum hash tables to speed up random reads. The intern then departs back to beer pong and nobody at AwesomeDB now understands the security infrastructure of their product.
3A) A mid-level executive, seeing the announcement, instructs me that I no longer have to keep AwesomeDB behind my 'stupid' reverse proxy + ACL solution and we can finally use native authentication and be really secure!
3B) Alternatively and more reastically, someone at Qualys adds "authentication is not turned on in AwesomeDB" to their scanning product and I get nastygrams from a contractor clicking "Scan" and copying and pasting the results for each of my fifty AwesomeDB instances.
4) During testing, I capture the credentials in plain text in about three minutes because it's a database. Why on earth would it terminate TLS? If we are down the 3B path above: Ha! Your move, security team!
5) Another bug report is filed against AwesomeDB to terminate TLS. Equal shaming is performed, because every service in the world needs to link in OpenSSL and terminate TLS, because that's never been a problem or backfired in any way. Oh yeah, and everyone has a different mechanism for handling certificates, and most don't bother to consider revocation.
(Just like the broader point I'm making about auth, stop adding TLS support to your systems. Please. If you speak TCP I can wrap you in something uniform that does a much better job anyway. You know how I know your integration is shitty? Because OpenSSL is hopelessly undocumented and the APIs even more so. You Googled and read a guide. How about instead, I can drop a pretty awesome TLS listener in front of you, potentially even in hardware! Whoo!)
6) TLS support is added to AwesomeDB. Cool! Now we can really be secure! (But nobody implements client certificates, so we still have to do the username/password shared secret business. And alternate DH parameters? What are those? They weren't covered in the guide, so AwesomeDB hardcoded a couple.) At this point, most people would put credentials directly in their repo. I'd sit around making the Vault argument, but nobody would listen. We have now traded my ACLs and authentication stack for security on our private GitHub repos. I set to work automating changing all of our infrastructure passwords every time an intern leaves the company. We still secure the network, of course, but now there's special strings of characters in existence to work around a competent network security strategy that already existed.
7) As a growing company we introduce LDAP and Kerberos. Oh wait, AwesomeDB and its home brewed authentication stack do not support LDAP because what startup needs LDAP?
8) Another bug report is filed against AwesomeDB for "LDAP support." A bunch of startup engineers spend several months trying to understand the guts of LDAP (I feel your pain) and, worse, the cornucopia of vendor shenanigans in that space, and set the bug report to "holy shit, someday." A rough version that works with one exact schema and hopelessly avoids being just right for my needs might ship, or might not.
9) I redeploy my perfectly functional reverse proxy solution which has supported LDAP since the mid 90s and wire it up to Google OAuth for bonus points (getting rid of passwords for all corporate systems overnight! Neat!) while half the world pointlessly fights for every infrastructure service that exists to basically link in an operating system's worth of encryption and authentication support just to be a document store or search server.
Given the choice I would very much rather secure my network or use well-established software to secure an infrastructure service. Asking every database vendor in existence to reimplement this stuff is a bad idea. People who blindly advocate for this sort of work, then paper over it with "you can't trust the network" without really supporting that statement, just generate busy work for companies that write infrastructure services. If you can't trust your network, what business do you have operating a production service that maintains user data? Do you really think having ElasticSearch do its own HTTP basic is more secure than any other solution? They at least had the JVM/netty/whatever to fall back on, where a lot of these infrastructure services have had to write their own authentication.
There is a middle ground here and a lot of advocacy, both on HN and elsewhere, that misses the forest for the trees. Of course, none of this applies to MongoDB, because they were like sure, the world needs another bespoke binary protocol. You absolutely should be keeping your data on a secured network, though, and opaque strings of characters for system A to log in to system B might seem like defense in depth, but they end up in your source code, another system, or the working set of your executing process, and they are routinely captured by active adversaries. So.
Having a mode where an infrastructure service assumes a connection is legitimate and just services the client is important. Some of us have a little more confidence in our security strategy than others and feel that we know what we are doing. Wiring that mode up to a 0.0.0.0 bind, by default, is the bug.
Oh yeah, almost forgot:
10) All that custom authentication in AwesomeDB that you had an intern write becomes a CVE factory and starts reflecting on the security of the product. Womp womp.
Because getting RBAC right for individual indexes in Elasticsearch is not necessarily trivial. Example, I have multiple groups and users. And lets say I want to give some users access to these indexes but not these other ones and this group can access these indexes read-only and this group can add or delete but not access the admin endpoints etc. Just "putting it behind your company's proxy" doesn't work out of the box. And integrating this on Elasticsearch client Nodes - nodes where node.data is set to false and node.master is set to false, would have been very logical.
Also not every company has AAA with full LDAP integration, i.e any early stage start up that consists of a small handful of people wearing many hats because some of which are not their area of expertise. Lots of these types of companies are also using Elasticsearch for full text indexing.
Conflating the two is wrong and, yes, many services themselves conflate the two which reduces the flexibility of both. Why not let my system do the AA and tell the backend what user it is so that RBAC can be implemented? I haven't used Shield but I'm guessing this is difficult to separate; in a number of other systems it is.
The key to your office is not nearly the same as the key to each drawer of your filing cabinet. Each can be shared independently and serve different purposes. You might decide to never use your office key, for example, because of the armed linebacker in your lobby. In my experience, most of the effort to make an infrastructure service speak authentication is intended for gatekeeping and RBAC comes later. ElasticSearch obviously went all the way with it. This whole conversation about infrastructure authorization tends to make people forget what the actual threat vectors are, which is really the subtext of my thesis here.
Every company has at least a subset of AAA, and hand waving it away as "eh, early startup stuff" is also dangerous. Putting stuff behind Google SSO is nice for startups these days and, boom, AA-sorta-A.
Out of interest how did you handle the Google OAuth integration? Some kind of JWT module for httpd/nginx that validates the token, and acts on the subject?
Openresty/nginx is like an operations Leatherman. Very handy.
The default is now a randomly generated password. You have to overwrite it with --initialize-insecure if you want to setup with a blank password.
If you want to double check, try 'mysql --protocol=tcp'.
Security and usability are always at odds. If you make security a pain, people will give up on security. Mongo is the ultimate manifestation of that principle. Developers were driven to Mongo by opaque and painful admin tasks.
`apt-get install mysql-server` makes you enter a "root" password out of the box.
`apt-get install postgresql` create a local user `postgres` with access the local database.
Both bind to only localhost by default.
Those sound like easy, reasonable defaults to me.
(And this isn't exclusive to Debian-likes: RHEL almost certainly shares the exact same benefits)
Does it!? I'm sure I've had to do that myself.
No they are not. Start the server for the first time? ask users to create an admin account, then display a notice saying they didn't configure SSL or something else properly. Problem solved.
Each time the server is started and the setup isn't secure enough, display a message. That's how you do it.
If you put something on the internet, it's on you to keep it secure.
Absolutely not. How does having to log in affect usability of a system itself in any way?
Security and convenience are often (but not always!) at odds.
Since then, I've learned to not assume that products are secure-by-default. At the same time, I kind of thought we learned our lesson and cut that shit out low these many years later. Add a line to a text config file that's probably buried eight directories down in a hierarchy that's owned by root? (I'm just hyperbolically guessing for effect; I generally avoid Mongo.) Do it, or you're hacked? And it's been this way for years? Come on.
It's not though. And it never really has been. A simple approach to security is to only expose the absolute minimum to the internet - you close everything and then open one thing at a time until your service works. Had those 30,000 MongoDB instances been sitting on IP addresses behind routers that only allowed local traffic, or specific IP addresses, then they would have been a lot harder to steal data from. The fact that they were willing to give up data to an unauthorised user would still be a problem, but it'd be a local network problem rather than something anyone with a port scanner could do.
This is not a case of discovering some arcane setting you have to change to make sure your data is secure. This hack was a case of people putting services on open connections without bothering to follow even basic practises. It's bad that MongoDB gave up data without auth details, but your infrastructure can be designed to block an attacker even when they have security details so the fact that MongoDB had this problem is never going to be the whole story.
But that's bad old NT4-era Microsoft. Not 201x MongoDB.
As much as I like using open source alternatives like Postgres, SQL Server for me is by far the best database system, from standpoints of performance, features and tools.
* Almost anyone, there are definitely potential pitfalls.
This is the equivalent of leaving your data in a file cabinet in your lobby and being surprised someone took it. The answer isn't to add self-locking little office locks to the file cabinets by default, it's to educate folks that this is a file cabinet that's designed to be inside the office, not in a publicly accessible place.
No, seriously. Please. I've been banging that drum for years.
1. note the default value of "bind to *": http://dev.mysql.com/doc/refman/5.7/en/server-options.html#o...
- require authentication for remote network connections
- allow r/w from any local connection, tcp or unix
Random devs installing mongo on their workstation will Just Work, but won't expose customer data to the internet.
If the software is insecure out of the box, that's crap vendor behavior.
Originally, I thought the problem would disappear once MongoDB changed the default from "0.0.0.0" to "localhost". However, that has had no impact on the number of exposed MongoDB instances on the Internet. Bad defaults are definitely a problem but in this case that isn't what's happened.
Also note that the most popular location where these instances are hosted is on AWS where you explicitly have to open up the firewall.
One of my concerns is that people are piling onto MongoDB because "it's web scale" and ignoring potentially systemic security issues that aren't specific to MongoDB. The same issue exists for Riak, Cassandra, Memcache etc.:
If you're running any servers yourself, there's no reason to open up network access to the world. At most the specific ports you want to expose, to the specific IP ranges you want to allow, should be white listed. A default of everything to everywhere is insane.
Even with SSH key based authentication (v.s. say passwords) I'd consider it inept to expose all your infrastructure publicly. Set up a VPC or whatever the local equivalent is for your hosting environment and proxy SSH access via a bastion host. If you whitelist the inbound addresses to the bastion host (let's say to restrict it to just your office) then you've also eliminated the vast majority of auth log spam too.
Unfortunately the people that would understand and implement things like this are the same set of people that wouldn't have publicly exposed MongoDB instances in the first place.
Both Windows and UNIX/Linux programs have had interactive, guided installers, graphical or text-based, for years. Package managers support post-install scripts.
Organizational division-of-responsibility issues are not for applications to solve, but having sane defaults and performing user-supplied configuration is. This could've been handled with:
> Do you want to allow MongoDB to be accessible externally? (WARNING: this means ANYONE on the internet can connect. You will be asked to set up a username and password to protect your database. Don't do this if you're unsure or if your organization has to approve access.)
> Now that your MongoDB is open to the Internet, choose a username and a password to protect it from intruders.
When it comes to server security, you are responsible, and should never make assumptions. If you don't know how, or simply can't be bothered to check, then you shouldn't be administering a server.
Also a firewall is a most basic requirement, even something simple like UFW would do.
You didn't do anything special to connect to MongoDB, and are able to connect to it from your other servers. Why would you expect the same wouldn't apply to everyone?
I will say that people who ran MongoDB on their app server get somewhat of a pass on this.I could imagine that they would find the idea of MongoDB running open to the world with no auth to be so nuts that they assumed it was only allowing connections from the local machine.
They leave mysqld bound to 0.0.0.0 because they don't know any better. They SSH as root because they don't know any better. They have a default WordPress install with the config db sitting in webroot.
But hey, their website works and might one day make them some money.
When you build a product or service, you should own it end-to-end OR simply let it be known that you are unable or unwilling to do so.
If you take it seriously, you'll know what can sink your project/business -- and stuff like leaving your entire database publically available with no auth, is one of them.
But also, if Ubuntu didn't run network services by default, there would be no need to secure them with iptables. So defaults affect defaults.
Source code to print your own:
Any machine connected to the Internet should start with a configuration equivalent to:
ufw default deny incoming
ufw allow ssh
I gave them full access to logs, etc. and the support response was basically that they couldn't determine why it wasn't working or alerting, but it it works for them, so I should migrate to their hosted solution.
I sometimes think of the potential company-ending cluster-fudge that could have caused, while I stare wide-eyed at the ceiling at night.
Opening/closing ports is the firewall's job. Mongo's job is storing and retrieving data (now, how poorly it does at that is another conversation).
Having both Mongo and the Firewall handle ports means you do everything twice, which violates OnceAndOnlyOnce.
I'm not a security expert (far from it) but I hope that I understand enough the importance of security to learn a bit about it and implement it as much as I can.
Secure defaults is now maybe the first concept I'm trying to explain to people in my company.
First off, there's the constant problem of default passwords. The big IoT compromises lately were due to default passwords, and ever since they became popular they've been a bane to security everywhere. Wifi APs used to use defaults, but luckily most come with a randomly assigned password now (except those provided by an ISP, occasionally). All software that matters should not use default passwords.
Second, a "default" is not intended to be an operational setting. When developing software, a default is literally a placeholder setting so that your software won't immediately crash if a configuration value is missing. Some software is effectively meaningless without a configuration, but if they are obscenely permissive they can allow a user to still take advantage of it, which is how MongoDB is set up.
Finally, there is effectively no such thing as a secure default, because something isn't really secure until it's been vetted through a process. Pretending your software is secure when you have literally not given it a second thought is just setting you up for eventual disaster. People should be actively thinking about their security if they care about it; defaults just leave them blissfully ignorant.
In my ideal world we would require software to no longer have defaults. Demand that users be aware of the configuration and running of their software so they understand what's involved with how it runs. This would teach them more about the software they're using, which would enable them to take better advantage of it, as well as make it more secure. If you don't have time to do that you probably shouldn't be using it.
Everyone has a different security model. In my case all my DB servers live behind and API layer on the internal network, and the DMZ web layer talks to the API. That makes keeping things secure MUCH easier...
Here's some documentation to get you started.
This really should have been titled "the importance of good DBA's..."
Hadoop, by default, enables WebHDFS, which provides read+write access to HDFS over HTTP. The service runs on name node using the same port which displays the Hadoop Jobs Web UI.
Now, it is very easy to not know anything about WebHDFS and every day authenticate/login to server, submit jobs, and track the status on WebUI (which is public). But then anyone can send a POST request pretending to be any user and Hadoop will happily execute the command (including deleting all files).
Maybe Hadoop developers believe that all Hadoop installations are behind a firewall (ours was, but an exception was made for Jobs Web UI as it seemed to be a read-only status page for running jobs) and/or have Kerberos integrated with Hadoop for authentication.
But that being said, enabling HDFS access over HTTP by default seems to a debatable choice. Also, if you have a small Hadoop cluster running, do set "dfs.webhdfs.enabled" property to False in hdfs-site.xml.
That is fundamentally wrong application design. The application connects to the database, so the database must not be public accessible via internet.
I mean, if you push something into a production environment, you must think a bit about the tools you are using and how they are configured for security.
On a development system, I don't want to setup users or auth-mechanisms.
There are so many resources from MongoDB, how you can secure your MongoDB database:
- Security Architecture White Paper: https://www.mongodb.com/collateral/mongodb-security-architec...
- Security Best Practices blog post: https://www.mongodb.com/blog/post/how-to-avoid-a-malicious-a...
There is even a checklist which you can work through:
And for those, who don't want to read anything ;-), there is a comprehensive Video-Course on the MongoDB-university:
If I enter a car and don't use the seat-belt or switch on the light at night (defaults are off), then I cannot blame the car-manufacturer. If I use something important (here the database in a production environment), then I have to learn the ropes and read the material specifically on security topics.
I guess secure by default means there aren't likely to be massive hacks like this, but it does not mean people are running securely.
Convenience "sells" even at the expense of security (and in this case, actually working properly at all!) and if you don't make it as easy as possible be prepared to support people who refuse to read the instructions.
This was only recently added as a startup check.
Secure out of the box is nice, except that when the boxing is done by others, and delivered to under-informed, unexperienced people. It's too easy to npm yourself a "full stack" or just pick a Docker container with full stack on it, then place it on public inter-web... there are dangers to be had, and others can save you. It's up to you.
And yeah, the click-bait titles including the word "hack" are laughable.
Default bind to localhost. If you want to handle external connections, the network layer provides your security.