We are completely locked out of our Atlas account and the support portal right now. We Okta-auth with Mongo and all attempts to auth right now are failing with "The request contained invalid data." displayed on their login screen.
Of course, the support portal requires you to auth to use it...to get help with auth failing.
Anyone else seeing issues getting in to their dashboard?
Edit: Auth started working for us and dashboard access became available for us around 5:15 pm ET.
The login issues are unrelated to the security incident. We notified all of our customers and users concurrently resulting in a spike in login attempts. Please try again in a few minutes if you are still having trouble logging in.
I'm trying my personal account as well and it's telling me MFA isn't set up (it is) and it's making me go through the MFA setup flow again. All attempts to setup another 2FA code in 1Password or to get even an SMS code sent to my phone are failing.
Edit: Personal account with a TOTP 2FA is working again now as well.
SMS 2FA is better than no MFA at all, despite the very valid concerns about SMS. It at least protects against credential stuffing and similar automated attacks.
I guess I've always cynically assumed that companies want my phone number to make the data they gather more valuable by making it easier to link with a unique index like a phone number.
Well a simswap attack requires the account password, since otherwise you would not be able to receive an SMS message for the two factor part.
But without two factor, only your account credentials are needed.
So yeah, it's definitely better than nothing, you are effectively forcing your opponent to social engineer your carrier, and doing that generally requires knowing the full number and usually at least your name, if not more identifying information that's harder to get, like social security number or equivalent.
Sure, TOTP or other two factor mechanisms are better because they require access to one of your authenticated devices (assuming the TOTP isn't done by a secure enclave), but SMS two factor is definitely better than disabling two factor.
But you have setup SMS 2FA enabled, which is convenient this time but a big security hole. You should consider disabling it once the situation comes back to normal.
I bet that's because different parts of their stack disagreed. Obviously a two factor setup should not be acceptable when one is already in place-- if the frontend thought it wasn't but the backend/auth services thought it was, it could explain that.
Nice and to the point, makes it clear that this is early, explains the current scope, tells us to expect a follow up as the information makes its way to them.
I like this tbh and I hope people won't punish them for not including more info when this is clearly in the early days of investigation.
It was only DETECTED on the 13th, and they suspect had been going on 'for some time'. And basically not sure if user data was touched but they suspect or haven't provided it yet buly saying'NOT'.
Your options are: (A) Vendor waits until all the facts are in place and the investigation is finished or (B) Vendor tells customers as early as practical so they can take their own mitigation steps.
You do not have the option of (C) Vendor should tell me about a breach they don't yet know about.
I don't use Atlas but I do use self hosted MongoDB, and have been pretty happy with that product. I have the impression that a lot of the dirt slung at Mongo was about unreliability and data loss of the core product early on, which (knock on wood) hasn't been a problem for me on the small to medium scale use cases I've deployed it on. Seems reliability has taken a lot of positive strides over the years.
Thankfully all of my users are extreme statistical aberrations who do not re-use the same memorized password (or a variation on it) for more than one thing, ever, at all
OR
they diligently watch every single possible place they have ever re-used any of their memorized passwords, with the globally mandated and complied with reporting, so they can know if a password they once re-used at grandmas-cookies.blog.example.com has been compromised.
The fact that all websites, servers, systems (etc.) check to see if passwords are known to be compromised (since NIST says verifiers will do that) makes things a lot easier, too.
This highlights risks of extreme consolidation - even if Atlas customers were not affected it is natural for them to be concerned after announcement overwhelming web site or support channels.
More independent MongoDB DBaaS providers is what would offer true redundancy in this case, though it is highly restricted due to SSPL license change.
Hopefully FerretDB will be successful building feasible alternative
MongoDB is investigating a security incident involving unauthorized access to certain MongoDB corporate systems. This includes exposure of customer account metadata and contact information. At this time, we are NOT aware of any exposure to the data that customers store in MongoDB Atlas.
We detected suspicious activity on Wednesday (Dec. 13th, 2023) evening US Eastern Standard Time and immediately activated our incident response process. We are still conducting an active investigation and believe that this unauthorized access has been going on for some period of time before discovery. We have also started notifying relevant authorities.
What should you do next?
Since we are aware that some customer account metadata and contact information was accessed, please be vigilant for social engineering and phishing attacks.
If not already implemented, we encourage all customers to activate phishing-resistant multi-factor authentication (MFA) and regularly rotate passwords.
MongoDB will continue to update mongodb.com/alerts with additional information as we continue to investigate the matter.
Isn’t mongodb data losses commonly referred related to their use of fsync()? From what I vaguely remember they call fsync() every 100ms or so and just assume everything went fine, resulting in potential data loss.
No. The default write concern as of version 5 is "majority" which "Requests acknowledgment that write operations have been durably committed to the calculated majority of the data-bearing voting members" (as long as writeConcernMajorityJournalDefault is true which is the default).
It all depends on your writeConcern setting. This option is super critical to understand when deploying an MongoDb. Older versions had defaults that were more optimized for availability than consistency.
Even so, not flushing each write is not as bad as it sounds, if you have a 3-node replicaset and your w-parameter is set to majority (default), it means at least 2 servers have the write in memory. It would take both of them crashing at the same time for the unflushed write to be lost.
The idea is that MongoDB allows you to choose which corner of the CAP triangle you want, if you chose AP, that’s your decision. The defaults can of course be argued and I believe it’s been gradually moving over to more and more C for each version. Nowadays the journal does get flushed as next comment described.
Yep, Jepsen has not only confirmed but later reconfirmed that MongoDB has for many years been on par with other NoSQL databases when it comes to transaction guarantees and data consistency.
Also, "2020-05-26: MongoDB identified a bug in the transaction retry mechanism [2] which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8." [1]
- Highly-flexible. Because you're not developing against a schema, you can, for example, retool a feature and its data quickly without having to stress about migrations. A big advantage for a startup looking to move fast.
- Queries look more like application code so you're not wasting mental cycles and time trying to translate an idea into a SQL query. From experience, this leads to less-fragile queries.
- Little to no concern over injection attacks (you have to go out of your way to create potential for them).
- Easier to write non-trivial queries than with SQL (IMO).
- Type-casting data can be done in code as opposed to with SQL you have to use inline, platform-specific functions like field_name::timestamp.
- A single source of truth for how to query and develop with it (with SQL, you're almost always developing against a flavor of it).
- Scales reasonably well (and easily) for a majority of use cases.
- No room for dogmatic fervor/confusion around a specific variety of MongoDB as there's only one variety.
> Because you're not developing against a schema, you can, for example, retool a feature and its data quickly without having to stress about migrations. A big advantage for a startup looking to move fast.
Why wouldn’t you need to worry about migrations without developing against a schema? You’ll need to worry more about migrations because your data will be more messy.
Not necessarily. If you're utterly careless, sure. A MongoDB migration is far less of a headache than a SQL one (you're just writing code to map/replace values). With SQL, you have to be frustratingly surgical about everything which can slow momentum to a crawl (read: punishment for mistakes in the migration is far worse than with MongoDB in my experience).
> Why wouldn’t you need to worry about migrations without developing against a schema? You’ll need to worry more about migrations because your data will be more messy.
If you're careless with your data, yes. "With great power comes great responsibility."
Mongo is the main nosql choice. Mongo is great if you think a flexible schema is good. Mongo is not great if you think a flexible schema is bad. That sums it up
This is too reductive, you can essentially have flexible schemas with most modern relational databases and without the downsides of document-based DBs.
In 99% of the cases, even if you need a flexible schema, PostgreSQL will remain the best choice.
I use it on-prem (well, on a VPS). It stores JSON documents, and it's easy to work with. If your data looks like a tree, it works pretty well, also for large documents. If you depend on relations between documents, you're better off with an SQL database, but note that for many cases --I'd say practically all mundane cases-- there's really no need for relations the SQL way. MongoDB also does relations, but a bit more convoluted.
But you don't need to horizontally scale if you use a beefy Postgres instance. Horizontal scaling is a problem created by using an inefficient DB in the first place.
I have DB clusters with masters with a terrabyte of ram. Going beyond that is a PITA with warming up cache, backups, and so on. There is a reason there's a huge industry for shared databases.
Also, query for query, Mongo isn't going to be that much slower than PG, and faster for some usage patterns...
It's a really great alternative to firebase for mobile apps. Works pretty nicely with Realm so you get offline first db with powerful syncing. All the benefits of realm on the edge device with the power of the mongo platform. I dismissed mongo atlas for years because "mongo", until I finally gave it a chance. Overall been pretty pleased.
We use MongoDB in conjunction with RealmDB to build an offline first mobile app. For this specifically it works very well. You basically define which parts of your document collection should be synced to which device, for example based on a query that contains the user id.
MongoDB takes care of syncing the right data to the right device when internet connection is available. On the device, the data is stored locally in RealmDB, which represents the other side of the sync.
This is not easy to do with PostgreSQL, which we use for all other scenarios requiring a DB.
That's why I've added the last part. Mongodb highlights old Jepsen analysis' on their page (https://www.mongodb.com/jepsen), but they never got around to ask them for another analysis to show things are so much better with MongoDB 5 or 6? Yeah. Sure. People are free to believe whatever they want, I'll continue to highlight the best info available.
> That they haven't paid for a thorough follow-up analysis after their supposed fix is all one needs to know
As much fun as Jepsen is, I'd probably not follow up with a company that turned my product into a mockery. I enjoy the hell out of reading that report as an outsider, and I personally would be a repeat customer, but I can see how a company might not love the writeup.
Sure, but to me that just looks like they have no faith in the supposed quality of their product. If they really fixed MongoDB there's nothing to fear. Aphyr is harsh, but fair.
Again, I can see why they would not be interested in engaging again. You can think that it's because they're afraid, but they also may have just really disliked the service they paid for.
aphyr seems to still work on it, so probably just not that many companies who want to pay to get told the truth anymore: https://github.com/jepsen-io/jepsen
Why are people still choosing Mongo over Postgres these days? If there's something I'm missing, I'm genuinely curious as I'm not against json data and frequency use jsonb tables in Postgres.
People use MongoDB because it’s easy to get started. It does “db stuff” and “authentication”. I’ve given up trying to fight the trend. I just recognize immediately when it is used early on that the devs are still operating with training wheels on.
Sometimes this is the case but not always... It's nice to just work with objects in some languages... for some projects. That's engineering - picking trade offs :)
Yeah, there is. PG has its own warts that Mongo does not have. Also the JSONB query language sucks. See appending an array in PG [0] vs Mongo [1]. Also PG does not provide a way to atomically append to the array, you have to lock the row via SELECT ... FOR UPDATE. Mongo's $push, $set, etc, are atomic.
Recent 7.0.0 version has dropped old and introduced quite broken new query planner. Caused a lot of our queries to miss. We’ve had the displeasure to work with the support on multiple related issues.
The hype died but the tool is almost mature now and quite useful, plenty of paying enterprise customers who fly low but have fat wallets. I’d give it a few more years to be a primary data store, but it’ll do if you want a horizontally scalable, capable and performant document database to give some breathing room to the primary SQL cluster.
For now, there are way too many critical issues found late in the release cycle.
They state these things in their quarterly filings with the SEC, in which to my knowledge it is not legal to knowingly misrepresent facts. If you have actual proof that MongoDBs auditors are lying to the SEC, you can probably get a pretty good whistleblower reward or at the very least make a ton of money selling this money to hedge funds specializing in shorting failing companies.
The SEC listings don’t contain real numbers about who is using the software. They can claim a bsbillion people use it as long as they can launder investor money into fake usage (like substack does) These numbers support my claim. If they had REALLY been growing they’d be a huge company right now, not having to pay people to use there products
My “inside information” is just basic knowledge of the software industry. If MongoDB is growing is like claiming Morbius is a good movie. It’s just silly. Go ahead disagree with be, but it’s kinda silly.
Just because you don't like something or it isn't favored in the HN diaspora does not mean it is an unsuccessful product. There is a great big world outside of your bubble.
Anyone can provide Postgres, MySQL or other open source databases as a service.
For this reason, there are many providers to choose from, and there is a healthy amount of innovation and competition in the space. Prices are set by market and demand, as it should be.
And then there is MongoDB where only a handful of providers could negotiate a license, and the price is set by MongoDB Inc.
In my opinion this is by no means "fine" from a user perspective as we are talking about database software.
If anyone did freeriding, it is MongoDB Inc. who chose to freeride on the open source community for marketing purposes, before switching to SSPL.
It is the opposite. Amazon would have released MongoDB as a service, same as they do for PostgreSQL or MySQL. As MongoDB changed license they implemented DocumentDB instead.
Note AWS significantly contributes to PostgreSQL and MySQL communities (though you could always want even more) but of course does not to MongoDB. While this is fine for MongoDB Inc I think it is not great for MongoDB community at large
What's nice about Postgres is there's a ton of Postgres compatible products that do scale for the 10% who actually need it. And it's still all just Postgres / SQL.
Nosql is a fun target to beat up on of late. But there are good, even infamous, reasons to avoid SQL. Particular if you want to accomplish flexible record queries from untrusted clients.
All you do is poop all over the story about postgres. I'm convinced that no use cases will convince you of anything. I'm not really looking to involve myself in a database holy war.
Is jsonb in Postgres not flexible enough? I dump external json in there all the time (like large API responses). The jsonb operators work well. And there's an escape hatch that lets you easily convert json to a table. And importantly, you get indexes with Postgres.
Might I suggest you actually use mongo on a project before engaging further on the topic. It clearly has had a lot more thought put into it in the context of json documents.
If postgres works great for your use cases, great, go be happy with the tool.
Definitely a better solution than the brick wall he hit. Complicated beastie. I can see the appeal if you just love, eat, and breath SQL.
But a query language within a query language feels bolted onto something alien to do something pretty basic. Pass. If I'm teaching this to new engineers, pass also.
MongoDB’s SSPL is neither an open source license[1] nor, most likely, a free software one[2]. Its definition of offering the licensed software as a service is so broad most Linux distributions[3–6] flat out refuse to ship MongoDB (not even in a nonfree repository or the equivalent) so as to (among other things) avoid placing the operators of their package mirrors in legal jeopardy.
"It should be noted that the new license maintains all of the same freedoms the community has always had with MongoDB under AGPL - they are free to use, review, modify, and redistribute the source code. The only changes are additional terms that make explicit the conditions for offering a publicly available MongoDB as a service.
Obviously, this new license helps our business, but it is also important for the MongoDB community. MongoDB has invested over $300M in R&D over the past decade to offer an open database for everyone, and with this change, MongoDB will continue to be able to aggressively invest in R&D to drive further innovation and value for the community."
Really? How many open source databases do you offer? Some may say it’s not right for randos to complain when you give something away and they complain that it’s missing basics. I just happy someone else wrote most of what I need and I can extend it if needed.
This is almost certainly normal activity under a 10b-5 plan, meant to protect specifically against suspicion of insider trading, which is what you’re implying.
If you’re trying to solve a problem and think “I’ll use mongodb”, well now you’ve got two problems.
Just pick postgres. If you have unstructured data as input, either put in the effort to create some kind of schema for it if you can or just use jsonb if you can’t.
I'm of a similar opinion. I experienced all the issues of MongoDB's infancy; it wasn't a good time, but the mmap functionality seemed worth it at the time if you had enough memory or constrained your access patterns appropriately.
Nowadays, PostgreSQL has JSON/JSONB types, a full suite of extensions like pgvector and PostGIS, and I can scale with Citus or use it in any of the big managed clouds.
From a functionality perspective, MS SQL Server makes a more compelling alternative to me simply by way of its native graph database support.
Mostly that Postgres is amazing, runs anywhere and supports jsonb (which I use a lot). And I don't want to worry too much about mis-modeling; that's the entire point of a relational database with a type system.
Plus all the Mongo horror stories from people who I know and hold in high esteem. And recently talking to a data person explaining what a pita getting data out of their Mongo monstrosity into a format adequate for analysis.
We started our startup on Mongo. Hit some pretty hard performance problems and eventually did a multi month long migration to Postgres (Aurora on AWS).
MongoDB is only a valid choice if all you're doing is story key document pairs. The moment you need joins or any sort of aggregations like count/sum e.t.c - Mongo perf is horrendous. Postgres runs circles around Mongo in every way.
With jsonb columns, not much is lost. SQL is a huuuuge bonus. Mongo query language is a giant pain for everyone on team to learn and manage.
Much like saying “no offense” doesn’t make something not offensive, saying “honest question” doesn’t make a question not disingenuous. If you find yourself typing “don’t downvote” you should consider rephrasing your question to not be worthy of downvotes.
This post could just be “can you explain your experiences that have lead you to this conclusion” and we’d all be better off.
Of course, the support portal requires you to auth to use it...to get help with auth failing.
Anyone else seeing issues getting in to their dashboard?
Edit: Auth started working for us and dashboard access became available for us around 5:15 pm ET.