

Scaling MongoDB at Mailbox - ashishgandhi
https://tech.dropbox.com/2013/09/scaling-mongodb-at-mailbox/

======
MichaelGG
>one performance issue that impacted us was MongoDB’s database-level write
lock.

People give MSSQL shit for having row-level locks (if you don't use their MVCC
option), yet how is it that Mongo runs with a _database-wide_ option and
people don't immediately laugh and walk away? Is the hype so powerful that
people just shrug about a huge mutex?

~~~
lucisferre
The problems people encounter with SQL performance generally (or at least
also) are on the read side where query optimization can become a pretty hard
problem. Basically, the short answer is people are using SQL and noSQL very
differently (as one would expect).

cue: someone telling me SQL query optimization is easy and people are just
idiots.

~~~
justinsb
It's a valid point taken by itself. But I would respectfully suggest that it's
not a particularly useful statement, because you're not comparing SQL to
anything. "Complex SQL queries are hard" because "complex queries are hard".
If there was a magical tool where you could do complex querying and always
have the right answer in the minimal time, then these discussions would never
arise. (To a debatable extent, that is one of the things people are paying for
when they choose a "better" database: the hope that they get closer to the
ideal on complex queries)

Personally, I would much rather perform a complex query in SQL than NoSQL. The
base-line is likely to be not-terrible, which may be good enough for single-
use queries. Optimization is much easier (IMHO) - SQL is a DSL for querying
and indexing. Compared to having to write code to do that (which seems to be
the primary NoSQL approach), the DSL approach seems much more efficient. SQL
isn't perfect, but it beats no-DSL-support or another DSL that is even worse
than SQL :-)

~~~
annnnd
The caveat being that in document-oriented store (like MongoDB) the queries
tend to be simple(r) - after all, you have no joins, so how complex can they
be? Instead of complex queries you need to use appropriate document schemes
and/or application logic.

MongoDB has its problems (for instance document schemes can't be enforced,
disk space consumption concept is unintuitive,...) and relational DBs have
their strengths (ACID, SQL is standard among DBs,...). But very few people
ever reach a point where the global write lock becomes a problem. 10gen knows
exactly which features are vital to most of their users and which are not...

~~~
justinsb
But you still need to join in NoSQL, for any query that isn't along the
natural hierarchy of the data.

The relational model takes this observation and concludes that there should be
no "natural hierarchy" in the logical model (the physical model is a separate
question.) It's a _theoretically_ beautiful idea. The counter-intuitive _real-
world_ result is that the theoretical approach also yielded faster systems
than alternative philosophies. I think that's the reason why the relational
model has dominated for the past 30+ years.

You might well argue that that's because a _good_ NoSQL implementation has yet
to be created (the no true Scotsman argument). That may well prove to be true
one day, but I would bet that a good non-relational implementation will be a
derivative from the relational world (Postgres with hstore, or Google's F1
with Protobufs); not from a project whose starting axiom is to get rid of
SQL/relational.

------
stonewhite
I dont understand how MongoDB can stay as a shiny tool given its long list of
shortcomings. It is so elastic that I think it really has no true form (object
cache, mq, relational??, key-value bucket). Lately I haven't read any news
about a company migrating to Mongo, but rather most were either departing from
mongo because of some catastrophic outage or some mission impossible operation
to handle the shortcomings, like this one. In its current state it looks more
like a prototyping tool that allows you to delay some tech decisions until
your product matures, rather than a production tool. Or it is just me that
wants at least 5 major versions on his production database.

~~~
threeseed
Simple. There are some use cases for which MongoDB will be 5x faster than any
other system. It is relatively unique among databases in that it is a document
store as opposed to a relational system.

It is also very easy to use and manage.

~~~
Pxtl
> It is also very easy to use and manage.

This is something that SQL database fans don't get. A small start-up can't
afford to have a team of guys fiddling with incredibly obscure performance
tuning settings.

A developer might not know how to optimize a slow-running Postgre query, but
he knows how to cache data and denormalize in a manner that makes the query he
wants to run fast. And a document-store plays nice with this.

Any simple relational beauty of SQL databases falls down once you get over a
few million rows, and then the hideous hacks and bizarre settings start coming
out. At that point you need a pile of domain-specific knowledge to make them
play nice.

Mongo takes the domain-specific knowledge out of the equation and lets the
developers apply their existing skill-set to the data.

I've you've got a thousand employees? Get a few DBAs and Postgre.

If you've got, like, five?

~~~
stonewhite
One thing you don't know or faced yet is that mongo fails, and sometimes it
fails so hard that you would rather kick some rocks.

For example, if you start a background index generation on an eventually
consistent replica set, indexing on secondary nodes are done foreground. Which
means you only accept reads from slaves but slaves are unresponsive because of
the index generation. In this state, if you try to do anything fancy your data
will go corrupt. Only way out is to wait through the outage (which I find it
pretty hard to do so). This is still not solved in 2.4, waiting for 2.6.

Replica sets with all secondaries which can't elect a primary because it lost
a node, or the mostly random primary-secondary switches that drops all
connections, seldomly primary reelecting itself meanwhile dropping connections
for no apparent reasons. Mongo offers tin foil hats for integrity, consistency
and reliability. So yeah, I'd rather examine and understand why an SQL query
is slow. Because it is at least deterministic, which in mongo nothing really
is.

Postgres supports free form json, XML or hstore document formats by the way,
couchDb has its own specific features as a document db too. I still don't see
why people want to go on with mongo this bad.

~~~
Diederich
I've been running, on a five minute cron, kill -9 on the master mongo instance
in our QA lab for a good long time. 24/7.

There's a program running that, 100 times per second, reads a document, MD5's
the field, and writes it back. At the same time, it reads a file from the
local filesystem, MD5's it, and writes it back. The document and the local
filesystem file started with the same value.

After a few thousand kill -9's on the master instance, the local file and the
mongo document are still identical.

I've been running MongoDB in production since 2010.

It's definitely possible to use Mongo in a way that isn't safe for your
particular use case. But we're doing it correctly.

I haven't lost a bit of data in more than three years of MongoDB.

Mongo has a lot of limitations. We're currently researching various 'big data'
solutions, because for us, Mongo doesn't fit that.

For ease of development (in dynamic languages, where your in-program data
structures and in-database documents look almost identical), safety, and lack
of headaches, MongoDB has been a consistent win for me and the teams I've been
on.

~~~
justinsb
Is your test producing a new document value 100 times a second, or just
writing the same value back over and over again?

It sounds like it might be the latter, which is not a particularly stressful
test (because you can't detect data rollback).

I'm more familiar with relational database internals, but I wouldn't be
surprised if a DB just optimized out the unchanged-write entirely (they'd
still need to read the current row value, but they don't have to invoke any
data modification code once they see the value hasn't changed).

For a good test, you really want to simulate a power-loss, which you aren't
getting when you do a process-level kill, because all the OS cache/buffers
survives. You can do simulate this with a VM, or with a loopback device. I'd
be amazed if MongoDB passed a changing-100-times-a-second test then. I'd be
amazed if any database passed it. I'd even be amazed if two filesystems passed
:-)

~~~
Diederich
I'm reading the document, hashing it, and writing the hashed value back. So it
changes every time.

I plan on extending this test by blocking 27017 with iptables, then doing the
kill -9, then wiping out all of the database files. That'll be fun. :)

------
raj_bains
You need consistency (distributed mvcc), row-level locking and scale-out
architecture. Clustrix might be a better fit for the workload. Clustrix
customer Twoo.com has 336 core deployment (168 master, 168 slave, 21 nodes
each), they have millions of users and billions of transactions per day. Their
application still thinks it's talking to a single MySQL database and they
don't have a DBA and they have never thought about shard keys etc. We in the
database industry should be solving these problems for you.

~~~
wheaties
Wish someone would do that for Postgres.

~~~
lucian1900
There's Postgres-XC [1], but it's a fork and I don't know how actively
developed.

1\. [http://sourceforge.net/apps/mediawiki/postgres-
xc/index.php?...](http://sourceforge.net/apps/mediawiki/postgres-
xc/index.php?title=Main_Page)

------
mlmilleratmit
Nice work -- that's a hard problem. I've felt some of this pain as I helped
Cloudant customers make hot migrations from mongo to into Cloudant. Your first
two figures make it clear just how challenging hot-replications are
(especially master-master), not to mention handling failure scenarios. For all
of the great things about Mongo, there's something very awesome to be said for
CouchDB's MVCC replication model. Glad to see that you're open sourcing those
tools. Maybe they could be extended to make a MongoDB <==> CouchDB replicator.

------
joeblau
I worked on a project where we used Mongo to collect analytics and while it
worked, there were always problems. We had a problem with our shard key so one
shard received more data which crashed it, overloaded the other two shards and
crashed the whole system. A second issue we had was trying to read from Mongo
while we were storing data which caused a lot of lock contention. The question
I have is, knowing what you know now, would you have still gone with Mongo?

------
jessaustin
Interesting:

 _...a single Python process using gevent and pymongo can copy a large MongoDB
collection in half the time that mongodump (written in C++) takes, even when
the MongoDB client and server are on the same machine._

~~~
MBCook
I would guess this is for the reason posted in the article: the Python code is
multithreaded (sort of, gevent) but the C++ code is only single threaded.

~~~
jessaustin
Sure, but that seems to suggest a _meta_ -reason: the standard utility
included with the package isn't asynchronous [sort of orthogonal to
threadedness] because it would be a PITA to write an asynchronous utility in
C++.

~~~
leif
That's not true, I've had no trouble writing very fast threaded C++ programs
with the mongo client.

~~~
jessaustin
1\. That's not what I said.

2\. I'm sure the Mongo devs would accept pull requests for mongodump.

~~~
leif
1\. > it would be a PITA to write an asynchronous utility in C++

2\. Doubt it, and there's no reason for me to write one for them.

~~~
jessaustin
Asynchronous ≠ multithreaded. TFA actually tried both ways, measured the
difference, and found async to be much faster.

------
mbroberg
We hear similar stories from the field here at Cloudant. Distributed
databases, scaling clusters -- that's hard stuff. Nice to see such a
thoughtful post.

------
mcot2
It's funny to read these comments trashing mongodb. Perhaps people should do a
little less HN ranting and a little more serious study about the problem they
need to solve. The design, benefits and tradeoffs of mongodb are well
understood. If they don't fit your application, don't use it.

------
aaronblohowiak
Summary: vertical partition on collection. They wrote some new code to make
moving a collection easier.

------
bernatfp
I'm aware of MongoDB issues with scaling, and I'm trying to avoid it as much
as I can, but I still can't find any other alternative that offers geolocated
queries.

~~~
bonzoesc
From firsthand experience, the PostGIS extension for PostgreSQL is fantastic.

A couple of my coworkers at Basho have done geospatial work with Riak, our
scalable, distributed database: [http://basho.com/indexing-the-zombie-
apocalypse-with-riak/](http://basho.com/indexing-the-zombie-apocalypse-with-
riak/)

------
ddorian43
Why has the new system less write lock? Is it sharded?

~~~
misiti3780
It sounds like the new system is a cluster with a single collection. So all
writes to the other collections no longer occur, therefore the write locks
decreased

~~~
ddorian43
thanks,

to everyone use hnotify!

------
jyz
I thought global lock is gone in v2.2 [https://blog.serverdensity.com/goodbye-
global-lock-mongodb-2...](https://blog.serverdensity.com/goodbye-global-lock-
mongodb-2-0-vs-2-2/)

~~~
esmet
It's at the database level now.

------
garbados
tl;dr your system is more complicated than you realize, so scaling it will be
more complicated than you think. Also, MongoDB's write lock is a pain in the
ass.

------
peedy
two writes don't make it wrong?

