
“MongoDB is dead. Long live Postgresql” - lest
https://github.com/errbit/errbit/issues/614
======
functional_test
Seriously, another case of using Mongo incorrectly? I want to believe all the
Mongo hate, but I can't because I always find out that the actual problem was
one or more of:

* didn't read the manual

* poor schema

* didn't maintain the database (compactions, etc.)

In this case, they hit several:

" Its volume on disk is growing 3-4 times faster than the real volume of data
it store;"

They should be doing compactions and are not. Using PostgreSQL does not avoid
administration; it simply changes the administration to be done.

"it eats up all the memory without the possibility to limit this"

That's the idea -- that memory isn't actually used though; it's just memory
mapping the file. It will swap out for something else that needs the space
unless you are actively using all the data, in which case you really are using
all your memory. Which is why you should put it on its own server...

"it begins to slow down the application because of frequent disk access"

"Finally we sleep quietly, and don’t fear that mongodb will drive out redis to
swap once again."

You should be running Mongo on a server by itself. At the very least, if
you're having disk contention issues, don't run it on the same server as your
other database.

I'm not sure you always need to read the manual for everything, but for your
production database, it's probably worth it.

~~~
rlpb
> Seriously, another case of using Mongo incorrectly?

If a large proportion of MongoDB users are using it incorrectly, then I'd
argue that it is a MongoDB problem, if only a documentation and messaging one.
Clarity on what is and is not an appropriate use should be prominent.

So, what is this proportion?

~~~
derefr
Or, to be even more specific--if there's a Right Way to use a program, that
Right Way should be encoded as defaults you have to override (if you know what
you're doing), and automated actions you have to disable (if you know what
you're doing.)

~~~
threeseed
It has nothing to do with the defaults. It is all about people forgetting that
MongoDB is a document database and not a relational one. I can write apps that
will be 10x faster with MongoDB and 10x faster with PostgreSQL. It's all about
matching your domain model to your database.

~~~
macspoofing
>t is all about people forgetting that MongoDB is a document database and not
a relational one.

It's been a while since I looked into Mongo, but that was a Mongo marketing
problem. They used to (still?) advertised themselves as a RDBMS replacement,
literally.

~~~
raverbashing
Yes, but it's more complex than for example, replacing MySQL with PostgreSQL,
where the basic structure is going to be the same.

You have to think and adapt to convert from a RDBS to MongoDB

------
cullenking
Maybe I am just incredibly lucky, but mongodb has worked fine for
ridewithgps.com - we are sitting at 670gb of data in mongo (actual DB size,
indexes included) and haven't had a problem. Replica sets have been fantastic,
I wish there was another DB out there that did auto-failover as cleanly/easily
as mongo does. We've had a few server crashes of our primary, and aside from
1-2 seconds or so of errors as requests come in before the secondary is
promoted, it's transparent.

With that being said, we are using it to store our JSON geo track data, most
everything else is in a mysql database. As a result we haven't run into
limitations around the storage/query model that some other people might be
experiencing.

Additionally, we have some serious DB servers so haven't felt the pain of
performance when exceeding working memory. 192gb of ram with 8 RAID10 512gb
SSDs probably masks performance issues that other people are feeling.

Final note: I'll probably be walking away from mongo, due to the natural
evolution of our stack. We'll store high fidelity track data as gzipped flat
files of JSON, and a reduced track inside of postgis.

tl;dr - using mongo as a very simple key/value store for data that isn't
updated frequently, which could easily be replaced by flat file storage, is
painless. YMMV with other use cases.

~~~
rbranson
How many reads do you do on that server? 192GB + 8 SSDs is a pretty serious
setup. Just the disks themselves should be able to push 500K+ random IOPS.

~~~
cullenking
It's actually very much over built. I had the budget, so I built for growth.
This setup should easily last us through this year, assuming 4x growth over
last year. I am actually putting together another similarly spec'd machine
with 384gb of ram (why not, it's cheap) because the current secondary is on
15k disks with only 24gb of ram. Still more than enough to handle the load
through this last year, but probably not this coming year, at least without a
bunch more ram.

In regards to actual iops, not sure what this thing can peak at off the top of
my head, but we'll easily be doing 100 queries a second this year, with a
considerable portion of those queries pulling out ~1mb documents.

Playing it conservative, so I am moving towards gzipping those large documents
(never need to access anything but the full data, > 90% of accesses are
directly served to clients that can handle inflating the data). For now they
will stay in mongo, but I am building out an evaluation of using a flat file
structure and just letting nginx pass them out.

~~~
weddpros
"just letting nginx pass them out": proxy cache could work as well... Volume
of data is much less here, but I let nginx cache gridFS stored photos...

------
pilif
The title is a bit misleading. This is basically an announcement of a fork of
Errbit that has Postgres support. Additionally, the fork was announced as an
issue on errbit with no discussion or as an official pull request.

I would not consider this good etiquette. If you fork your project (especially
without discussing the intention first), adding a bug to the original project
isn't a very nice thing to do.

An official pull request would be nicer or, even better, don't bother the
original project, but just announce your fork over other channels.

Even better would be to at least discuss the issue with the original project -
maybe they agree and you can work together.

~~~
spellboots
I disagree, I think that opening an issue on github is a good way to start a
discussion about a feature. Many projects accept feature requests this way and
if anyone did the same for one of my projects, this would be the way I would
prefer them to handle it.

~~~
felideon
The thing is it sounds like he is just promoting his own fork he started 11
months ago, rather than "starting a discussion."

 _> We suggest to put errbit on PG. For those who want to try - the code here:
[https://github.com/Undev/errbit/tree/pg-
upstream](https://github.com/Undev/errbit/tree/pg-upstream) _

The problem here is that the bad English grammar could have given the wrong
impression. Maybe he is just saying:

"hey guys, you should consider migrating to postgresql. here's some code you
can check out that has worked for us."

Rather than:

"hey guys, screw Errbit/MongoDB, use our fork!"

~~~
calinet6
... after thouroughly testing it in production for 11 months and verifying
that they had a point.

This is perfectly valid. A pull req might have been better, but this starts a
discussion as well, and might prompt the project owner to say "Sounds great!
Submit a pull request," or alternatively, "Sounds cool! We'll provide a link
to your fork on our page." Both good.

------
memracom
Lets just say that PostgreSQL answers the criticisms of relational databases
that led to NoSQL. The complaints all boiled down to saying that the RDBMS
forced you to do things one way and that it was cumbersome. PostgreSQL evolved
and fixed the most annoying issues like JSON support and schemaless key-value
store support. That's the way open source is supposed to work. Now folks are
learning that throwing out the baby with the bathwater leads to more
complexity than just learning how to use a relational database. The pendulum
has swung back.

~~~
yid
> PostgreSQL evolved and fixed the most annoying issues like JSON support and
> schemaless key-value store support.

As I recall, automatic sharding was on that list, and pg doesn't attempt to
tackle that afaik.

~~~
lucian1900
Mongo doesn't really do that in a way you can reliably use in practice either,
though. Its sharding offers a subset of operations over an inconsistent view
of your dbs.

You can do that with Postgres trivially, and even automatically with
postgres_fdw and writeable views.

------
mml
The hstore enhancements coming in psql 9.4 will pretty much put mongo out to
pasture.

"Mongodb" already nearly exists as a single column type, 9.4 will complete it.

~~~
threeseed
Right. Just like MySQL/Oracle was put out to pasture.

And if you think MongoDB is only popular because it is a JSON store then it
shows just little you know about the database landscape and about how
developers actually use databases.

~~~
mml
For those of us that use all 3 of those, your statement is in large part true.
I'd also hazard that Mongodb isn't actually that popular outside the HN buzz
bubble.

------
pilif
All the philosophical issues and /(No)?SQL/ discussions aside, as a heavy user
of Postgres and a user of Errbit, this is very good news to me. I have not
much experience with running Mongo, but I have a ton of experience with
running Postgres.

Even better: The application I'm using Errbit the most for is already running
in front of a nicely replicated and immensely powerful postgres install.

Being able to put the Errbit data there is amazing.

This is some of the best news I've read today :-)

------
jvvlimme
If you want to use MongoDB in a project and you don't intend to rely heavily
on the aggregation framework, the consider TokuMX
([http://www.tokutek.com/products/tokumx-for-
mongodb/](http://www.tokutek.com/products/tokumx-for-mongodb/)) as it
alleviates many of the shortcomings of MongoDB (data compression, document
level locking for writes, ...) + it adds transactions.

It's a drop in replacement so it will work with current drivers. (if you have
a running mongo cluster however expect quite some work if you want to migrate)

(I have no affiliation with TokuTek whatsoever except that I use their
product)

~~~
stonewhite
It is basic an ops-friendly mongo fork with _obviously_ better engineering
decisions. I hope mongodb will support pluggable storage engines soon.

------
endijs
I'm no MongoDB expert, but recently started to look into this db. Can anyone
tell me (from experience, not from promo materials) - for which use cases
MongoDB is good fit and for which ones it's not? It's clear that it can't fit
for everyone. That's why it would be good to know in advance, for what it most
likely to find and for what it's most likely not to fit.

~~~
mikegioia
It has the benefits and ease of use of a json document store, it allows you to
do SQL style where clauses, it takes about a minute to install and start
using, there are a wide range of drivers available for many languages, and it
has a simple javascript map/reduce.

on the flip side, it implements database level locking, uses more disk/RAM
than it probably should, and can start to give you headaches if you try to do
a lot of writes at once.

edit: to give you a real world example, we use mariadb for storing everything
persistently. however, a lot of data like "number of teachers in school A" is
aggregated and too difficult to run in real time when we render paged results.
to get around that, we use mongo as a document store and use its SQL like
querying to generate the paged search results. this lets us sort/filter on the
data without having to do everything in SQL.

~~~
jeltz
> to give you a real world example, we use mariadb for storing everything
> persistently. however, a lot of data like "number of teachers in school A"
> is aggregated and too difficult to run in real time when we render paged
> results. to get around that, we use mongo as a document store and use its
> SQL like querying to generate the paged search results. this lets us
> sort/filter on the data without having to do everything in SQL.

This use case should be possible to solve with the JSON type in PostgreSQL.
The indexing in PostgreSQL is just as advanced in 9.3 and will be better than
MongoDB in 9.4 if a couple of patches land.

~~~
mikegioia
It definitely is possible with PostgreSQL and it's honestly probably better to
do with Postgre, but we're using MariaDB for relational data and it was just
quicker/easier to dump everything to mongodb.

------
kldavenport
Their use case didn't seem especially Mongo-centric, I wonder why they chose
to go down the road. We used MongoDB TokuMX to improve performance:
[http://www.tokutek.com/resources/benchmark-results/tokumx-
be...](http://www.tokutek.com/resources/benchmark-results/tokumx-benchmark-
hdd/)

------
egeozcan
I'm nearly sure one day someone will write a MongoDB compatibility layer on
top of PostgreSQL

~~~
seiji
PostgreSQL has native json support now. What else is missing? Just a protocol
implementation?

I'd love to see MongoDB give up and become a PostgreSQL consultancy.

Everybody I talk to in the field has the _exact same_ Mongo story: "We love
JSON! We use JSON everywhere! We just wanted a DB with native JSON support. We
didn't look at the implementation details. We only looked at their marketing.
Now we wake up at 3am to fix it every night and lose data every day. Somebody
help us. We love JSON."

~~~
jcampbell1
It is probably not as simple as "supports json now". Imagine if HN comments
were stored as a JSON document:

    
    
        Client A: Read JSON.
        Client B: Read JSON.
        Client A: Append new comment to json document.
        Client B: Append new comment to json document.
        Client A: Save JSON
        Client B: Save JSON
    

A's comment will get deleted. My understanding is that Mongo DB does have a
way to append a record within a document, but Postgres does not.

I am in no way advocating for MongoDB (I dislike it). I am just saying that I
understand that MongoDB has much more sophisticated updates capability than
Postgres.

~~~
viraptor
In that case you can (should) just "SELECT ... FOR UPDATE" in your
transaction. This should prevent the issue. Client B will wait with the read
until client A commits.

~~~
AlisdairO
Or you can use serialisable transaction isolation and retry on conflict.

~~~
jeltz
For the simple cases you do not even need that.

~~~
AlisdairO
Indeed. Depends on whether you need to do a read-update cycle or you can just
do it in a single SQL command.

------
weixiyen
> Finally we sleep quietly, and don’t fear that mongodb will drive out redis
> to swap once again.

Well duh, Mongo was designed to live on its own server as it tries to claim
all of the free memory available. Putting it on the same server with Redis
makes no sense.

The case that caused you sleepless nights does not apply to 99% of projects
out there.

------
WoodenChair
"Nobody ever got fired for buying IBM." Nobody ever got fired for using
PostgreSQL.

------
r0muald
In case you missed it, this submission is not about PostgreSQL vs MongoDB.
It's about the crazy GIF parade in the comments interleaved with thumbs up
emojis. You don't see such stuff often on github :)

------
trekky1700
This just makes me wonder why they chose Mongo in the first place. It sounds
like they didn't really consider their needs when initially choosing
databases. Mongo has some benefits that when properly implemented far outweigh
the negatives. At the same time, it's still relatively young, and doesn't have
the "maturity of process" that makes older SQL engines so easy to
manage/implement. Eventually, I'm sure, Mongo will solve these issues and be a
great database for those who need to utilize its many virtues.

~~~
jmngomes
Because 1) a lot of startups seem to choose "startup technology", i.e.
whatever famous startups are using, just because it seems fashionable and/or
they don't consider if it'll actually solve their specific problem; or 2)
they're technically curious and end up using it just for fun, even if it's not
a good fit for their problem.

I've seen people using Redis for their MVPs, which is hardly necessary to
serve 100 or 1000 or 10000 users. When you have a hammer at hand, everything
looks like a nail.

~~~
aidenn0
As far as Redis goes, is there really much of anything in the space between
bdb style KV stores and Redis? If you have design reasons for wanting your KV
store in a separate process, why not use Redis?

~~~
jmngomes
I agree with your point, my example was about using Redis for performance
reasons when designing/building an MVP, a stage where you are unlikely to have
a (real) performance problem.

------
jeffdavis
"Its volume on disk is growing 3-4 times faster than the real volume of data
it store[sic]"

Are they saying that it has a high constant overhead to the data, or are they
saying the storage grows in a super-linear fashion?

~~~
remon
It's constant overhead with some spikiness to it. See my reply in this thread
for details.

------
poseid
Anyone compared MongoDB with other document stores, e.g. with
[https://github.com/triAGENS/ArangoDB](https://github.com/triAGENS/ArangoDB) ?

------
iand
Typo in title.

------
WalterSear
Does anyone have a recommendation for an authoritative guide to either
Postgres or Mongodb? One that does more than show you where the levers are,
that is.

------
coolrhymes
RDS now supports Postgres. It supports both hot & cold swaps. Hopefully in
future it will support read replicas.

------
filipedeschamps
This is what I call a click bait title.

------
1945
This title is why I've been reading Hacker News less and less.

------
mtolan
Is PostgreSQL web scale? I will use it if it is web scale.

