
Cache Reheating - Not to be Ignored - LiveTheDream
http://blog.mongodb.org/post/10407828262/cache-reheating-not-to-be-ignored
======
antirez
"A very nice attribute of the MongoDB storage engine is its use of memory-
mapped files. In this model the cache is the operating system’s file system
cache. Restart the mongod process, and there is no reheat issue at all."

This is not correct, even if the operating system handles the cache it needs
to reheat the pages that are used in order to cache the memory mapped file.

So usually for instance you have a big MongoDB database but the application
accesses the database in a very biased way, there is an obvious working set.
Part of the working set ends being cached on memory pages by the operating
system virtual memory system, so many times when you do a database read
operation you read reading from memory.

If the server is rebooted and the MongoDB process restarted the file is not
cached in memory, so the first access to every different "page" of the file
will actually seek and read from the disk.

~~~
mathias_10gen
That line was referring to restarting just the mongod process on a running
machine (eg, when upgrading or changing config flags). In that case, the data
should still be in the OS file cache and will result in minor rather than
major pagefaults when mongod accesses data so it will not go to disk. The case
where you do a full reboot is mentioned two sentences later: "Of course on a
full server reboot MongoDB must reheat too."

~~~
antirez
Sorry I missed that sentence where you mentioned it was just if you restart
the server.

And I agree it is an advantage that at least in this scenario no reboot is
needed, for instance if you plan to upgrnde MongoDB version. Actually when you
can plan the downtime it is usually simpler to handle in the general case of
many cache hosts running at the same time when you can do the upgrade
incrementally, but it is nice that a single big server can be upgraded without
the problem of the cache reheating.

Apologizes again for not reading the article with more care.

------
andrewvc
One good reason to use Redis with snapshotting to disk instead of Memcached.

Even if ALL your cache servers go down, you'll at least have the reheat load
distributed among a cluster of cache servers in a super-efficient direct
memory dump rather than hitting your DB servers over the network.

------
PaulHoule
Maybe I'm working with bigger databases than I used to, but lately cache
reheating has become a bigger issue, particularly with mongo.

I'll build a big db and then run ad hoc queries interactively and be like...
this is pretty cool. Then I'll reboot, start the query, go get coffee come
back and see that it's still running.

~~~
0x12
Why the reboot?

------
keithnoizu

      On a similar vein, I like to partially randomize my cache invalidation period if I don't have the luxury of using some other cache invalidation scheme(http://the-robot-lives.com/index.php/2010/03/fractional-keys/).  These prevents cyclic cache invalidation which can lead to interesting issues such as having your server fail every 24 hours after you initial cache restart when a huge some of 24 hour cached items invalidate and start hitting the database at the same time, or whenever the hourly, daily and short duration cache items all invalidate at the same time.

------
chrismealy
Does this mean mongo might be better than something like redis as a memcache
replacement?

~~~
fleitz
If you're suffering from cache reheating issues that only stem from restarting
the process then maybe. If the reheat issues stem from a reboot then you're
still screwed.

The numbers presented only make sense in the event that your cache hit rate is
uniformly distributed.

In the event that you have a normal distribution or pareto distribution (more
likely for a website) the majority of your cache hits come from very few
entries meaning your cache can probably be reheated to 80% with 20% of the
data, or 200GB instead of 1 TB.

A much better idea that actually solves the problem instead of discussing
implementation artifacts (memory mapped files) is to reuse your tracking data
(eg. this page gets X number of hits) to pre-heat the caches before the server
goes live.

All you need to do is build a histogram of your cache hits over the last X
minutes and flush it to disk every now and then.

------
dramaticus3
"Not to be Ignored" but we ignore it

