
Memcached – Caching Beyond RAM: Riding the Cliff - Rafuino
https://memcached.org/blog/nvm-multidisk/
======
gstaro
The way I understand this proposal, the idea is to find at which request rate
the latency of a storage configuration (eg, 2x Optane) explodes. If your
request rate is above that rate, get more RAM. I'm not sure I understand how
to figure out how much more RAM? Until the latency goes down I guess?

Also what about diurnal patterns? Can there be a way to dynamically react and
save power..

Interesting reference to the morning paper. But don't get how that fits into
the story?

[https://blog.acolyer.org/2018/10/26/robinhood-tail-
latency-a...](https://blog.acolyer.org/2018/10/26/robinhood-tail-latency-
aware-caching-dynamic-reallocation-from-cache-rich-to-cache-poor/)

~~~
dormando
1) Super workload dependent unfortunately. Probably "easiest" way is to set up
test systems with enough RAM and disk space. Then either slowly reduce the RAM
or start from a low point and slowly increase until you hit your target;
there's a command for memcached to live-adjust the RAM limit.

2) dunno! For RAM that's super hard. for disk I'm hoping tiering will do
something. ie; for on-peak you can add extra NBD space then remove it off-
peak.

3) referenced tail latency because most of the testing focuses on displaying
latency outliers and how the system generally minimizes them.

~~~
peterwwillis
I thought the RAM estimation was (1) benchmark your backend, (2) benchmark
your memcache<->backend, (3) measure peak traffic performance requirements,
(4) provide enough RAM that you reduce the latency to acceptable levels during
peak traffic. If you don't have measurements of the first three you're
shooting from the hip. It is workload dependent, but the more measurements you
have, the better you can estimate. (Used to be mandatory so you could do trend
analysis and purchase your new gear in time for inevitably outgrowing the old
stuff)

~~~
dormando
Ah my point of reference was more "how do you go from memcached RAM to RAM +
disk?" not "how do you go from no memcached to memcached".

If you don't have any cache layer at all, that's a standard approach.

~~~
peterwwillis
Wouldn't the RAM + disk just be an extension of that method, though? It's like
in the article, you test a subset of the system (RAM+disk) to find its limits,
then you compare that to your other measurements and load requirements, and
replace RAM with RAM+disk where it's cost and performance appropriate. But I
might have missed part of the analysis

~~~
dormando
The article was significantly cut down for brevity, so any detail of how to
find the right balance is beyond the scope :)

The original question was that it can be really hard to determine how much
adding RAM will offload from the disk. IE: with extstore recent objects are
served by RAM and never disk, so they don't count against your IO balance. If
you're coming from a RAM backed system that has 500k requests per second, it's
going to take some creative introspection to figure out how much RAM you'd
need to keep the disk accesses below 400k, or whatever your target is.

Along those lines I just suggested a simple experiment. which is to just take
an existing fully specced instance, add a disk, and test a few RAM settings to
see how it looks. Most actual users of extstore have just done that since
they're extending/replacing an existing cluster.

There're actually other ways you can simply read the numbers out of memcached
but it just takes more effort or familiarity with its internals.

------
baybal2
Bad hardware/badly configured hardware can add more latency than software even
if it runs in the same rack.

RDMA NIC + nvme got 90%+ latency reduction over noname NIC + sata ssd for
Alibaba in a large DC environment.

~~~
Rafuino
Do you have something more detailed I could read about that?

------
ksec
Not trolling but a real Questions.

Under what scenario would Memcached be better than Redis in 2019 if you are
starting new?

~~~
scarface74
Memcached scales vertically. Redis is single threaded. Multiple cores are
wasted.

~~~
indeyets
An official position is that one should run multiple Redis servers per machine
and scale by splitting datasets between them

~~~
scarface74
Which doesn’t necessarily help if you have certain shards that are hit heavier
and it’s more maintenance.

Also, if you are running on prem, you can optimize your VM for cores/RAM. At
least on AWS, you can’t perfectly optimize your core count versus RAM vs
speed.

But then again, if I were running on prem, I might use Redis because it
handles use cases by itself that it would take Memcached+ other servers to
manage. With cloud hosting, I could use managed purpose built services if I
needed more features than Memcached can handle by itself.

------
baybal2
Latency matter when you have a lot of roundtrips -- ie databases

