I'll note that I'm excluding API calls here. I don't commonly hear API calls referred to as 'resources' on a page, so I'm assuming that this wasn't being referred to.
So while the probabilities do suggest that most users will encounter at least one 99th percentile latency, it's very different if that latency is for a static resource vs a dynamic one. Thanks for pointing it out.
I really wanted to comment "just always make a blog post in reaction, twitter is not a good medium for explaining complex ideas" on that thread, but thought it would be unproductive.
Please just do this every time, people who care will read, people who don't care will probably not comment for fear of looking stupid.
Redis uses fork in a way nobody else seems to. Applications using tens of gigabytes of RAM -- databases, media editing, etc. -- just don't usually fork except during a startup process.
Redis uses fork for persistence. It's clever, and it works great in general, but it's weird! Unique, even. It seems much more likely to hit a fork() weakness in a noticeable way than almost any other application out there.
Since this change, there has been absolutely no reason to use anything other than HVM AMIs, and pure paravirtual instances can basically be considered a deprecated feature in EC2. New EC2 instance classes (e.g. t2) don't even support PV.
Basically, PV instances are just there to support current customers who are relying on already-built PV AMIs and have too much inertial to be nudged into switching over.
Linux beefy VM on VMware 6.0GB: 12.8 milliseconds per GB.
Linux running on physical machine (Unknown HW): 13.1 milliseconds per GB.
Linux running on physical machine (Xeon @ 2.27Ghz): 9 milliseconds per GB.
Linux VM on 6sync (KVM): 23.3 millisecond per GB.
Linux VM on EC2 (Xen): 239.3 milliseconds per GB.
Linux VM on Linode (Xen): 424 milliseconds per GB.
Around 30 times slower than bare metal, and I'm talking about old physical servers with slow memory compared to today's.
EDIT: Make sure to read this -> https://redislabs.com/blog/testing-fork-time-on-awsxen-infra...
There also a similar problem for master of MFS, it will be great if you have some experience in it.
suggests that it's page table validation time, and it can be avoided if you use a PVHVM guest. Has anyone checked whether these redis problems apply only to PV guest images on EC2?
When you crank up the density per node to 256 or 512 gigabytes even bare metal is problematic and in some domains like telecommunications they don't care that the spikes are concentrated because they cause cascading failures.
I think a userspace COW implementation in Redis would be a big project because you would need a different strategy for every data structure. Being single threaded also makes it challenging to get other/more cores to do the serialization and IO. It's very doable just not within the current philosophy of thou shalt have one thread per process.
The entire idea of getting a point-in-time snapshot of something subject to rapid change is problematic. Your options boil down to (1) make a "snapshot" and save that (Redis's current approach) or (2) accept that point-in-time consistency might be impossible, and work around it.
I wonder whether it'd be possible to have two persistence strategies in Redis: "consistent" and "low-latency". "Consistent" would use the current fork(2) COW behavior, "low-latency" would do some kind of one-chunk-at-a-time block copy and amortize the latency spike over the entire operation, while having the overall effect of less of a "cliff" to latency.
Should probably add a disclosure here. I ran the EC2 instance product management team for several years and continue to be involved.
It does mean that forking the entire process is pretty much the point of the exercise however.
Btw, I am not sure one can call "reactive architecture" if using a single thread.