Just as a quick comment the GC configuration on those Cassandra nodes is kinda crazy. I don't think anyone is recommending anyone to run 30G maxheap CMS. Somewhere around 16G you really should switch to G1GC.
Also I highly doubt you want to have 30G heap with 60G RAM available. Having a larger heap most likely doesn't help you at all. It will just eat away memory you could use for mmapping or off heap things and increase GC pause length.
(This is the bane of running JVM any apps. You really need to know what you are doing with the GC. Also with apps like Cassandra there is no silver bullet. Just some general guidelines to follow and slowly adjust your values and monitoring of the changes as the best possible settings change with the load/hardware setup)
It would be nice if some of these were the defaults as opposed to some settings folks have to change - say by running a "./configure" or some such.
In this case, for example, it felt a little weird that the machine type which had enough cores had 60GB RAM, and giving only around 10-20GB to Cassandra "felt" wrong, but turns out the exact opposite is true! YugaByte is in C++ and can handle much larger heaps.
Java's garbage collected heap is not the full extent of memory used by Cassandra. Cassandra can use as much memory as you can throw at it.
File system cache, in process decompressed page cache, key and row cache, off heap memtables etc. There are several different mechanisms for using memory that isn't on the garbage collected heap.
^
This is not a qualitative assessment of how Cassandra uses memory.
G1GC would make wonders of this setup. I guess thisis why it is benchmarked with CMS. I just finished a project where the biggest performance tuning was to move the GC to G1GC.
Also I highly doubt you want to have 30G heap with 60G RAM available. Having a larger heap most likely doesn't help you at all. It will just eat away memory you could use for mmapping or off heap things and increase GC pause length.
(This is the bane of running JVM any apps. You really need to know what you are doing with the GC. Also with apps like Cassandra there is no silver bullet. Just some general guidelines to follow and slowly adjust your values and monitoring of the changes as the best possible settings change with the load/hardware setup)