

Most popular JVM memory configurations - ivom2gi
http://plumbr.eu/blog/most-popular-memory-configurations

======
millerm
Umm. I trying to figure out how that title comes from what I just read in that
article. The article shows a couple graphs comparing the max heap size and the
max PermGen size of ~1000 different installations of java application servers.
So? How can you state that java doesn't need that much memory after all? After
all what? I know we've run out of PermGen with the JVM set to anything less
than 192mb. We've got a pretty large app that (unfortunately) depends on a
load of third party libraries. Too many libraries for my taste but that isn't
my call. So, we must have this set or the server won't even start.

All applications are different and have different memory requirements. Too low
of a setting and you are constantly thrashing or running out of memory. Too
large and you have huge garbage collection times. It has to be tuned. Now,
I've heard (as I haven't really looked into it yet) that the JVM in the JDK 8
distribution has gotten rid of PermGen setting. So, I'm curious to know what
'they' are doing now. Is the G1 garbage collector the new default in JDK 8?
Does that not have the PermGen or is this something different?

Anyway, I don't know what this post is about yet. They probably should have
just finished the thing instead of saying: "If you managed to get this far,
then you are most likely interested to hear about our forthcoming posts. Stay
tuned by subscribing to our Twitter feed." It was worthless.

~~~
ivom2gi
Apologies for the confusion. The aim was to demonstrate that against our
expectations the typical heaps in JVM landscape are not set that high and ~25%
of the apps are doing just fine with less than 256MB of memory allocated.

And another aspect we wished to demonstrate was the sheer size of permanent
generation on some of the applications. I have yet to face a leak-free app
where permgen > 256MB seems justified ...

~~~
millerm
So, basically your data could just be showing that 25% of your install base
have very small codebases. One of my applications, as an example, has 52MB of
jar files. That's compressed data. Now, I am not sure how that eventually
equates to how much ram is used to load these files (and I know that not every
class will be loaded), but unzipping these jars ends up with over 203 megs of
data. I have to have a large PermGen.

Perhaps the tool you install should be showing the number of classes loaded
and the size of the code in general. But, again what's in a jar file can't
tell you how many classes will be loaded because many common API such as
hibernate and spring generate thousands and thousands of dynamic proxies that
end up being stored in memory for the lifetime of that application.

The data also doesn't show if the higher heap users tested out running a lower
PermGen/Heap and ran out of memory, hence their increased size. I'd rather
allocate too much memory though than run out of memory at a peak time and have
one of my hospitals deal with downtime because I saved $25 worth of RAM.

Anyway, I'll read the followup but I hope it has more info. The key is you
have to profile your application, tune it and your done.

------
cagenut
I have the opposite problem, I can't get java to use "enough" memory. Every
time we double it in production that leads to another few weeks of re-
jiggering jvm flags and ratios and grepping&graphing the garbage collection
log or staring at visualgc for days.

We're thinking of giving up on scaling above 16gb/jvm and going to something
crazy like 16x2gb jvms. Treat them like the worlds biggest mongrel and let the
loadbalancer sort it out.

~~~
joshhart
Hi cagenut, I'm a dev @ LinkedIn and I have regularly pushed services close to
30gb of heap, which serve close to 10k qps. After that it does get wonky, but
I'm surprised you're having trouble getting it that far.

Mind emailing me your config? You should be using a CMS collector.
-XX:+UseConcMarkSweepGC. Also try reducing CMSInitiatingOccupancyFraction
until things stabilize. I've never had to drop it below 70%. Finally, ALWAYS
make sure Xmx and Xms have the same value so the JVM doesn't try to resize
itself.

~~~
cagenut
Hi Josh, Thanks for offering to help. Your colleague Jacob Kessler wrote the
closest thing I have to a bible on this:
[http://engineering.linkedin.com/26/tuning-java-garbage-
colle...](http://engineering.linkedin.com/26/tuning-java-garbage-collection-
web-services)

That (and a truckload of trial & error) led me to this config:
[http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303s...](http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303shbmq)

You're probably right, I probably could just keep dialing up xms/xmx/xmn, but
even then what exactly to watch for in the gc log and how to do it tools-wise
is a dark art. Exactly what field in what line to grep for and be mindful of
when is a constant source of debate/confusion, and once you get up to a few
hundred req/s visualgc just can't keep up so the graphs stop being useful.

As you can see from my config link there I've tried throwing money at the
problem but that didn't do it either. I'm at the point where I can hire an
engineer to truly master this, or go bootleg parallel.

~~~
joshhart
Jacob Kessler knows quite a bit about garbage collection :)

Understanding the gc logs can be difficult. gcviewer is pretty good though at
figuring it out. Always remember that garbage collection is slowest whenever
the JVM has to copy objects from one generation to another or between survivor
spaces. The logs print the sizes of the young gen, old gen, and survivor
spaces. Sometimes after a collection you can see the sizes of these change
dramatically - that means a big object has moved around. You can either change
your code to use big objects less often or hold onto them for a shorter time
or try to increase your survivor space size or new gen size.

A major, major, major thing you are missing is -XX:+UseParNewGC. With the CMS
collector, your new gen collection can be done in parallel, and the
parallelization scales very well with the number of cores. Without ParNewGC,
the service I work on would take over 1 second to garbage collect. With
ParNewGC, it only takes 50ms.

~~~
cagenut
According to my notes from when I first put that together -XX:+UseParNewGC is
automatically implied by +UseConcMarkSweepGC, for which my source was this
very old web page about a very old jvm version (again, dark art):
[http://cybergav.in/2010/01/11/tuning-hotspot-
jvm-1-4-2-with-...](http://cybergav.in/2010/01/11/tuning-hotspot-
jvm-1-4-2-with-cms-low-pause-garbage-collector/)

Also, more concretely, this is the most frequent type of entry in my gc log:
[http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajb...](http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajbp2na)

I (perhaps wrongly?) interpret the ParNew there to imply that's the young gen
collector in use.

------
philippelh
Funny, isn't it?

"...whether it is a 32 or 62-bit infrastructure..."

and a bit later...

"And those two who think 258MB is better – are you just bad at calculating the
powers of two..."

~~~
ivom2gi
Congrats! You were the first one to spot the easter egg we planted, if you
contact us referring to your original comment you can have the Plumbr for free
for one year.

------
cinbun8
The post compares the Xmx setting of all VMs that use the free Plumbr. There
is no way to know if these settings are used in production or someone just
downloaded Plumbr for an evaluation. Most devs do not even bother to check how
much memory their tomcat instances use, which would explain the size of the
memory. They only ever tweak it when they run a load test or when the server
runs out of memory.

These metrics do not really reflect the heap sizes used in production, so I
would not conclude that java does not need much memory. If anything, those
that use Plumbr in evaluation mode do not run their JVMs with large heaps. You
do not need a large heap to cause a OOM.

Still, it is interesting to know what the Xmx settings are on these boxes.

~~~
reeses
There's also a selection bias in effect. Plumbr is not _yet_ a household name.
Speculating wildly, I suspect that they don't have enough clients using it at
scales that would provide meaningful statistical input.

Many Java devs, when looking for object/memory/resource leakage, will turn to
the well-known general purpose tools, such as JProbe, YourKit, Introscope, or
even just JVM profiling output and verbose gc. Crazy people will start using
DTrace or JMX hooks. The rest will just say they need a hardware refresh.

As you say, this is an analysis of metrics gathered from people looking for a
'free' solution. The reason for the 'small' mx settings could be that
developers are finding ways around procurement to find leaks on the same
machine running their IDE, email, browser, etc.

------
bjoernbu
The HN headline is somewhat misleading. It is different from the original
article and I was expecting some neat techniques to overcome the typical
memory issues:

Everything is an object, generics need wrapper classes, padding everywhere.
Those make HashMap<Integer>, ArrayList<Integer> or even "two dimensional
arrays" a bad joke compared to C(++) equivalents. Projects like Lucene have to
rewrite standard library eqiuvalents for specific cases since otherwise so
much performance would be lost.

Examinating app server -Xmx settings is more handwaving than anything.
Especially people that refrain from doing certain things in java due to memory
issues, have a completely misleading influence on such a metric.

------
sil3ntmac
Sorry for my naivety, but when does Java use the stack? Only for primitive
operations?

~~~
pkolaczk
If you turn Escape Analysis on, it allocates objects not escaping the local
method scope on the stack.

------
_pmf_
It would also not hurt to mention the topic of virtual memory vs. "memory
usage issue" here.

------
martinced
PermGen, as if all the JVMs out there were Sun/Oracle or the OpenJDK ones ; )

~~~
pjmlp
I still don't understand how developers fail to be aware of the amount of JVMs
and even native compilers available for Java out in the wild.

------
BonoboBoner
said no one ever?

