
Sub-10 ms Latency in Java: Concurrent GC with Green Threads - haxen
https://jet-start.sh/blog/2020/08/05/gc-tuning-for-jet
======
jakewins
For years of my life, all I thought about was stuff like this. If you've ever
ran latency sensitive systems on the JVM.. man is it ever a pain.

Who was it that turned GC off entirely, minimized allocation and just
restarted their VMs when they ran out of RAM every couple of hours, was that
Netflix?

Either way. It makes me excited for Rust and the languages it'll inspire, all
this labor gone away.

~~~
dan-robertson
The part of GC which causes the most latency issues is compaction rather than
merely collection. Using a language like rust won’t help if you have memory
fragmentation and indeed allocation tends to be much faster with a GC than
with malloc. I think the advantages of rust are more to do with often avoiding
heap allocation entirely (and predictably) or value semantics leading to fewer
small allocations or the language’s semantics not forcing you to choose
between more reliable allocation-heavy immutable-everything code and faster
harder-to-think-about mutation-heavy code.

~~~
darksaints
Don't forget scanning. Yes, moving blocks of memory around is expensive, but
it can also be done concurrently. Scanning, AFAIK, cannot be done
concurrently, and thus remains the primary blocker to lower latency. And
scanning is something that is entirely eliminated with static memory
management.

~~~
pron
Scanning is most certainly done concurrently with ZGC. Even root scanning is
on its way to become fully concurrent, which is why we're nearing the goal of
<1ms latency.

~~~
jerrinot
I didn't know about fully concurrent root scanning, thank you!

How does root scanning work wrt to Loom? Are stacks of virtual threads treated
as roots? I guess there is no other option?

~~~
pron
No, virtual thread stacks are not roots! This is one of the main design
highlights of the current Loom implementation. In fact, at least currently,
the VM doesn't maintain any list of virtual threads at all. They are just Java
objects, but the GC does treat them specially.

~~~
kjeetgill
Interesting! In what way are they handled specially?

~~~
pron
Unlike any other object, the location of references on a stack can change
dynamically, so the GC needs to recognise those objects and walk their
references differently. There are other subtleties, too.

------
MaxBarraclough
Doesn't seem helpful to use the term _green threads_ here. This isn't a JVM
with green threads (those are a thing of the past as I understand it). They're
using plain old OpenJDK, and they're ensuring the GC gets a CPU core to
itself.

Neat that they were able to get a dramatic improvement in GC latencies on both
G1 and ZGC.

No mention of the Shenandoah GC. Would the same trick help out there too?

~~~
haxen
We did measure on Shenandoah as well, it helped but not enough to be within 10
ms. Since this post is about Hazelcast Jet getting the best latency, we didn't
report that.

~~~
MaxBarraclough
Interesting, thanks.

------
jjav
Even on ten year old hardware single digit ms latency in Java server apps
wasn't very special. Java (JVM) is an extremely performant platform so I
always find odd how a meme has somehow built up on it being otherwise.

True that one can end up writing terribly inefficient Java code, but one can
write terrible code in any language. If I need to write server code where
performance is particularly important and I don't want to deal with the cost
(in debug time and dev expertise) of C or C++, Java would be my first choice.

Also I'm of the school of thought that performance always matters. Autoscaling
in cloud providers sure makes it easy to scale horizontally to make up for
slow server code, but once you reach certain size, go have a chat with the
finance team about the AWS bill.

------
ryanthedev
Can we talk about no async/await support? Can't build scalable apps when I'm
in callback hell. It's like ES5 all over again.

~~~
kasperni
Java is getting virtual threads/fibers instead of async/await.

From [https://cr.openjdk.java.net/~rpressler/loom/Loom-
Proposal.ht...](https://cr.openjdk.java.net/~rpressler/loom/Loom-
Proposal.html)

\-------------

An alternative solution to that of fibers to concurrency's simplicity vs.
performance issue is known as async/await, and has been adopted by C# and
Node.js, and will likely be adopted by standard JavaScript. Continuations and
fibers dominate async/await in the sense that async/await is easily
implemented with continuations (in fact, it can be implemented with a weak
form of delimited continuations known as stackless continuations, that don't
capture an entire call-stack but only the local context of a single
subroutine), but not vice-versa.

While implementing async/await is easier than full-blown continuations and
fibers, that solution falls far too short of addressing the problem. While
async/await makes code simpler and gives it the appearance of normal,
sequential code, like asynchronous code it still requires significant changes
to existing code, explicit support in libraries, and does not interoperate
well with synchronous code. In other words, it does not solve what's known as
the "colored function" problem.

~~~
ryanthedev
This is some sauce. Ty.

------
hansdieter1337
I want to see Java on mission critical computers in space!

~~~
pjmlp
Not on space, but still mission critical,

"Aegis Battleship Weapons System"

[http://www.artist-
embedded.org/docs/Events/2011/JTRES/Slides...](http://www.artist-
embedded.org/docs/Events/2011/JTRES/Slides/JTRES-Winkler.pdf)

"French radar system for ballistic missile tracking and measurement"

[https://www.militaryaerospace.com/defense-
executive/article/...](https://www.militaryaerospace.com/defense-
executive/article/16720700/thales-chooses-aonix-perc-virtual-machine-software-
for-ballistic-missile-radar)

"NASA Ground Control Station for Multiple UAVs Flight Simulation"

[https://www.semanticscholar.org/paper/Ground-Control-
Station...](https://www.semanticscholar.org/paper/Ground-Control-Station-for-
Multiple-UAVs-Flight-Angonese-Rosa/12e4a305fd7251803c743f1ad96718c72d198165)

------
ginko
I would certainly hope that green threads have less than 10 megaseconds
latency.

~~~
hnarn
Was the title changed? Because it says "ms" both here and on jet-start.sh and
that unit is milliseconds.

~~~
ginko
It used to be 'Ms'. Guess it's fixed now.

------
tupac_speedrap
Glad to see some love for legacy programming languages like Java.

~~~
pron
Well, that's understandable because there are still some legacy companies and
organisations left that write a lot of new software in Java, like Apple,
Amazon, Netflix, Twitter, Google, Alibaba, Tencent, NASA, GitHub, Microsoft,
Facebook, Spotify and nearly all Fortune 500 companies. Plus, if you care
about both performance and observability, there aren't many viable
alternatives.

BTW, many if not most of the cutting-edge advances in compilation, low-
overhead deep profiling, and garbage collection are done on the Java platform,
so it's still the technology leader in those areas.

