
Comparing Go and Java, Part 2 – Performance - spahl
http://boundary.com/blog/2012/09/17/comparing-go-and-java-part-2-performance/
======
jamwt
Please, if you're writing req/resp benchmarks, please include 98/99/99.9%
latencies. These are essentially the only numbers that actually matter.

At the mean, at concurrency 50, we have things like 37.5ms vs. 12ms. To a
user, that is "instant" vs. "instant". Sure, if we stack a bunch up on a page,
maybe we'll start to care, but..

Much more directly, in real-world scaling scenarios, it matters far more if
the 98% number is 2s or 248ms than the mean is 37ms. A 2s latency means 2 out
of every 100 requests (and likely > 2% of page views), the user experience
will suck. And, all it takes is 6 requests to ensure that 25% of users
experience this (your homepage alone probably requires 6 requests).

I'm not just raising this to be pedantic--I've seen plenty of systems that
have been tuned to do very well on things like req/s or even mean latency but
do very poorly for 1 in 100 or 1 in 1000 (vs. being a very fair scheduler at
the cost of overall throughput). We reward these systems by measuring and
praising the wrong thing. (e.g. mongodb vs. riak)

edit: (btw, not intended as a specific defense of either go or java wrt the
article, just a general statement about benchmarking these systems)

------
KevinEldon
The article mentioned DropWizard (covered more in part 1). DropWizard is a
very nice (the best?) way to write RESTful web services in Java. Coda Hale,
that author, has glued together some of the best Java libraries with very
sensible configurations, almost to the point of saying "look stupid, this is
how you're supposed to do it (in Java)". The application configuration
patterns (ugh... I said "patterns"), clear separation of resources, built-in
metrics, and health checks is worth studying (at least for journeymen like
me).

Check it out: <http://dropwizard.codahale.com/>

~~~
rudiger
How does this compare to using Spring application development framework (with
Maven, of course)? My understanding is that the Spring framework is how you're
"supposed to do it" in Java, and it certainly is very popular.

<http://www.springsource.org/>

~~~
stickfigure
Spring and Dropwizard are really not comparable. Spring is a large, invasive
framework which covers many different facets of enterprise application design.
Dropwizard is a set of commonly-used libraries (jetty, jersey, jackson, slf4j,
a couple others) with a little bit of glue to stitch them together.

~~~
KevinEldon
Very well stated. You said it in half the words I did. I especially like your
point about Spring being "invasive". For me, Spring MVC was so invasive I
couldn't justify using it... the LDAP, JDBC, and Security components work
usable without too much interference though.

------
shanemhansen
No surprises here. Google did a benchmark on go, java, scala, and c++ that's
worth reading. [http://www.readwriteweb.com/hack/2011/06/cpp-go-java-
scala-p...](http://www.readwriteweb.com/hack/2011/06/cpp-go-java-scala-
performance-benchmark.php)

Here's my personal experience. Simple go code is actually comparable to c,
even for non-io bound tasks. I was very surprised when OpenSSL's AES
implementation and go's AES implementations performed similarly in my
microbenchmarks. The jvm actually performs very well (go figure that over a
decade of optmization in running enterprise workloads would result in a fast
runtime). I've inspected the generated assembly in a go code and compared it
with that from the equivalent c code and there's no doubt go isn't the most
efficient language ever created.

Use go if you want something that's (pretty darn) fast, productive, and has a
standard library written by some of the most respected names in the field.
Don't use go if you need the most mature and performant runtime and libraries.
I have no doubt that they will get there eventually.

~~~
luriel
> No surprises here. Google did a benchmark on go, java, scala, and c++ that's
> worth reading.

No, is not worth reading, is misleading at best and has been throughly
debunked:

<http://blog.golang.org/2011/06/profiling-go-programs.html>

Not to mention it used an ancient version of Go, even Go 1 is dramatically
faster than that, and since Go 1 there have been even more dramatic
performance improvements, but the main issue is that the guy that wrote the
benchmarks really had no idea what he was doing (there were similar criticisms
from outside the Go communities about the quality of the benchmark).

~~~
shanemhansen
Honestly I'm happy to hear that. I'm one of those rare individuals who gets to
write go at work. I'm really happy with go performance and memory usage. Like
really really happy. The different between the cpu usage of a go program using
protocol buffers and a python program using protocol buffers is pretty
dramatic (go is the clear winner).

Most folks would assume a paper coming out of google involving go has some go
experts involved. Clearly this is not the case.

------
mritun
I'd make a guess in absence of code.

Since none of the implementations were IO bound (unsaturated link), I'd bet on
memory management as the deciding factor (it's hard to assume that an
authentication service will be CPU bound).

Then it's not difficult to see why Java won. When it comes to garbage
collectors, JVM has the best (production) implementation of GC bar none.

Go runtime has a lot of catching up to do.

~~~
bdcravens
First paragraph: "In Part 1, we looked at the code of two web services that
implement an authentication web service. One written in Java, and one written
in Go. It’s time to beat up on them a little bit."

<http://boundary.com/blog/2012/09/13/comparing-go-and-java/>

~~~
masklinn
And the code repo: <https://github.com/collinvandyck/go-and-java>

------
reddit_clone
Does anyone else find 4 lines of error checking (log and panic) boiler plate
code for every line of functional code a bit tedious?

~~~
shanemhansen
It's annoying, the only thing it has going for it is it's better (in my
subjective opinion) than the alternative.

In go you can happily ignore an exception by assigning it to _. That's
probably a bad idea for a larger piece of code, but for little scripts, go for
it.

I may be already permanently damaged by go. Every time a function can return
an exception I have to stop and ask myself: "self, what should you do if this
happens?". I'm starting to think it results in better code. Granted, not every
shell script/utility benefits from this level of introspection, but that's
what python's for I guess.

p.s. Anyone who's had to deal with checked exceptions in java puts up with a
crazy about of boiler plate too.

p.p.s. Disclaimer I have been professionally employed writing all languages
mentioned above, so hopefully I'm relatively unbiased.

~~~
stickfigure
I have invented a remarkable new programming tool that wires up to your chair,
keyboard, and 110v AC. It gives you an electric shock every time you complete
a line of code, reminding you to stop and think about it. Think of the
productivity!

Checked exceptions are indeed obnoxious and a major language design failure,
which is why pretty much every modern language since just has plain old (non-
checked) exceptions. And even in Java, you can work around the brain damage by
wrapping checked exceptions with runtime equivalents in API facades.
Exceptions are still incredibly useful, and lack thereof is my biggest
complaint about Go.

I'm also disappointed by the convention of capitalizing/lowercasing names to
export them or not. Realize that you want to export an existing private
method? What's that, your IDE doesn't support refactoring? Get typing, you
have a lot of method calls to update.

~~~
wonderzombie
Exceptions make it harder to reason about your code's execution path. If you
raise an exception, it is often not obvious who in the call stack is
ultimately going to catch and handle it. The logic for that can live pretty
much anywhere. This is not to mention the try/catch/finally pyramids you get
from trying to cope with nested failure cases.

Go uses a well-understood mechanism: return. Control reverts to the caller.
It's simple, which was an explicit design goal of Go.

IMHO, if you have a choice between exceptions or not, they just aren't worth
the value they deliver. As Josh Bloch says, use them for exceptional
circumstances only, to indicate truly exceptional circumstances, such as
catastrophic errors.

Re: refactoring: <http://golang.org/cmd/gofmt/>. Check out the -r option.

In practice, is this really a huge deal? Modulo go fmt, what editor doesn't
support multi-file S&R with regex? Isn't this what a compiler is for? All in
all, this sounds like bikeshedding about syntax. We're all entitled to our
opinions but it's awfully hard to say anything interesting about syntax which
has not already been said a bajillion times.

------
sitharus
That seems reasonable to me - Java's had decades of optimisation, but Go's a
fairly new language and runtime.

I've just started playing around with Go, I'm quite liking it. It's a
different approach to things, which is always fun.

~~~
koenigdavidmj
It seems like it might be a fun project to compile Go to Java bytecode. GCC
might actually have all the relevant bits---it includes gcj, a Java compiler
including bytecode generator, and a Go frontend.

Knowing this, someone's already done it---anyone aware of such a thing?

~~~
strlen
The big pull of go for me is AOT compilation to native code (which translates
to fast start time for command line utilities), close integration with the
OS,support for value types, and stack allocation.

These are not currently available on the JVM (although this may change as of
JDK 8).

~~~
pjmlp
Sure they are, there are more JVM out there than Oracle's you know?

<http://www.atego.com/products/aonix-perc/>

<http://www.excelsior-usa.com/jet.html>

[http://researcher.watson.ibm.com/researcher/view_project.php...](http://researcher.watson.ibm.com/researcher/view_project.php?id=174)

~~~
strlen
I'm aware of real time JVMs, but they seem to target smaller scale mission
critical applications (I'd imagine embedded or Scada?)

None of these are free (or "open source" in the OpenJDK sense -- e.g., where
most of the system with the exception of a few libraries is open), however,
with the exception of the last -- where the first revision was available as
part of Jikes RVM.

The only commercial JVM that I do believe has some production server-side
deployments -- Oracle's JRocket -- still doesn't support value types and has
(in many cases) actually performed worse than HotSpot with large heaps.

So to put it bluntly none of these fit "I'm willing to deploy this to
thousands of nodes in production" criteria. To me HotSpot (due to its
stability, performance, and support for concurrent GC) VM is more of a reason
_to_ use Java/JVM languages -- otherwise I'd much rather use C#, F#, D, Go, or
OCaml.

Finally, I should it's quite possible to do manual memory allocation, bypass
bounds checking, and to control memory layout using direct byte buffers and
Unsafe in HotSpot/OpenJDK. Problem is (and I've mentioned it) is that it
incurs serialization costs from copying objects to these byte arrays. Of
course may this may be acceptable for many kinds of application, especially if
only a _small_ part actually needs this (which I'd imagine describes many --
if not most -- kinds of applications).

~~~
pjmlp
I usually work with Fortune 500 companies, so there the price of these JVMs is
a water drop in the type of budgets we work with.

But for the scenarios you described, I agree with you.

~~~
strlen
Cost is an issue if you're deploying on large scale. The fact the source is
not available is an even bigger issue.

OTOH if using a proprietary solution ($$ and closed source) was fine, I'd
personally go with C# or F#. If you're fine using a closed-source runtime, why
not just use a better language (not to mention Microsoft's products are free
for startups via BizSpark)?

~~~
pjmlp
Speaking about our own projects.

Because Mono is not a solution when you need high scalable servers running in
commercial UNIXes, z/OS and similar systems? These type of solutions always
get to use C++ or Java in our projects.

My current project is actually done in C#, but it is a 100% Microsoft stack.

------
spiffworks
Is this right? Java really has better latency than Go right now? And with 2
processors, Go's latency is 25ms for 50 concurrent requests. Anybody with
better experience care to comment on this?

~~~
pjmlp
A little known secret is that most mainstream languages have nice libraries to
handle multi-core programming that go beyond basic thread handling, similar to
what Go offers.

You just have to know where to look for, plus you don't need to throw away
mature languages and tooling.

~~~
erichocean
For the JVM, Akka is pretty sweet. It comes with both Java and Scala bindings.

<http://akka.io>

------
davidw
It'd be fun to throw Node.js and Erlang into the mix.

~~~
dgv
You can see vs Erlang in
[http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...](http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=erlang)
and have a idea, Go is based on Newsqueak (lauched later of Erlang, other
perspective to implement CSP).

~~~
davidw
Sure, but this benchmark was more about concurrency, whereas those benchmarks
are more generic. Erlang is not the fastest language out there, but it's
supposed to be good at this concurrency stuff...

~~~
igouy
Erlang is supposed to be good at distributed, _reliable_ , soft real-time
concurrent systems.

<http://www.erlang.org/faq/introduction.html#id49480>

~~~
davidw
Somehow, it seems that when Erlang was getting a bit of hype, they abandoned
the field for an even smaller niche, leaving the "real time web" type things
to other languages. Granted, the language _was_ created for that smaller
niche, but if that's the only bit of terrain they want to defend... I see
other languages squeezing them out over time.

I don't know if that makes sense, but languages need somewhat large
communities to really thrive, in my opinion, and defining yourself into a
small niche isn't a good way to get them.

~~~
igouy
For a short time some people hyped Erlang to be _the_ solution for all things
concurrent and chatterers didn't read the FAQ.

------
kristianp
Here's an insightful comment from the blog page, about cpu and memory usage:

I downloaded patricks fork and got 4300r/s with maxprocs 1 with the go
version, out the box.

Recompiling and running with gomaxprocs=100, i got 4400r/s.

mvn deploy and running auth-1.0.jar on JDK 1.7 on my box similarly peaked out
at 3100r/s.

It's worth noting though, that I was using patricks httperf bench.sh
modification, and it appears that httperf was cpu bound in both cases, with
the kernel and postres taking about half a core between them.

Using wrk, by contrast, spun main (the go program) up to 2.5 cores, and
6000r/s. Java under wrk lit all cores for a time, then hit `Exception in
thread "async-log-appender-0" java.lang.OutOfMemoryError: Java heap space`
`Caused by: ! org.postgresql.util.PSQLException: FATAL: sorry, too many
clients already`

A bit more investigation and tuning later, using 10 clients `wrk -c 10 -r
100000 -t 4 -H 'Authorization: basic apikey_value'
<http://localhost:8080/authenticate`> Java was spinning out 10kr/s. Go was
limited to 6kr/s. It's worth noting some more details however: Go only ever
spun up two postgres forks, whereas java spun up 10. There's scope for
optimization there. Go used 8mb of ram, whereas java was sitting on 130mb
after a few runs. The Java version, cranking out at 10kr/s was maxing the
kernel out on one core, so that's probably approaching the practical limit for
single machine tests.

I suspect putting a load balancer in front of a couple of instances of the go
program would allow you to totally smash the java performance, given that it
lit 2 cores at 6kr/s, and java lit 8 at 10kr/s. The memory usage tradeoff is
significant - JVM sitting at 130mb and Go sitting at 8mb. Clearly everyone
needs to draw their own conclusions on this, for their own purposes. The JVM
solution is carrying a stats server, a ton of other tooling and so on. The Go
system has some - pprof was included, but it's limited by comparison. Arguably
gdb and so on can actually be of real use in the Go case, but that's also an
exercise for the reader.

Interesting any which way you look at it. There are a lot of other interesting
side effects (that need working out) in both programs as evidenced by this
simple testing. Postgres also needs some tuning if you really want to slam
this with anything remotely resembling a real world scale test.

My tests were done on OSX 10.8 12B19 on a 2.3GHz i7 wiht 8GB of DDR3 at
1333MHz, an intel 510 SSD, totally untuned postgres. The machine is a Macbook
from whatever year that is.

Here's wrk, for anyone looking for it: <https://github.com/wg/wrk>

