

Java is Faster than C++ and C++ Sucks Unbiased Benchmark - gongfudoi
http://kano.net/javabench/

======
aggieben
_sigh_.

The amount of work it takes to produce a really good benchmark is something
more than what this guy did.

Exhibit A: his analysis of optimization impact.

 _I've been accused of biasing the results by using the -O2 option for GCC,
supposedly because -O2 optimizes for space, thus slowing down the benchmark.
This is not what -O2 does. According to the GCC -O documentation..._

-O2: Optimize even more. GCC performs nearly all supported optimizations that do not involve a space-speed tradeoff. The compiler does not perform loop unrolling or function inlining when you specify -O2. As compared to -O, this option increases both compilation time and the performance of the generated code.

 _On the other hand, -O3 performs space-speed tradeoffs, and -O performs fewer
optimizations. Thus, for these tests, I think -O2 was the best choice._

Uh...what about optimizations performed in the JVM (or, more specifically, the
JIT)? How do they compare? How do you know that -O2 is the right choice, other
than "it's kinda in the middle between lots of optimizations and not many, so
that sounds good."? He could have started with this:
<http://java.sun.com/j2se/1.4.2/1.4.2_whitepaper.html>

Exhibit B: he may have introduced a ton of bias through test selection and
doesn't even know it

 _Some of the C++ tests would not compile. I've never been very good at
decoding GCC's error messages, so if I couldn't fix a test with a trivial
modification, I didn't include it in my benchmarks._

What?? If you can't read GCC's error messages, I don't get any warm fuzzies
about your ability to decide which tests compare the things you want to
compare or not.

There's plenty more to be critical of here, but the above two really were the
sore thumbs in this article.

/me gets back to _real_ work and plans to dissect this more tonight

------
jcl
The testing decisions feel very arbitrary. If he really wants to show that
Java is better than C++, he must give C++ every opportunity to do better, then
show that Java does even better despite this. The correct optimization flag to
use is the one that gives C++ the best performance, not whichever he felt like
using. Likewise for the choice of benchmarks: he should use the version of the
C++ benchmark that runs the fastest, and he should have a C++ expert fix the
broken benchmarks. Otherwise the only conclusion that can be drawn is: "Java
experts are better at tuning Java programs than C++ programs".

While he's at it, he can try the new Shootout benchmarks, which clearly show
C++ in the lead, even when using his vaunted "-server" flag:

[http://shootout.alioth.debian.org/gp4/benchmark.php?test=all...](http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=java&lang2=gpp)

...How old is this, anyway? There's no date in the article, but he's running
gcc 3.3.1 and Java 1.4.2 on Fedora Test1, on a Pentium 4 laptop. The version
of gcc places it in 2003, so I'd place more confidence the newer Shootout
results.

