
Rust on Computer Language Benchmarks Game - oscargrouch
http://benchmarksgame.alioth.debian.org/u64/rust.php
======
tomp
It's ridiculous _and_ sad that they would include Rust, after banning LuaJIT a
few years ago, citing "too much maintenance effort" (even though LuaJIT
benchmarks were significantly more complete that Rust's)

~~~
masklinn
The shootout has been utterly ridiculous for quite a long time. I remember
when the maintainer(s?) threw a fit when pypy devs tried getting python
versions not relying on cpython tricks (which didn't bench well for pypy) even
though multiple programs per language had not been an issue before. Then they
ended up banning multiple implementations, but their implementation pick is
completely arbitrary.

~~~
brandonbloom
Seems like an opportunity for an enterprising hacker to create a more open and
collaborative "benchmarking social network" of sorts.

~~~
mutagen
TechEmpower has done pretty well with their web framework benchmarks [1].
While I get the feeling that they don't have the bandwidth to expand into
general language benchmarking they've created an open process [2] that might
be easily adapted to general language benchmarking.

They seemed to have addressed most of the fairness and "What about language X
on platform Y using framework Z" kinds of questions with "Send us a pull
request and we'll include it in the next benchmark".

I'm tempted to tackle this even though I don't currently have the time.

[1]
[http://www.techempower.com/benchmarks/](http://www.techempower.com/benchmarks/)
[2]
[https://github.com/TechEmpower/FrameworkBenchmarks](https://github.com/TechEmpower/FrameworkBenchmarks)

~~~
pekk
Except they obstruct issues and pull requests which would fix artificially
hamstrung implementations, rather than actively seeking implementations to
keep them honest. The benchmarks then represent a fantasy world where
completely normal optimizations are disallowed. The point of saying "send us a
pull request" is just PR and it seems to be working.

~~~
RyanZAG
AFAIK the only optimization they disallow is the use of caching results. It's
clear why they do this: if you allow caching, you're just testing memory fetch
speed and all your results across frameworks will probably look about the
same. The idea of the benchmark is to actually do the JSON
serialization/database access/etc.

I believe they have plans to add additional tests which will allow for caching
and that kind of thing.

I don't think that removing caching across all frameworks equally is creating
'artificially hamstrung implementations' as all the frameworks would benefit
from caching.

------
xmonkee
How is Go slower than Java? Actually Go is pretty slow compared to many
languages I would've expected it to be faster than.

~~~
RyanZAG
There's this strange idea still floating around that Java is somehow slow.
Compared to hand crafted micro benchmark C code, Java is slower. But then so
is everything else.

The simple facts are that the JVM is the most advanced and efficient VM
available today, and the JVM is specifically targeted to increase the speed of
Java. It's not much of a stretch to expect good speed out of that regardless
of strange preconceptions you may have had.

The Go VM has a long long way to go be even remotely comparable to the JVM in
garbage collection etc.

~~~
pekk
How much slower than C do you have to be in order to be called slow?

Apparently, the cutoff is defined like this: as slow as Java is not slow, but
any slower than Java is slow.

~~~
RyanZAG
Go is anything but slow, it's just not as fast as Java. I was just addressing
the idea that Java is somehow slow which seems to float around occasionally.
Java is arguable the fastest language that isn't directly C-based and anybody
thinking of it as slow is simply misinformed.

------
hangonhn
Anyone else a bit shocked by how well Javascript on V8 performs? I might need
to rethink my assumptions but it actually runs faster than Python 3 according
to this.

~~~
frik
Python 3 is slow, one should compare PyPy, HHVM-PHP, LuaJIT, etc.

A company did that back in Dezember:
[http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=update)
(found it on HN)

Revel uses LuaJIT (afaik), HHVM is HipHop PHP (Facebook) and you can find PyPy
there too.

~~~
RyanZAG
That benchmark isn't benchmarking what you think it is.

 _" This test exercises database writes. Each request is processed by fetching
multiple rows from a simple database table, converting the rows to in-memory
objects, modifying one attribute of each object in memory, updating each
associated row in the database individually, and then serializing the list of
objects as a JSON response."_

You're actually just benchmarking the mysql driver for each of those
languages, not the actual JIT capabilities.

~~~
frik
The website design is a bit confusing, you probably missed the other tabs:
"JSON serialization", "Single query", "Multiple queries", "Fortunes", "Data
updates", "Plaintext". It's a micro benchmark but at least it covers a lot of
languages and frameworks. (I saw that benchmark site on HN some months ago)

------
Rickasaurus
What's the deal with pidigits? Losing out bad to PHP here even.

~~~
iopq
because it's not using GMP

~~~
saidajigumi
So if I understand correctly, the pidigits test in this case this isn't a
self-contained, fully ported microbenchmark, but rather a test of some random
Rust code vs. an optimized library[1]?

If that's true, then what's the point of including pidigits if the entire
algorithm isn't going to be ported into all target languages?

[1] GMP, The GNU Multiple Precision Arithemtic Library:
[https://gmplib.org/](https://gmplib.org/)

~~~
pcwalton
As I understand it, the rule is "if your language comes with a bignum
implementation, you must use it, otherwise you may use GMP". I've been
seriously tempted to suggest removing our bignum implementation for this
reason (though I won't).

~~~
qbproger
Python uses GMP while having a bignum implementation. I believe any language
is allowed to use it.

Python:
[http://benchmarksgame.alioth.debian.org/u64/program.php?test...](http://benchmarksgame.alioth.debian.org/u64/program.php?test=pidigits&lang=python3&id=2)

Ruby:
[http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...](http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=pidigits&lang=yarv&id=2&data=u64)

------
Scaevolus
Why don't any of the Rust benchmarks have their source code on the site?

~~~
steveklabnik
Unsure, but we're pretty sure they take them from here:
[https://github.com/mozilla/rust/tree/master/src/test/bench](https://github.com/mozilla/rust/tree/master/src/test/bench)

All the shootout* ones.

~~~
pcwalton
I actually have no idea where they come from. nbody in particular is very
strange because we've been on par with C in that benchmark for years in my
tests.

~~~
igouy
In the space where the source code would be it says "See rust/src/test/bench"

------
riquito
Isn't a bit soon to benchmark Rust?

~~~
brson
Maybe not. Rust is starting to mature and the code generation is already quite
good (though there are still many opportunities for improvement). Hopefully
this exposure will inspire people to improve these benchmarks. With a little
effort Rust should be very competitive on them.

