
My experience with the computer language shootout - kingkilr
http://alexgaynor.net/2011/apr/03/my-experience-computer-language-shootout/
======
jerf
It's true that it's arbitrary; the problem is, there's no non-arbitrary
alternative, so it's a weak charge to make.

Unless you're going to allow every language to converge on simply blasting
some bytes into RAM and executing optimized machine-language code and watch
all the languages cluster to the exact same location give or take how long it
takes to load the machine code, you have to draw some lines about what's a
"real" implementation and what isn't. If I were running it, I'd be even _more_
strict and insist that the solutions _must_ be "idiomatic"... and what's
idiomatic and what isn't would also be arbitrary, because there is no escape
from the "arbitrary". Yet the end result is useful. Not totally determinative,
but it's not anywhere near "useless" either.

~~~
akkartik
Combining your comment and DarkShikari's
(<http://news.ycombinator.com/item?id=2404278>), it seems like a good shootout
should simply compare the simplest, most idiomatic solution possible in each
language.

~~~
stewbrew
The problem with a comparison of idiomatic solutions is that they rely in
different degrees on what's in their libraries. IIRC the shootout tries to
avoid this by comparing similar solutions using the same algorithm that are
actually implemented with the interpreter/compiler in question. IMHO that's
the more sensible way to do this.

~~~
akkartik
Perhaps that concern is eliminated with good test programs? As an extreme
example, if I wanted to compare how different languages do array slicing it's
meaningless to force python to implement array slicing without the [:]
operator. But then this whole test program is meaningless anyway.

You're right that there's a lot of nuance here, and perhaps there's no better
solution. It's not even clear to me if there's a meaningful question to be
answered. "This ocean's warmer than that one? What part? In summer or winter?
Did you measure near a blue whale's fart?"

------
DarkShikari
In my experience language shootouts in _general_ are often terrible by design
-- with these being just a prime example.

A lot of the problem is in the questions they're designed to answer versus the
questions people use them to answer.

For example, if I'm comparing Python and C, I typically want to know "how much
slower would my program be in Python?", not "how much slower is my program in
Python if I spent so much time hyper-optimizing it that I might as well have
written it in C?"

But the test cases usually try to answer the latter, not the former.

~~~
dkarl
It might be more reasonable than it seems at first glance. It's true that it's
good to know how fast typical code runs, but there's another important
question: when I run into performance problems and need to optimize a
bottleneck, how fast can I make it before I have to resort to non-portable
code or C extensions that complicate my build process?

~~~
dagw
Anybody writing, for example, Python code to solve these sort of problems in
the real world would instantly reach for numpy. Which, while not part of the
core language distribution, is pretty close to being a standard library for
most python programmers. I'm sure several of the other languages have similar
libraries that are being ignored in these benchmarks. Without taking things
like that into account, theses results don't say too many useful things about
real world performance.

~~~
igouy
They say what difference numpy might make!

[http://shootout.alioth.debian.org/u64q/program.php?test=spec...](http://shootout.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=python3&id=2)

[http://shootout.alioth.debian.org/u64q/program.php?test=spec...](http://shootout.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=python3&id=6)

~~~
dagw
Cool, I missed that

------
glenjamin
I don't see why the same program has to run on PyPy, CPython and Python3.
Isn't the idea to do different implementations for each language. I know
technically they expose the same (or a similar) language, but they're
dissimilar under the hood.

I'd imagine the implementation varies across the other languages by more than
just syntax.

The point that worries me the most, is the amount of microoptimisation being
applied to these "benchmark" programs, making the results more or less
pointless for real-world use.

~~~
igouy
There's nothing to say that all of CPython, Python 3, and PyPy _must_ be
measured.

For the moment, they all are being measured, so it's interesting to see that
programs written for CPython might perform badly with PyPy, and programs
written for PyPy might perform badly with CPython.

~~~
glenjamin
I think in general, programs written in "Python" will perform better in PyPy
then CPython, but the current submissions are hyper-optimised for the
implementation details of CPython.

~~~
igouy
PyPy was shown for 2 years. PyPy performance with programs hyper-optimised for
PyPy would have been shown if someone had contributed them.

Is libc.write a great example of programs written in "Python"?

------
wriq
It's a shame that it all seems to be up to one person and what he will allow
considering how often the shootout is referenced in discussions. I wonder if
Mike Pall's experience was similar when he posted the Lua/LuaJIT versions.

~~~
mikemike
Same experience here with LuaJIT. My submissions using low-level types (byte
arrays and such) were put into 'interesting alternative', too.

Almost all other languages can use byte arrays, when they are the appropriate
data structure for the job. The C submissions make heavy use of GCC
extensions, Haskell gets to use mutable (OMG!) byte arrays and Free Pascal has
about as much in common with Wirth's Pascal as the name.

But Python and Lua are not allowed to do that? Apparently not all languages
are treated equally. Dismissing submissions by resorting to a flawed
definition of 'standard' and then suppressing further debate is really lame.

I contributed almost all of the Lua programs to the shootout, but I do not
feel particularly encouraged to continue contributing any programs.

~~~
igouy
Here's how Alex Gaynor compared his program to your programs-

"It's not used to do any crazy hackery like the LuaJIT one, just to access the
libc write() function."

~~~
mikemike
So what? Look at the C, Ada or ATS submissions for even more 'crazy hackery'.
My programs are quite straightforward in comparison.

------
onan_barbarian
The Shootout's C implementations do occasionally turn into a festival of SSE
intrinsics and the associated unrolling. Some benchmarks, anyway.

Still, not using SSE when it's available is dumb and perhaps all the other
languages need to start playing, too. A little tricker in Python, of
course....

~~~
kingkilr
Well, that's what the compiler is for! PyPy uses SSE under the hood for
various float handing things.

~~~
onan_barbarian
Unfortunately, SSE - especially in integer-land, so I suppose we're talking
SSE2 and beyond - often requires more specialist care and feeding than any
automatic method (compiler or run-time) can provide.

Some cases aren't hard to pick up (e.g. bulk operations on big arrays) but
others require trickery of the kind that compilers usually don't (or couldn't)
have.

This isn't made easier by the notoriously non-orthogonal nature of the SSE
integer operations and the rather limited number of ways that you can get in
and out of SSE-land (to, say, affect a conditional or get something into a
GPR).

------
robryan
Maybe you should have to submit and have it ran against a secret program that
covers many areas of an implementation, so it's unknown precisely what needs
to be optimized to improve on the test. Have a limit for how often you can
retest it to so the only real way to be the fastest is to optimize lots of
stuff which is a net win for language users.

------
scythe
Why not at the least accept the first submission for the PyPy entries? It
shouldn't matter that it doesn't run on CPython -- if it's idiomatic Python
and it runs on PyPy it looks valid.

At the same time, I can see where the guy is coming from with ctypes.

