

Analyzing Programming Languages Using Rosetta Code - nkurz
http://se.inf.ethz.ch/people/nanz/research/rosettacode.html

======
malisper
There is a problem with comparing programming languages using something
similar to Rosetta Code. All of the examples tend to be short and somewhat
trivial, while in the real world, programs are magnitudes larger. Some of the
differences in programming languages don't show unless you look at large scale
programs. In addition, by looking at clean examples you lose out on how the
program changed over time and how long it took to develop the program, two of
the biggest differences between programming languages.

I care much more about how long it takes to create/modify a program, than I
care about how fast it runs or how concise it is.

~~~
icefox
I know that [http://www.codewars.com/](http://www.codewars.com/) does keep
track of how long it take someone to write a correct solution and could
compare the different languages on that metric, does Rosetta Code have that
data available or is that not measured?

~~~
ygra
Rosetta Code is simply a Wiki with a collection of tasks and a collection of
implementations in various languages. So no, there is no such data. After all,
you tend to not create the implementation on that site but offline and just
add it to a page when you're done.

------
igouy
Exhaustive statistical analysis and over-generalization based on inappropriate
data - garbage-in garbage-out.

\---

>> "… the average Go program is 18.7 times slower than the average C program.
Programs in other languages are much slower than Go programs … 4.6–13.7 times
slower than Go on average" page 7 <<

 _Elephant in the room_ \-- What kind-of programs did the Rosetta Code program
authors intend to write? Did they write their programs as-if run-time
performance did not matter? Did they trade-off program performance for pretty-
code / shorter programs? Did they trade-off program sophistication for program
clarity?

For example, look at the gap between the Shortest C++ programs on the
benchmarks game and the fastest. (I pestered a C++ program contributor to
write shorter programs.)

[http://benchmarksgame.alioth.debian.org/u64/code-used-
time-u...](http://benchmarksgame.alioth.debian.org/u64/code-used-time-used-
shapes.php#shortest)

>> " …every large project is likely to include a small set of core
functionalities whose quality, performance, and reliability significantly
influences the whole system’s; Rosetta Code programs are indicative of such
core functionalities." page 10 <<

Large projects are likely to identify that small set of core functionalities
and the programmers are likely to make different trade-offs for those parts of
the project, for example -- optimize performance over short code, optimize
performance over programming time.

\---

>> To have reliable performance measurements, the scripts repeat each
execution 6 times; the timing of the first execution is discarded (to fairly
accommodate bytecode languages that load virtual machines from disk: _it is
only in the first execution that the virtual machine is loaded from disk, with
corresponding possibly significant one-time overhead_ ; in the successive
executions the virtual machine is read from cache, with only limited
overhead). page 4 <<

The method does not take into account JVM dynamic optimization, and because we
are not told the actual run-times of the programs there is no way to even
guess whether the program has run long enough to gain the benefits.

\---

>> "Which programming languages are less failure prone? To answer this
question, we measure runtime failures of solutions of tasks T EXEC marked for
execution that compile without errors or timeout. … Thus, a terminating
program fails if it returns an exit code other than 0." page 8 <<

Why would program failure be a property of a programming language rather than
an indication that _the programmer_ did not test their program? (Or an
indication that the program was written for some other implementation of the
language? Or written for a workload smaller than the workload tested, given
available memory?)

Even if the program did not fail, is the result correct?

\---

>> "… we maintain that paradigms still significantly influence the typical
“style” in which programs are written, and it is natural to associate major
programming languages to a specific style based on their Rosetta Code
programs. For example, even if Python offers classes and other object-oriented
features, practically no solutions in Rosetta Code use them." page 11 <<

Not whether the Python programs should be considered imperative because they
did not use OO features, but whether the Java and C# programs should not be
considered OO because they did not go beyond the boiler-plate required by the
language.

------
ygra
What should be noted, though, is that Rosetta code examples are usually not
written for speed of execution, but instead in an educational manner and thus
favor clearer code over faster code. At least that's how I wrote my
contributions and I feel fits the aim of the project better.

~~~
vanderZwan
Isn't that kind of the point though? Highlighting the lanuage differences
proper, instead of comparing benchmarks that are optimised with non-idiomatic
code?

~~~
ygra
For code length maybe. But for performance I'd find that questionable. For
most things you don't need all the performance you can get from the machine.
If clicking a button spends 100 cycles too many, who cares? Performance is
something where you investigate when you have a problem and then optimize the
part that is problematic. Sure, if performance out of the box is inacceptable
and cannot even be optimized, the language should probably not used at all.

But for idiomatic code I'd find speed of execution a questionable benchmark,
simply because you rarely find idiomatic code in performance-critical places.
That holds for C as well as for Java. There are all sorts of rules that have
to be followed to make code fast, such as no allocations in the hot path, etc.
that don't usually hold for normal code.

~~~
buster
Or, simply: If i have language X and language Y and the most straigforward and
easy-to-read version of my program is 100x faster in language X and given that
both languages have the same featureset, i'll chose language X and not
language Y. Someone will always say "i can make it just as fast in language Y,
i just need hideous tweaks and 10 times the LOC". But that's a moot point.

~~~
dllthomas
I think the interesting question is when there is a fast-and-complicated
method available in both X and Y, and a slow-and-simple method available in
only Y. Saying Y has to use the slow and simple method because that is
"idiomatic" would be misleading, when you could be comparably fast with work
comparable to doing it in X and that is the solution that would be used when
performance matters.

I don't know whether this is actually a situation borne out by reality or
merely a hypothetical, though.

------
jeffreyrogers
The Go team must feel (deservedly) proud. Since the Rosetta Code programs are
typically written to be clear, this shows that without resorting to technical
tricks and knowing the intricacies of the language and its libraries Go
performs better than both Java and Haskell, but of which are often lauded for
their speed.

From anecdotal experience, I've written some Haskell code, and compared it to
optimized versions. There is a whole lot of knowledge that goes into
optimizing Haskell, since you need to know how to make things eagerly
evaluate, etc. Plus functional languages make it really hard to reason about
memory usage (doubly so for lazy, functional languages).

~~~
cryptos
I'd say that this "benchmark" is senseless. Ruby is for sure generally not
faster than Java, and C is not about 18 times faster than Go. So no reason to
pat any language on the shoulder.

------
msluyter
How is it possible that ruby is faster than Java? Are they counting jvm
startup time or something? That seems rather suspicious to me.

~~~
ptx
Also weird is that Java is supposedly more consise than C#. But it turns out
they're using "cloc" to measure of "lines of code", which doesn't take into
account the fact that C# programs typically put opening braces on a new line.

------
kazinator
There is a potential problem using Rosetta for this kind of comparative study
is that there are relatively few programmers working on the Rosetta code
samples.

For some languages, the main person who does all the work is not very skilled.
Perhaps that person is just doing it as a learning exercise. Or in other
cases, the person just doesn't have it on their agenda to make the program as
short as possible.

All else being nearly equal, languages whose Rosetta contributors fall into
the above categories will "lose" to those languages for which the "Rosetta
person" is an expert programmer and advocate of the language (perhaps one of
the authors of the language, even!), who is working deliberately to make the
code as slick as possible to promote the language, and regards Rosetta to be a
competitive playing field.

------
malvosenior
I wish they would have included Clojure or some other Lisp. I've been doing
some Haskell and am actually a bit disappointed with how much less concise
Haskell is compared to Clojure. I'm definitely a newb, but it feels like you
have to do a lot more boilerplate/duplication especially around records.

------
Kipper100
This was partly copied by a study that Wolfram Research did on the Rosetta
Code site...

[http://blog.wolfram.com/2012/11/14/code-length-measured-
in-1...](http://blog.wolfram.com/2012/11/14/code-length-measured-
in-14-languages/)

------
okasaki
I knew Java used a lot of memory, but that it's pretty much the worst is still
kind of a shock.

~~~
cryptos
Keep calm. This is not a relevant benchmark for real world scenarios.

------
cdnsteve
Nice to see a new set of quantitative data on this. Well done! I'd love to see
the study expanded to include JavaScript :)

~~~
haches
I guess the graphs would then no longer fit on a single page ;)

------
mmphosis
Python for readability. C for speed.

~~~
cdnsteve
Python is also the most failure prone. " By contrast, the scripting languages
tend to be the most failure prone of the lot; Python,in particular, is
significantly more failure prone than every other language"

~~~
Paddy3118
I've asked for more info on that so I can see what needs fixing and if these
non-zero-exit values appear for me.

