
Evaluation of C, Go, and Rust in the HPC environment [pdf] - steveklabnik
http://octarineparrot.com/assets/mrfloya-thesis-ba.pdf
======
bsdetector
The author has a "lack of equal experience in the three evaluated languages",
especially C, but is comparing them for development time and SLOC based on a
sample size of one developer and one very small program.

He rewrites a distributed program as a threaded one, because Rust and Go
didn't have a distributed computing library, but says Go and Rust have
"similar performance" in HPC.

He says he was surprised at the C performance, but there's no investigation of
a cause. Maybe just because C is slow...

I don't mean to be overly critical, but these results in the paper mean
absolutely nothing. It's a waste of time.

~~~
pnathan
pretty sure all programs were multithreaded.

I do agree that the advisor should have pushed back lots harder on the
methods; it's a weak paper, even as an experience report.

~~~
hurin
> I do agree that the advisor should have pushed back lots harder on the
> methods; it's a weak paper, even as an experience report.

I agree - this seems to be mostly the advisors fault, by the looks of it it's
not that the student didn't put a lot of work into it but methodological
problems are what advisors are supposed to be there to address; I saw a lot of
this back in college, at least with technical papers it's easy to tell what's
wrong, with the humanities it's just a complete disaster.

------
dgacmu
The problem with reading this in BS thesis form is that it hasn't gone through
the wringer enough. Most of the performance comparison results vs C are bogus
because (to its credit, as noted), the author accidentally compiled some
important support libraries for the C version _without optimization_ and used
those on the cluster. Oopsie.

 _As briefly mentioned in the previous chapter this performance regression
might have been caused by the two unoptimized libraries that were compiled on
the development laptop and copied to the cluster._

~~~
mrfloya
Hey author of the thesis here.

You are correct this was definitely an oversight and I would have redone the
measurements if time allowed for it. However I just want to state that the
libaries were built with optimizations just not on the target platform which
might not have been clear from the qoute.

~~~
acadien
Hey Floya, I'm writing this here hoping you'll see this comment. You're
getting a lot of unjust responses in here. The commentors are reading your
thesis with the eyes of a trained skeptic expecting peer reviewed,
collaborative work that takes years of experience to produce. They're
nitpicking flaws and mistakes _we all make_ when we first start researching.

Your thesis is in fact really impressive and very nicely written! Your
conclusions and future work section is _excellent_. Really very well done.
Congratulations on finishing it. I hope you don't let the irrationally
negative feedback here dissuade you from continuing your work, you certainly
have a bright future ahead of you.

------
deadgrey19
Keep in mind, this is a bachelors thesis. It's not a peer reviewed research
article in a respected systems journal or conference. I mean this in both a
positive and a negative sense. Positive in sense that the purpose of a
bachelors thesis is to introduce undergraduates to an extended, self directed
project with an element of research to it. As someone who has supervised many
such projects, I think that this is a pretty good one. Great work! At the same
time, since the author is an undergraduate, presumably with a full coarse
load, this is not a high quality research publication. The results are
probably a bit shaky and the methodology is perhaps a bit off. But that's ok.
Every excellent piece of work has to start somewhere.

~~~
tomrod
Given the comments in this thread, I think the author should be thrilled! What
great feedback!

~~~
deadgrey19
Agreed!

------
steveklabnik
GitHub repo with all of the code: [https://github.com/mrfloya/thesis-
ba](https://github.com/mrfloya/thesis-ba)

Most interesting part of the thesis, for me, is page 42:

> In this case it was even an advantage that the Rust version was developed
> last since it revealed a critical error in the other implementations.

Null strikes again!

~~~
yiyus
He would have found the problem in Go too if he was using the comma-ok form:

    
    
        n1_idx, ok1 := g.nodeIdx[n1]
        n2_idx, ok2 := g.nodeIdx[n2]
        if !ok1 || !ok2 {
    

When you do not use the comma-ok form, you are specifically asking the map to
return the zero value for non initialized elements (which is something fine to
do in many cases).

~~~
heinrich5991
Well, you're implicitely asking for it. In Rust you have to explicitely
specify your intent, i.e. not by omitting something, but rather by specifying
that you want to ignore it.

~~~
yiyus
True. I agree it could be better if omitting the check looked like the
exception and not the other way around. You could also argue that encoding
non-presence on the type is more powerful/elegant/whatever than multiple
return values. But what I wanted to point out is that the problem is not that
"Go indicate[s] this error case with a return of zero". The language has an
easy way to check this and he ignored it (he eventually used comma-ok in his
code, although he makes the check first and the assignment later in different
operations for no apparent reason).

------
tines
I feel that it's a shame to test C rather than C++, because the latter offers
so many advantages to developer productivity -- the paper cites memory
management and type-related errors as being major causes of developer
unproductivity, and C++ has automatic memory management and templates and
such.

Also, it doesn't restrict the use of C libraries, so even if you've been
writing C for HPC code all these years, there should be minimal technical
difficulties in switching over, except for the time it takes to learn C++,
which shouldn't be long if you know C well.

------
Sanddancer
Looking through the links, and methodology makes me question a lot of things,
most of them regarding his choices in how he allocated memory. First and
foremost, he was using the glibc malloc, which belongs in a museum, not in a
running system. One of rust's big advantages in performance comes from that
alone, given all the memory that's being allocated and destroyed. Which leads
to the second question of how he did allocate objects. He knew he was going to
allocate a number of objects, yet he didn't pre-allocate a large chunk of
memory ahead of time to make it so he wouldn't have to call malloc and friends
as often. I wouldn't be surprised if spending a bit of time just tuning memory
didn't cut significant time from how long execution takes. I really wonder how
much time was spent in the system context of all three of those; I'd be
willing to bet that the final part of the examination would probably fare much
different had he used the same malloc for the C program that rust and go
already use.

------
gaius
Any HPC language evaluation that doesn't include vs FORTRAN is pointless. It's
still king of computation for a reason.

~~~
runlevel1
Experiments aren't usually written in FORTRAN.

The programmer is by far the slowest part of HPC. It often makes sense to use
the language you're fastest at instead of the language that will run the
fastest.

It's very common for experiments to be written in languages not know for their
performance (Java, Matlab, Perl, and Python for instance).

For reference, here are some of the packages commonly used in experiments[1]:

    
    
        BLAS        Fortran                 http://www.netlib.org/blas/#_software
        NCBI BLAST  C++                     ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.2.30+-src.zip
        BFAST       C                       http://sourceforge.net/projects/bfast/
        BioPerl     Perl                    https://github.com/bioperl/bioperl-live
        Bowtie      C++, C                  http://sourceforge.net/projects/bowtie-bio/files/bowtie/
        Clustal     C, C++                  http://www.clustal.org/omega/#Download
        cp2k        Fortran                 https://github.com/cp2k/cp2k
        Gromacs     C                       https://github.com/gromacs/gromacs
        HTSeq       Python                  https://pypi.python.org/pypi/HTSeq
        MUSCLE      C++                     http://www.drive5.com/muscle/downloads.htm
        MrBayes     C                       http://sourceforge.net/p/mrbayes/code/HEAD/tree/
        OpenFOAM    C++                     https://github.com/OpenFOAM/OpenFOAM-2.3.x
        SAMtools    C                       https://github.com/samtools/samtools
        SNAP        C                       http://korflab.ucdavis.edu/software.html
        fftw3       C                       https://github.com/FFTW/fftw3
    
    

[1]: I went down this list and picked some of the ones I remember using (been
out of the HPC world for a couple years):
[https://portal.tacc.utexas.edu/software](https://portal.tacc.utexas.edu/software)
It'll be a bit skewed towards genomics.

~~~
east2west
The list is not just skewed toward bioinformatics; a majority of them is
bioinformatics software. I would not call BioPerl or HTSeq high-performance-
computing software. Some of the software are written by academic programmers
with all attendant baggages. I am not aware of any bioinformatics software
written in Fortran, which is not popular in the bioinformatics community. But
this is not to say Fortran might not be a good fit. The Broad Institute is
rewriting some of its tools in C++ with help from Intel to leverage vector
instructions. Who is to say if they had started with Fortran they might not
have gotten vector instructions for free earlier.

------
ericjang
I'm a bit surprised that the author didn't mention julialang.org in the report
- it's much more suited for technical / scientific computation, while Go and
Rust seem better for systems-level concurrency/design.

~~~
petermonsson
I'm willing to cut the auther some slack here. A bachelor thesis is limited in
scope and a bachelor student can't know all languages.

------
drwells
My biggest criticism of this work is that it does not consider C++. Most of
the big HPC projects today are written in C++, not C, and to the best of my
knowledge this transition happened in the 90s.

~~~
buster
Not Fortran?

~~~
drwells
I have not done a census of the various libraries around, but all of the free
finite element libraries in common use (deal.II, libmesh, and FEnICs come to
mind first) are written in C++. I suspect Fluent and Abaqus are written in C
(maybe Fortran?) but they are a bit older.

Trilinos is another C++ example, while PETSc and HYPRE are written in C.

My tentative conclusion from this is that newer projects tend to use C++ and
slightly older projects use C.

Yes, there are things like ODEPACK, QUADPACK, and FFTPACK, but those are not
under development anymore (as far as I know). The only widely used Fortran
library still under development I can think of is LAPACK.

I did not count the occasional 'bespoke' code. There are still some Fortran
applications in development for particular purposes (like MOM), but those are
harder to survey.

~~~
yiyus
I do not think free finite element libraries are representative of commercial
software. They usually include a mix of different languages, and that includes
lots of Fortran, which is used both in the core (in solvers and linear algebra
libraries, for example) and at the highest level for user subroutines. I have
seen people using either Fortran or C++ with Abaqus, LS-Dyna or MSC, but
nobody writting C (except for some very experimental solvers).

Most users are usually fine with whatever the GUI allows and a bit of Python,
but for those of us developing new models, Fortran is probably the most useful
language, followed by C++.

------
angry_octet
An interesting case study, though not really HPC or scientific computing
related. It would be good to see some profiling to understand exactly why each
implementation was slower in particular ways. From a HPC context, linking with
Fortran and C codes, and exchanging data with them, is very important.

------
semi-extrinsic
Would've been nice with some internal links in that pdf, especially in the
table-of-contents (links to sections and subsections). This is simple with
hyperref.

~~~
ThePadawan
Has the link has changed since your post? The ToC is linked for me.

~~~
semi-extrinsic
Ah, sorry, it's just Quickoffice's pdf viewer (on Android) that somehow
manages to not understand the links.

------
short_circut
"In times where compilers are smart enough to basically rewrite and change
code for performance reasons it is completely inexcusable that the order of
source arguments to process is still that relevant."

After reading the comments on C and the difficulties the author had in the
compilation process.... All I can say is Garbage in garbage out. I have worked
in the HPC enviornment for a number of years using both Fortrash and C. The
errors the author encountered would have been avoided by someone with any
experience developing large softwares. I learned these things after working on
my first project and now they barely, if at all, factor into my development
time. Make sure you build and keep a dependency graph while you are developing
and this becomes a non issue.

~~~
chm

        Fortrash
    

Why the hate? One could misuse any tool, but that doesn't make the tool bad at
what it does.

~~~
short_circut
I don't disagree. Fortran is brilliant at what it does.... but I find many of
its object oriented features to be very obtuse compared to C and C++. The way
it uses pointers drives me insane. Building certain datastructures in fortran
just takes too much time.

I kind of wish I could find a good way to make Fortran and C play nicely
together. (There are some compiler dependent peculiarities that I have
encountered that make it difficult.)

Fortran's intrisics and file handling capabilities make it indispensable in
the work I do.... So it is really a love hate relationship.

~~~
AlexeyBrin
> I kind of wish I could find a good way to make Fortran and C play nicely
> together.

Fortran can call C and vice-versa, at least since Fortran 2003 there is a
standard way to call C.

~~~
short_circut
Oh yeah? I have looked and looked for it but been unable to find any reference
to how to do so. Would you mind linking a resource to me?

edit: Nevermind I found a few. When I looked for this a few years ago I just
kept hitting dead ends.

------
pnathan
Interesting work.

It has some pretty basic flaws that are pervasive that the reader should be
aware of. First off is the "productivity" measure: the last time I looked
around, this was nearly impossible to quantify for software development. The
author chose SLOC and development time as stand-ins for productivity. SLOC has
a connection with code quality and time of development, but development time
is well known to vary[1].

In particular, development time in this thesis is linked to results in a
pretty classic "Psychologist's fallacy"[2]; the author generalizes their
experiences as conclusive. This implies in particular that "development time"
in this thesis should be thrown out for meaningful conclusions, as the sample
size is 1. It is, however, an interesting experience report.

Another basic flaw is the connecting of 'modern' with 'good'. The author
remarks the main disadvantage of C and Fortran are their age; this crops up
here and there. Workflow tooling in Go and Rust are major focuses by the
developers: C workflow tooling is usually locally brewed. This does not make
C-the-language worse.

More on a meta level, my advisor in my Master's work drummed into me that I
should NOT insert my opinion into the thesis until the conclusion. So I found
the editorializing along the way very annoying.

And finally, and very unfortunately, the experience level of the author
_appears_ to be low in all three languages; this is significant when it comes
to implementing high performance code.

\---

Now, for the interesting / good parts of this experience report.

Standout interesting for me was the Rust speedup on the 48-core machine. I did
not expect that, nor did I expect Go to make such a good showing here as well.

In _general_ Go performance/memory made a surprisingly good showing (to me)
for this work. I shall have to revise my opinion of it upward in the
performance axis.

I am both surprised and vaguely annoyed by the Rust runtime eating so much
memory (Would love to hear from a Rust contributor why that is and what's
being scheduled to be done about it).

Of course the stronger type system of Rust catching an error that C and Go
didn't pick up is both (a) humorous and (b) justifies the type community's
work in these areas. I look forward to Rust 1.0!

One note by the author that is worth calling out stronger is the deployment
story: Go has a _great_ one with static linking, whereas C gets sketchy and
Rust is... ??. I believe Rust has a static linker option, but I havn't perused
the manual in that area for some time. For serious cloud-level deployments
over time, static linking is _very nice_ , and I'm not surprised Google went
that route. It's something that would be very nice to put as a Rust emission
option "\--crate-type staticbin".

Anyway. I look forward to larger sample sizes and, one day, a better productiv

[1] [http://www.amazon.com/Making-Software-Really-Works-
Believe/d...](http://www.amazon.com/Making-Software-Really-Works-
Believe/dp/0596808321) The situation is actually far worse than just varying
developer time.

[2]
[https://en.wikipedia.org/wiki/Psychologist%27s_fallacy](https://en.wikipedia.org/wiki/Psychologist%27s_fallacy)

~~~
steveklabnik
> Would love to hear from a Rust contributor why that is and what's being
> scheduled to be done about it.

I haven't spent time with the code yet, so I can't comment. It shouldn't be
the runtime, though, as our runtime is about as big as C or C++'s. My first
thought would be that Vec's growth factor may be poor in this instance, but
without reading the code, who knows.

> Rust is... ??

Rust statically links everything but glibc by default. Experimental support
for musl was added last week.

~~~
madez
I read more than once that rust has no runtime. Could you please clarify
whether it has a runime, and if it does, what it consists of?

~~~
steveklabnik
So, basically _every_ non-assembly programming language has a runtime. Even C
and C++. But not all runtimes are created equal. Languages like C, C++, and
Rust have a very minimal, no setup required runtime. Other, higher level
languages often have something significantly more heavy. People will sometimes
say "no runtime" to mean the class of languages which have very small
runtimes.

Rust's runtime is here: [https://github.com/rust-
lang/rust/tree/master/src/rt](https://github.com/rust-
lang/rust/tree/master/src/rt) and [https://github.com/rust-
lang/rust/tree/master/src/libstd/rt](https://github.com/rust-
lang/rust/tree/master/src/libstd/rt)

As you can see, it's very, very small. It mostly handles things like
unwinding, at_exit handlers, and the like.

------
dibanez
Well, Go and Rust can do better threading on this problem than a not-really-
optimized C code with OpenMP. Thats a start. Message passing has to be shown
next, and then one can decide whether the installation and runtime of
Go/Rust/Python/Julia on something like an IBM Blue Gene/Q with its custom
compute node OS is easier than a link flag.

------
vans
80 pages for that, with one user and one implementation of one algorithm ?
Sorry but the rate knowledge/size is very low, so don't be impressed by the
page number, it harms the paper more than anything else. Even for a bachelor
thesis, it's mostly "filling"

------
ohquu
Why are there so many bold terms and phrases throughout the text? It's
extremely distracting.

~~~
garthk
I often find bold text helpful for key points, or for the first mention of key
terms to help orient me in a document. There was something about the mix in
this document that felt inconsistent. I found myself pausing to try to figure
out why the author wanted my attention on this particular word or phrase.

------
MordodeMaru
I wonder what would've happened with the results if biicode would've been used
as deps/package manager for C.

~~~
jryan49
Yeah it would of been nice to see the dev time split up into setup and actual
programming.

------
BuckRogers
Wildly offtopic- reading papers like this bring to mind for some reason, that
the next evolution of CPython, the one that deals with the Python2/3 rift
should be written in Rust.

