
Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml - trishume
http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
======
Avi-D-coder
Haskell without lens, text, vector, etc... is a bit like rust with only core
not std.

The haskell standard library is tiny. Libraries like lens are not optional. In
practice you won't understand any open source Haskell without rudimentary
understanding of lens. I get why parser libraries were banned, but excluding
lens, vector, and text?

I like Rust a lot, but haskell minus it's more advanced type system is just
Rust plus GC. Lets not pretend this is a fair comparison of languages when
it's primarily a comparison of standard libraries.

~~~
sbergot
This is why I gave up on Haskell. Lens works as advertised, but is a pain to
learn and to use in practice: the abstraction is tough to grasp and it is hard
to form an intuition about it. The compilation errors are laughingly esoteric.
The number of adhoc squwiggly operators is ridiculous. You also need to
understand a lot of language extensions to get how the type checking works.

To me it looks like an impressive proof of concept for a future programming
language based around it.

If I were to start a project with Haskell the use of lens would be explicitly
forbidden.

~~~
dymk
It's about as esoteric as somebody learning C++ for the first time. And from
that perspective, it's totally normal for errors or syntax to be weird looking
for a _long_ time.

Most of us, including myself, are biased towards languages like Java, C, C++,
Javascript, because those are what we learn first - and so our expectations of
what errors (or syntax) look like are shaped by our early experiences.

So I don't think it's fair to say that Haskell's compiler errors or quirks are
fundamentally less intuitive than something that GCC/G++ spits out even on a
sunny day. Just odd when we expect errors to look a particular way, but
Haskell is playing a totally different (not exactly harder) game.

~~~
StreamBright
Not at all. Several languages Rust included takes understandable errors
seriously. I am a Rust newbie but the errors are extremely easy to grasp and
fix my code.

~~~
dymk
You say "not at all", but only cite Rust (which I didn't mention). C++ has
_horiffic_ error messages, certainly at the level of a bad Haskell error
message.

I'd say my point stands pretty well.

~~~
liquidify
I don't think C++ errors are bad any more. 2019 compilers generally produce
very good error messages. The situations where you get into pages of template
nonsense in an error are becoming fewer and further between all the time.

~~~
fmap
C++ has bad error messages because of language design. Contemporary C++
compilers are very good at reporting clear error messages about common
mistakes, but template heavy code still yields arcane error messages.
Templates are untyped, so there is no way to give sensible error messages when
defining or instantiating a template. Instead you have to typecheck after
template expansion, at which point you are left with an error message about
compiler generated code.

There are some proposals which address this (e.g., concepts), but none of them
are part of the language standard yet. Concepts in particular made it into the
C++20 draft, but they also made it into a draft of the C++17 standard and were
ultimately rejected. Somewhat vexingly C++ concepts actually come with the
same problems only at the level of concepts instead of at the level of
templates.

------
azov
One big difference (which the article seems to mostly ignore) is that the
shortest project was written by a single developer, while all others were a
team effort.

Having the whole project - design, architecture, all implementation details -
in one head is not a trivial advantage. Even ignoring communication overhead,
there might be subtle duplication of code simply because different people
choose to do similar things in slightly different ways.

While the ratio of Rust to Python code kind of matches my expectations, I
wonder how much of it might be due to the difference in team structure vs the
difference in chosen language.

~~~
celrod
These were different teams of students taking a class. Variance in student
effort/quality is generally high.

There were two Rust teams. One had 3x the code and passed less tests than the
other. This is our only reference for how much noise is owed to team rather
than language; no other language was used by more than one team.

Python did best (least amount of code, yet also the most features) less
because of Python, but more because the best programmer was using it.

Would the other students have been able to take advantage of the duck typing
and metaprogramming in the same way, or ended up following different designs?
I'd have my doubts about the second Rust team.

Although that Python allowed these is a feature, but I think we're really just
looking at noise.

~~~
gricjeh
And then again still, the “best programmer” sacrificed code quality in pursuit
of quickly producing features. The article doesn’t define code quality
measures but I’d still take the lines of code metric as either not a measure
of value in itself or with a massive grain of salt.

~~~
firethief
That was the right tradeoff for the situation. I think any attempt to use this
case study to draw conclusions about maintainability would be a mistake;
that's just not what this is useful for.

~~~
256
Hi. I wrote the python implementation. I've had prior experience with
compilers and I'm obviously biased but I would say it's quite readable. The
main compiler part (without optional features or the LR(1) or lexer generator)
is 2040 lines and contains very little magic. It would be easy to translate
1-1 to idiomatic C++ for instance - it would just be much more verbose. eval()
is only used by the lexer generator. The metaprogramming could have been
avoided by using dataclasses (but the marker used python 3.5).

(To be clear, the optional portions in which I implement SSA, various
optimizations, a Hack-style allocator and some ad-hoc codegen is much less
readable.)

~~~
new4thaccount
Sounds really cool! As someone who programs a bit for work to help with
automating processes and fun, but is not a computer scientist I'm a little
jealous of never having taken a true database, compilers, PLT, or theory of
computation course.

It sounds like you and your classmates are top notch and will go on to some
pretty freaking cool careers (ex: I'd work at Jane Street if I was not a
parent and a lot smarter :)). Out of curiosity, what are the typical places
your classmates go upon graduation?

------
bfung
Excellently written, great topic, and done w/o flaming/too much bias. I'm
amazed as many older folks in the industry would not be able to have this
level of content and maturity to write an informative article.

~~~
WalterBright
It's difficult to write a fair comparison without being a fairly competent
programmer in each of the languages. The trouble is, if a person is an expert
C programmer and then translates it to Python that he's only modestly familiar
with, the Python program will look like C. It won't be idiomatic Python.

For example, my early Fortran programs looked like Basic. My early C programs
looked like Fortran. My early C++ programs looked like C. And my early D code
looked like C++.

It takes much more than being able to write a program in X to be able to write
one that makes proper use of the language.

~~~
galaxyLogic
That is all true. At the same time here the groups were allowed to use the
language of their choice. Presumably they chose languages they felt they were
competent in.

Of course an expert of a given programming language can write much better in
it than a novice. But a comparison like this is not necessarily about
comparing top-programmers in every language, but average programmers, because
we want to know results that are true "on average" .

The author does note he "knew (they) were highly competent". So they were not
exactly novices in their language of choice. Writing a compiler is not a task
for novices in general.

~~~
tigershark
There were people with 2k to 10k loc of experience in some language. That
seems _extremely_ low for any meaningful comparison and I would really hope
that “average” programmers have way more experience than that... I think I was
pretty junior after writing north of 100k loc and working on 1M loc projects.
And for sure I don’t consider myself highly competent in F# after writing some
thousands lines. I agree with the conclusions when they say that the design
decisions are much more important than the language of choice. But I still
firmly believe that the language makes a very big difference in real world
projects. In toy throw away projects obviously metaprogramming cuts a lot of
locs.

~~~
jodrellblank
When reading about APL recently, arcfide - the chap working on a GPU compiler
- has expressed that he likes APL's terse code because you can throw it away
and rewrite it without too much trouble. His compiler is around 750 lines of
code after 6 years of development, but[1]:

" _If you look at the GitHub contributions that I 've made, I've made 2967 of
about 3000 commits to the compiler source over that time frame. In that time
I've added roughly 4,062,847 lines of code to the code base, and deleted
roughly 3,753,677 line of code. [..] It means that for every one of those 750
lines, I've had to examine, rework, and reject around 5400 lines of code._"

Yet a 750 loc codebase sounds like someone with very little experience and a
few days or a long weekend.

I'm not saying only the trivial "some languages are denser than others", but
also that it would be interesting to compare projects in total lines of code
written including all commit history, and that it would be interesting to
compare people's experiences and project designs in terms of "how many times
something got implemented in several ways before settling on a final version",
or "writing once and the design got set in concrete because it was too big to
bother changing", how many prototypes that work differently were explored
before deciding - and what that does to people's skills, and to project
designs.

I'm guessing thousands of lines of F# would make you more skilled than the
same thousands of lines of C#, even moreso if the F# was higher-abstraction
than the C#, would you agree?

[1] - [https://www.sacrideo.us/smaller-code-better-
code/](https://www.sacrideo.us/smaller-code-better-code/)

~~~
ginnungagap
The github link in [1] is dead, where is the repo now? I was curious about
that compiler

~~~
jodrellblank
It would be unfair to leave it without the link to the HN discussion where he
explains in detail his reasons for writing it that way and defends against a
lot of criticism, and video stream walkthrough of the codebase as of a couple
of years ago

[https://news.ycombinator.com/item?id=13797797](https://news.ycombinator.com/item?id=13797797)

[https://news.ycombinator.com/item?id=13638086](https://news.ycombinator.com/item?id=13638086)

------
jpittis
> I think the smaller differences are also large enough to rule out
> extraordinary claims, like the ones I’ve read that say writing a compiler in
> Haskell takes less than half the code of C++ by virtue of the language

Specifically the "by virtue of the language" part:

Seems to me like it's unreasonable to claim the languages are on equal footing
because fancy parser libraries aren't allowed to be used for the project. The
fancy parser libraries exist for certain languages specifically because the
languages enable them to be written. (For example in Haskell: monadic
libaries, libraries that take advantage of GADTs, etc.)

~~~
trishume
I don't think monadic parser libraries have a real claim to be that
difference. All the languages listed have excellent parsing libraries that
make things similarly easy, if not by language power than by grammar DSL with
embeddable code snippets.

I think if any library could make a real difference for Haskell it's most
likely to be
[http://hackage.haskell.org/package/lens](http://hackage.haskell.org/package/lens),
which a Haskeller friend of mine claims could likely make a lot of the AST
traversal and rewriting much terser.

~~~
pyrale
Recoding a viable subset of lens would have taken 50 locs in haskell.
Likewise, rewriting parser combinators would not have taken long for
experienced devs. The problem here is that requiring people to recode the libs
on top of the compiler is disingenuous. And if you ban idiomatic libs, you
also ban most online help, tutorials, etc.

~~~
loup-vaillant
(A suitable subset of) Parsec is about 100 lines of OCaml. Implementing a PEG
syntax on top of it is about 150 lines of Haskell (or less, I'm a Haskell
noob).

Building up the knowledge to get to this point however… nope, those students
were better off going hand written recursive descent (or Lex/Yacc, since an
equivalent was allowed).

[https://github.com/LoupVaillant/Monokex/blob/master/src/pars...](https://github.com/LoupVaillant/Monokex/blob/master/src/parsec.ml)

[http://loup-vaillant.fr/projects/metacompilers/nometa-haskel...](http://loup-
vaillant.fr/projects/metacompilers/nometa-haskell-grammar.tar.gz)

------
galaxyLogic
I think the big big result from this study is: "Python: half the size !".

A dynamic language like Python is better here, 2x better. I assume similar
results would apply to other dynamic languages like JavaScript, Lisp,
Smalltalk, Groovy etc.

This does not say that static typing should not be used but I think it shows
unequivocally that there is a considerable extra development cost associated
with static typing.

You might say that surely that additional cost would be compensated in
reducing the cost of maintenance later. Maybe but I'm not sure.

Any development effort of significant size (like writing a compiler here) is a
combination of writing new code and adapting and modifying code already
written. "Maintenance" is part of development.

This is quite a quantitative study which gives credence to the claims of
advocates of dynamic languages.

~~~
trishume
I agree it's definitely an interesting result and a point in favour of dynamic
languages.

A caveat is that I'm pretty sure my friend intentionally sacrificed code
quality to do it, I don't think you'd find that project as readable and
understandable as the others. Another caveat is that you have to be okay with
your codebase being extremely magical and metaprogramming-heavy, which many
industrial users of dynamic languages avoid.

As I mention, I'm personally into statically typed languages mostly for the
performance and correctness benefits. I think it's plausible that on larger
projects with teams the correctness benefits save enough debugging time that
overall implementation time is lower, but I'm less confident in this now than
I was before this comparison.

~~~
bjourne
The takeaway for me was: "The next comparison was my friend who did a compiler
_on her own_ in Python and used less than half the code we did because of the
power of metaprogramming and dynamic types."

So it's the output of ONE Python programmer vs teams of other languages
programmers?

~~~
galaxyLogic
That is a good point worth emphasizing.

It is well-known that teams cause overhead. Think of the Mythical Man-Moth.

But in the end the other teams had 3 people that were able to maintain and
develop their project further, if needed. The single-person team had only one
such person.

~~~
pm90
But the Python code was also smaller. Easier to learn and maintain.

~~~
blumomo
"Smaller code" does not mean easier to understand. It just means less
characters to read. Maybe those characters are heavily loaded with meaning, as
in the case with meta programing. You might need twice as much time to
_understand_ the Python Code vs. the Rust code. The Rust code might be easier
to extend, etc. So this is all comes to trade-offs at the end. All this being
said, I'm still a huge Python enthusiast.

~~~
gmueckl
Smaller code may also mean more entangled and more rigid design. But that is
hard to tell from just the line count.

------
FullyFunctional
That was a really interesting article. I completely agree with

>>> Abstractions may make things easier to extend in the future, or guard
against certain types of errors, but they need to be considered against the
fact that you may end up with 3 times the amount of code to understand and
refactor, 3 times the amount of possible locations for bugs and less time left
to spend on testing and further development.

Choosing when and how to abstract is key. Abstraction in a fashion extends the
language so you are trading off a burden on the reader (who have to learn and
trust the abstraction) for increased expressiveness. Don't overdo it. (And
don't abstract idioms).

~~~
rjayatilleka
Could you clarify what you mean by "don't abstract idioms?"

~~~
FullyFunctional
One example (taken from the real world):

    
    
       addr = (addr + PAGESIZE - 1) & ~PAGESIZE;
    

contrast with

    
    
       addr = P2ALIGN(addr);
    

The former is idiomatic and immediately readable. For the latter I have to go
lookup the definition of P2ALIGN (and make sure I got the one actually used as
there may be multiple!), check that it does what I expect, and memorize it.

The value of abstracting the idiom is debatable. If it _is_ used sufficiently
often, then it might be worthwhile, but often it isn't.

------
boyter
I really wish when counting code that people would use one of the more modern
code counters, or at least use cloc. Tokei, loc, scc, polyglot, loccount or
gocloc give a much better idea over wc because they account for comments and
blank lines and in the case of scc and tokei strings.

I had a similar experience in university. Class had to implement a modified
Turing machine. We could use whatever language we wanted. On person did it in
C++ and it was several hundred lines. Another in Java which was slightly
smaller. I implemented mine in Python and it was small enough to print on a
single piece of paper. I think it was something like 30 lines or so. I
exploited break statements to make it terse but readable. It did mean I was
marked down a bit for them but it was by far the shortest program in the
class.

~~~
trishume
A few groups did give me output from a modern line counter, often Tokei or
loc. I mention in the post that the ratios between source lines, word count
lines and bytes were pretty similar across all projects. If people want to get
a sense of how that compares to their intuitions in source lines they can use
the ratio from our Rust project to convert.

I ended up using raw lines for comparison because for relative measurement it
didn't matter for the above reason, and I knew everyone had the same `wc` and
I could get them to send me the output of the `wc {glob}` version that lists
lines and bytes of all files so that I could drill down into which parts were
different.

~~~
boyter
Ah! I assumed you had access to the code. That makes more sense.

------
chenglou
Hey trishume! It’s been a while (this is the Reason person).

Great post; it echoes all the experiences I/we’ve personally had, especially
the alternative Rust section: on the spectrum of how much intelligence one
needs to understand fancy abstractions, all of us programmers, good and bad,
are pretty much lumped to the lower end. I’ve seen plenty of skilled folks
“decompensate” their expertise by reaching for overly complex abstractions.
The definition of a “strong” programmer, as per the post and usually in real
world, should probably be rectified to include not only “the person knows how
to use this abstraction”, but also “the person knows when to stop”.

In the same vein of idea, it’d be interesting to know how Go would fare in
this project. Go’s philosophy is pretty much the opposite of a language that’s
usually used for compiler research; but I’ve seen indicators that using it
could be surprisingly effective (and in light of this post, maybe less
surprising).

More importantly, it’d be nice to know the perf characteristics of each
project =)

~~~
Matthias247
> In the same vein of idea, it’d be interesting to know how Go would fare in
> this project.

I would say a bit worse than the mentioned alternatives, which all have better
type systems and thereby e.g. make it easier represent and manipulate the
trees that are everywhere in compilers. But most likely it's still fine, and
the line count metric would be more influenced by the fact how experienced the
author is than the language.

Things will get worse in Go if one wants not write a lexer/parser manually but
use tooling for that. Parser combinators and other tools benefit a lot from
generics.

On the other hand e.g. writing a network server with decent performance will
be in Go a lot easier and more straightforward than in any of the other
mentioned languages. While all those languages are general purpose programming
languages they definitely have their strengths in different areas. Some are
better for some tasks (e.g. compilers), other are better for others.

~~~
chenglou
I’m familiar with these positions (static types, ADT, parsing, etc). But my
comment on “less surprising” was meant for Python; it turned out that you can
get pretty far with it with no static types nor ADT. So it’s very possible
that Go can fare not too badly. The reason why I wondered about Go is because
I’ve seen several HN posts about using Go for writing parsers/compilers with
surprisingly ok result.

(I don’t advocate dropping static types or ADT; just that in the spirit of
this blog post, it might be worthwhile to examining our assumptions.)

------
ufmace
I haven't made it through the whole thing yet, but I do want to register a
vote in favor of using Lines Of Code count as a rough measure of program
complexity. I think it's a perfectly valid things to do, provided that it
isn't used as an evaluation metric and nobody is gaming it, and everyone is a
reasonably good programmer, not doing crazy things like trying to stuff a
massive algorithm onto one line to be clever, or copy-pasting the same code in
20 places because they don't understand functions and classes.

~~~
hart_russell
I'd like to hear others' opinions: There's a guy at my work who loves to use
doubly, triply, quadruple-ly nested ternary operators. I always find them
super hard to read. Am I just a dunce, or do I have a point in thinking it's
unnecessarily terse.

~~~
lyjackal
Usually it's awful. I wish JavaScript had if else expressions. Sometimes it
can help to break up the lines with indents if you need a single expression:

    
    
      var myvar = foo < bar
        ? bar
        : bar < baz
          ? baz
          : 0

~~~
_jsdw
I settled on this a while ago which I think is perfectly readable:

    
    
        const myvar
          = foo < bar ? bar
          : bar < baz ? baz
          : lark < 10 ? lark
          : 0

------
wjw94
My take-away from this article is that the university I graduated from was
terrible

~~~
davemp
I assume OP is in grad school. I took a few grad classes during my undergrad
and the difference was night and day (teaching and cohort). So maybe your
school was fine and you were just in the wrong classes. :)

~~~
trishume
Nope undergrad. UWaterloo just has a special co-op program that extends the
degree and fits in six four-month internships in alternating school and work,
for a total of two years of work experience by graduation.

------
tanin
The Scala result confirms my bias :P I love Scala because it strikes a great
balance between being succinct and offering type safety.

I wish you would use Ruby instead of Python. Python is strangely inconsistent.
For example, Python doesn't really offer a rich standard library; people have
to resort to ugly solutions for a simple problem (here's an example:
[https://stackoverflow.com/questions/363944/python-idiom-
to-r...](https://stackoverflow.com/questions/363944/python-idiom-to-return-
first-item-or-none))

Ruby would be better at an example of how a dynamic language can be more
succinct.

~~~
nindalf
Your linked stackoverflow question - how do you return the first item of the
list or None.

Proposed solution - `a[0] if a else None`

Your reaction - Python doesn't offer a rich standard library.

I disagree, and I suspect most programmers would too. We'd much rather have
ergonomic libraries to make http requests, parse command line arguments,
datetime, itertools, data structures like heaps, filesystem access, data
archiving, data serialization and deserialization and a million other nice-to-
haves. It's actually at the point where I've heard criticism of Python's
stdlib doing too many things, rather than too few.

If you don't want to write 19 characters to find the first item in a list,
pick another language. But don't mischaracterise python's standard library.

~~~
mruts
I have also found Python’s stdlib to be laughably bad. Just compare the data
structures available to Scala. I write Python everyday and honestly, it’s a
chore. The language is primitive and inexpressive.

The sad part is that it totally didn’t have to be this way. But instead of
evolving, Python is just stuck in the past.

~~~
snazz
Could you point to some specific examples?

~~~
mruts
Python has Lists (arrays), Dict, and Sets. That's pretty much it.

There's no linked lists. There's no sorted maps, no sorted sets, no maps that
preserve insertion order, no queues, no priority queues, no bitsets. And
there's no immutable collection in Python besides strings.

Meanwhile, Scala has all of that and more: [https://docs.scala-
lang.org/overviews/collections/overview.h...](https://docs.scala-
lang.org/overviews/collections/overview.html)

Moreover, scala has synchronized collections that can work over multiple
threads. I guess all collections work that way in Python, but that's because
Python doesn't even support true thread parallelism in the first place!

Also, if we're talking about the number of methods on the collections, Scala
has way way more. map/foreach/filter/foldl/foldr/option/drop/take/first/last
etc etc.

~~~
zodiac
> And there's no immutable collection in Python besides strings

frozensets, tuples?

------
awwaiid
If you're into this sort of thing, definitely check out the ICFP Programming
Contest -- take next Friday off and join in on the fun!
[https://icfpcontest2019.github.io/](https://icfpcontest2019.github.io/)

~~~
saagarjha
> Any programming language(s) on any platform(s) may be used.

How does this work? What’s stopping me from using some hyperpowered esolang I
cooked up for the competition?

~~~
dbpatterson
That the task description is only posted at the start of the competition?

~~~
geofft
I mean, if you design an esolang for the competition task and implement an
optimizing compiler that's faster than any general purpose language out there,
that absolutely feels like something that should be rewarded by the spirit of
the competition! So it's okay that it's allowed by the letter of the
competition, too.

------
spullara
I'm sad that no one wrote it in the subset of Java that it was going to
compile.

~~~
mruts
They probably wouldn't have been able to finish the project in time.

~~~
ncmncm
And a good professor would have flunked them for poor judgment.

~~~
trishume
My professor was definitely a good professor and no way they would have done
that. It clearly would have been done as an added challenge for themselves for
fun and there's no reason to penalize that.

~~~
abecedarius
It's definitely fun: you can also take part of the challenge to be figuring
out what parts of the full language are worth adding to make the overall job
easier (so you're not taking the language subset as set in stone). When I did
this for Python I started out in unrestricted Python, gradually both
increasing what's implemented and reducing what's used. (Result at
[https://codewords.recurse.com/issues/seven/dragon-taming-
wit...](https://codewords.recurse.com/issues/seven/dragon-taming-with-
tailbiter-a-bytecode-compiler))

------
analog31
I conducted a similar exercise, albeit on a much smaller scale, when I was
"language shopping." I took a project that I had written in Visual Basic (my
go-to language at the time), and re-wrote it in a variety of programming
environments.

The project was just complex enough to require maybe 100-200 lines of code at
the end of the day, thanks to liberal use of libraries. Not huge, but not
atypical for a "scientific" programmer who uses code as a way to solve
problems rather than to create software for others to use. It's pretty
representative of my life as a programmer.

The exercise forced me to learn enough of each language to get a feel for it,
and then I could look at the programs alongside one another to assess their
strengths. I also imposed some rules, such as that the language had to run on
multiple platforms, and be FOSS. In addition to comparing the languages, I was
also implicitly comparing libraries and even access to online help. This was
also my first real exposure to StackOverflow.

I tested Javascript, Python, GNU Octave, and wxMaxima. Ultimately Python won
out and is my language of choice today. This was just my own little exercise,
and not worth publishing, but has made me a believer in learning multiple
languages.

~~~
new4thaccount
Give Julia a try if you haven't. If you're coming from Python and also have
some Octave experience, it will be easy and worth a look.

For scientific coding I usually use either Python or Julia and Fortran or
Octave at home sometimes.

------
julvo
I like the idea of using this project to compare languages.

However, to me it seems the comparison is more of a comparison between
specific implementations rather than languages. The high variance in LOC
between the two rust implementations makes me wonder if there's a similar
variance for other languages and the samples we have lie somewhere between
e.g. 0.5 and 2 times the average size for a given language.

Therefore, the python implementation could just be a really compact one, even
when compared to other python implementations. It could be interesting to ask
the Professor if he'd be willing to collect some stats over the years.

------
edem
These comparisons are actively harmful. What the OP is doing is the same as if
you were trying to determine the winner of a race by looking at the
competitors at the start. You don't know their speed, their strengths etc. You
can't determine how maintainable the code you produce this way, how it
performs under real world usage, etc. Please stop doing these.

~~~
jquery_dev
I very much disagree with your opinion. OP never tried to determine a winner,
just wanted to share some insights into the project.

If we go by what you say, we would never have any comparisons at all, as there
are always unknowns.

------
dennisgorelik
That is an important conclusion: "design decisions make a much larger
difference than the language".

Software developers' expertise varies more than expressiveness of programming
languages.

------
zimablue
I feel like what you're actually testing with this is "amongst the top
percentile of programmers, what are the proclivities of people by choice of
favourite language"

Because the implementations vary so much, that's the source of a lot of the
LOC-difference. They effectively delivered more or less (if you count more
stages as "more" which I would as it will make it easier to reason
about/debug, and count a typesystem as "more", since it gives you more
guarantees).

python - solves the problem brutally fast, some ugly shortcuts haskell -
solves the problem quite slowly/delivered the most rust/c++ - intermediate
scala - solved fast, took shortcuts ocaml - this is the one that surprised me
I'd have expected it to be the shortest with python

~~~
dean177
Why would you expect the ocaml one to be shorter?

------
ncmncm
I find it curious that both the team that used C++, and the author, both
appear to believe that sum types and related facilities are not usable or
conveniently expressible in C++.

I got that Boost Spirit, a powerful parsing library, was forbidden. Were all
the Boost libraries similarly embargoed?

~~~
xyzzy_plugh
> Boost Spirit, a powerful parsing library

I regularly use Boost Spirit as an example of "sounds good but actually a
nightmare", and am always surprised to see it mentioned in the wild. It is
truly a modern horror.

Boost is like Apache -- a collection of libraries of various quality and stage
of development, not all alike. A lot of stuff in there is designed to prove
out experimental language features ahead of the next standard revision. A lot
of that stuff is convenient, but a complexity nightmare under the hood.

For your own sanity, and of those you care about, don't accept all of Boost
equally. Be suspicious, and treat each package as if it were some random
library you found on the internet.

(There are of course tons of great things in Boost, but tread carefully)

~~~
ncmncm
You won't catch me arguing in favor of using Spirit.

In general, it is rarely a good idea to rely on a library you don't
understand.

But Boost has serviceable variant, option, and result types.

~~~
jcora
Yeah the issue is whether the language makes it easy to use them on all levels
from syntax and features (simple constructors, pattern matching) to use in the
stdlib and wider community

------
anaphor
I think the right metric to measure here is not quantitative, but more
qualitative.

In other words, how easy it is to adapt the language to the problem you are
trying to solve.

So, in order to do that you need to have a good understanding of the
principles each language is based on. Once you've got that, then you look at
the resulting code and "measure" how easy it is to understand.

As was already said, this requires having someone with a very good
understanding of the underpinnings of each language, which is not really going
to be reasonable for most people.

The problem is that if you simply tried to port an idiomatic solution from
e.g. C++ or Python into Haskell or Scala, or whatever, then you would probably
end up with something very ugly, because you didn't adapt the language to your
problem. You tried to force it to do something it wasn't necessarily designed
to do.

~~~
Buttons840
A contest to see how easily code could be broken would be interesting.

"I purposely broke the code somewhere in these 20 lines. Find the bug and fix
it."

You can use the compiler, you can use the test suite. You cannot diff the code
against the original working code.

------
HelloFellowDevs
Love reading your posts Tristan, a question about the choice of Rust. Was it
specifically based on the previous experience you and your groupmates drew on
or were these languages assigned to students? Was there any point in time
where you and your team were trying to optimize your compiler?

~~~
trishume
We got to choose. Everyone on our team had some experience in Rust and OCaml,
the two of us more experienced in Rust were pretty indifferent between OCaml
and Rust, and our third teammate wanted to use the project to learn more Rust
so that's what we went with.

------
riquito
Another big takeaway is the reusability of integration tests. Write once,
verify implementations in any language.

Static or dynamic typing doesn't matter that much if you can verify your
software, and there are many different reasons to choose a particular
programming language.

~~~
mirekrusin
You need to look at the whole picture, time taken from day one to release,
performance/correctness, then maintenance/onboarding, how easy is it to
refactor/add features etc? It's virtually impossible to definatelly conclude
in absolute terms that one language is superior than other.

One of my best experiences was dovetailing that happened on a project that had
to be c or c++ where I prototyped fully working system in ruby first, then
rewritten it in c. Everything went almost too smoothly.

------
deng
This is an interesting writeup, but I'd say the groups settling on different
algorithms matters much more than the language used. It is obviously a pretty
big difference whether you use a recursive descent or an LR parser.

------
stretchwithme
It would be interesting to see experts in different technologies compete, in
front of a live audience, to implement or change a variety of projects. It
would be useful to see what a master can do with each tool or platform.

~~~
galaxyLogic
It would be interesting and educational to see how they would do it,
definitely. But when comparing languages it makes sense to compare how well
average people perform with them. We can't always hire an expert they are not
universally available.

~~~
stretchwithme
It would certainly help people decide what tobecome an expert in. And there
must be some correlation with what an expert and an amateur can do.

Of course, I can think of examples where that isn't true.

------
mamcx
Is possible to see the code of this project? And is the same of
[http://thume.ca/2019/04/18/writing-a-compiler-in-
rust/](http://thume.ca/2019/04/18/writing-a-compiler-in-rust/)?

P.D: I'm building a relational language and my ideas mimic closely "writing-a-
compiler-in-rust" ie: using pratt parsers and similar, but not see much of how
do that on rust...

------
trimbo
Where's the code? I scanned this article a few times and can't find a link to
it.

~~~
trishume
None of it is open source because it was done for a school course that uses
the same project every term. The only use for having access to the code would
be cheating on that project. The school gets mad when people publish the
source for their school projects, especially when it isn't otherwise useful,
that's also why my ray tracer isn't open source.

~~~
empath75
I would not agree that that is the only use of that code. I’ve found a lot of
school projects on GitHub when googling how to do stuff as a professional
programmer and have found them useful quite often.

------
carapace
Ooo lolly! I want to do this in Prolog!

The bit about "no parsing helpers even if they’re in the standard library"
makes me wonder about using DCGs, but if disallowed re-implementation would be
straightforward and add only a small constant to the code volume.

I would like to see metrics on speed of compilation and size of resulting asm.

------
kris-s
Great writeup, I'm very curious about the performance between the compilers.

------
karmakaze
I've condensed the relative source code sizes and notes for quick ref:

    
    
      Rust_1  1.0 using recursive descent, visitor
      Haskell 1.0-1.6x depending on how you count for interesting reasons
      C++     1.4x for mundane reasons
      Python  0.5x fancy metaprogramming, dynamic typing, single author
      Rust_2  3x different design decisions
      Scala   0.7x LR table generator, online Java grammar
      OCaml   1.0-1.6x depending on how you count, similar to Haskell
    

> _AST visitors and recursive descent [...] weren’t taught in the course_

~~~
karmakaze
Looking at this, my takeaways are different:

    
    
      Similar implementations in Rust(1) and C++ are roughly the same
      Similar implementations in Haskell and OCaml are roughly the same
      A similar implementation in Rust(2) to one in Haskell/OCaml is roughly 2x-3x
      Don't underestimate dynamic typing, metaprogramming, and individual effectiveness
    

But for real world, long lived projects I still promote static typing with the
exception of startup development (MVP/product validation, early iterations) of
new product ideas.

------
dpc_pw
Interesting, but nothing to make any conclusions from. Too many variables to
be of any use. Almost everything was different about these projects (different
parsers, error handling, features, developer skill, team size).

------
galaxyLogic
Prolog, and Definite-Clause-Grammars
[https://en.wikipedia.org/wiki/Definite_clause_grammar](https://en.wikipedia.org/wiki/Definite_clause_grammar)

------
huxingyi
The actual effort of writing one line of code in the specific language doesn’t
mention in the article. Write one line of rust is extremely difficult than one
line of Python, or even C++.

------
PaulAJ
I'm not clear whether the SLOC counts are for the application itself or
including their test harnesses. It would be good to have a table of results
with this data.

------
choeger
First of all: amazing project. Kudos to your professor. Who is it, btw?

Second: intriguing how important meta programming becomes once you nailed the
problem domain.

------
iLemming
It's too bad Clojure was not considered. I think it would be the most concise
codebase and the most easiest to reason about.

------
namelosw
I'm surprised that OCaml takes 1.0-1.6x, given originally Rust compiler was
prototyped in OCaml...

~~~
dean177
It mentions that this is almost entirely due to interface files on ocaml’s
part. They unfortunately introduce a fair amount of duplication.

~~~
vphantom
Luckily, they don't bring with them a lot of cognitive overhead for the
developer, so their presence masks the expressiveness of OCaml in LOC stats
IMHO. I use interface files to make public signatures explicit, abstract away
some types and write thorough doc-comments. I'd be tempted to exclude them
from such a comparison to better relate code size with programmer efficiency,
especially when comparing to a language like Python.

On the other hand, in languages like C we couldn't exclude header files
because they include macros. (Although I suppose a fancier comparison could
count macros and exclude function signatures.)

------
Scarbutt
How much time was given for this task?

------
tomerbd
and how much time does it take to __maintain this code __?

------
pwpwp
Is the code online somewhere?

------
zerr
No AWK and Ada this time? :)

------
foobar_
Rust seems so ugly ... makes Perl line noise look like paradise. I guess it's
that time of the year ... when I have to learn an esoteric language to keep my
sanity.

------
mruts
I wonder how Racket would compare. I’ve written some DSLs at an old job for
Racket and found the libraries for language design almost unreasonably
powerful (and maybe unreasonably complicated as well). The dynamic typing
would also help in reducing the amount of code.

~~~
trishume
Yah I figure it would be like Python but better if you were allowed to use the
language design libraries. However the course specifically forbids standard
library components that are designed for implementing languages so that you
have to write the parser and things yourself.

~~~
neilv
But you could use Racket if you wrote your own parser generator macro as part
of the project's code?

And would you be allowed to use Racket's existing facilities for representing
and implementing macros?

(You might want to use the macro stuff for your AST, IR, and transformations.
Though you could still win with normal data types, but the macro stuff can
help you win even more.)

------
sonnyblarney
"Their project was 17,211 raw lines, 15k source lines, and 637kb"

So a couple of students cranked out 15K lines of code, in basically a subset
of their study time, for a single class?

I don't doubt they did it, what I doubt is the quality of what they wrote.
Because it's 'just a project' it doesn't have to be fully debugged ...

But that is a vast amount of code to hand-write over a short period of time.

At that level, I think it's basically just 'code like the wind' with not much
consideration for whether it works, architecture etc. etc. - which in a way
might put the conclusions of the 'mini study' at odds.

At 15K LOC dashed out very quickly ... these are not products, they're
comparing 'rapidly written out lines of code', which is something else
entirely from working code.

Also, given that everyone is in Uni, and likely may not have been exposed to
proper idiomatic code for the language ... this presents another issue.
Ideally we'd want to compare reasonably idiomatic code for one language, to
another.

Kudos to the author for this, but we should be aware of some caveats.

~~~
trishume
Note that project is the outlier for how much code it contained. That team did
have the most trouble getting things done on time and fixing all the bugs, so
they passed the least tests, as one would expect given how much code their
design choices required them to write. I think the causality goes from their
design to the line count, not from the line count to worse design, they had to
write lots of code to implement their more abstract design choices. I looked
at their code and didn't see any obvious deficiencies in anything other than
the design.

Also note that UWaterloo's CS and SE programs are different than other
universities. Everyone on all teams had at least two years of full time work
experience at 6 internships. The people I talked to were also programming
enthusiasts who read online a lot. The only teams that may not have written
the most idiomatic code are the person who used Python alone, and the Haskell
team because I've heard idiomatic high-end Haskell involves an insane amount
of knowledge of abstractions like lens, way more than all the other languages.

This compilers class is also well known for requiring lots of work, so
students often arrange their schedules to have a low load from other classes
while taking it, which is possible since the CS program requirements have a
lot of flexibility.

~~~
shaklee3
Two years of work experience isn't going to make them experts in a language,
especially during an internship. Languages like c++ would be a minimum of 5
years of professional programming to be competent I've found.

------
akhilcacharya
Loved this part and thought it was hysterical -

> Since my team had all interned at Jane Street the other language we
> considered using was OCaml, we decided on Rust but I was curious about how
> OCaml might have turned out so I talked to someone else I knew had interned
> at Jane Street and they indeed did their compiler in OCaml with two other
> former Jane Street interns.

Relatedly this is why I get annoyed at people that don’t believe 10x or Nx
developers exist - they absolutely do!

~~~
ncmncm
In the 90's, the Swedish Ericsson and German Siemens companies participated in
a joint (crash) project. Each sent 250 engineers. After six months, they
delivered. A friend counted up lines of code written by each engineer, at the
end. Fully half the code in the final product was written by one person: N
lines by him, N lines by the other 499 people.

He was a lead programmer, who issued two-week assignments to other engineers;
if one wasn't done come Friday, he would do it himself on Saturday.

He doesn't consider himself especially fast, because he knows someone else who
codes ten times as fast, and wears out two keyboards per year.

This project was also interesting because they took blood samples, by which
they got objective measurements of stress. Everyone's stress level increased
right up to the deadline -- except his. And their stress levels did not start
falling until many months after.

~~~
madez
I'd like to read up on this, so I'd be happy if you could provide sources.

~~~
ncmncm
Sorry, personal communication.

------
nautilus12
I can hear all the Haskell fanboys screaming that they didn't do Haskell right
because a kleisli arrow didn't appear and ascend them into monadic heaven.

~~~
aetherspawn
To be fair, I don’t think any Haskeller would actually write this project
without attoparsec and maybe lens by choice. They’re basically base libraries
for this sort of thing.

~~~
nybble41
Agreed. The rule about not using any library that doesn't ship with the
compiler creates a heavy bias in favor of "batteries included" languages like
Python. I understand not allowing _parsing_ libraries, given the nature of the
assignment, but if even widely used libraries like lens are off-limits the
result is not going to resemble idiomatic Haskell. As a matter of basic
fairness, if some library or built-in language feature is to be permitted for
one implementation then analogous libraries should be permitted for all the
other implementations regardless of whether they happen to ship with the
compiler.

The eval function leveraged heavily by the Python version should probably also
have been off-limits, for the same reason that parsing libraries were
prohibited. Using eval amounts to embedding an existing fully-developed parser
and runtime environment into the project as a library.

~~~
dymk
I think a much more interesting comparison would be to ask an expert in each
one of these languages to implement a "production" ready version of the
project - so allowing popular, stable, community-accepted libraries to be
used.

Then we'd see what truly idiomatic solutions to this look like.

------
jwm4
Prohibition was an extremist political movement. Widespread public rejection
of it, in the form of stills, speakeasy and other means proved it to be a law
without public sopport, resulting in its repeal within 10 years - an absolute
public rejection of a limit unsought by the majority. OP pov has zero basis in
fact.

~~~
pkilgore
Am I.... Missing something?

