
Benchmarking D vs Go vs Erlang vs C for MQTT broker  - andralex
https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
======
rdtsc
> Go 1.2 produces a binary that performs on the order of 10%-15% faster than
> the numbers above, which might mean equivalent performance to the Erlang
> implementation.

Huh, impressive for Erlang, beating a newer, statically compiled language,
while being interpreted and with dynamic types.

~~~
wcummings
Erlang's VM is very sophisticated and mature, it is at least years ahead of
Go, imo. For example, compare Erlang's pre-emptive scheduling [1] to
"goroutines".

[1] [http://jlouisramblings.blogspot.com/2013/01/how-erlang-
does-...](http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-
scheduling.html)

~~~
chimeracoder
> it is at least years ahead of Go,

Yes, Erlang has admittedly had decades to improve the scheduler (versus a
couple of years for the gc compiler in Go), so this is quite literally true.

However, you should know that goroutines _are_ pre-emptively scheduled now;
the "scare quotes in your comment seem to imply that they aren't:
[http://golang.org/doc/go1.2#preemption](http://golang.org/doc/go1.2#preemption)

~~~
shanemhansen
I'm a go fanboi, but to be fair, goroutines are now more preemptive, but not
completely.

In practice that means that if you call a function somewhere in your infinite
loop, you'll get preempted. It's still possible for a single goroutine to
steal an os thread.

Practically speaking though, I feel like nobody should be scared away from go
due to it's lack of full pre-emption. Seriously, don't write an infinite loop
with no function calls.

------
f2f
i see no mention of "concurrency" in that article. i see that the go programs
make heavy use of goroutines so why were they run in "single threaded" mode?
just testing their mqttsrv with various levels of GOMAXPROCS rewards with
immediate benefits. here are some numbers for pingtest and loadtest in Go for
varying levels of GOMAXPROCS for them and the server:

(pingtest/mqtt gomaxprocs) vs pingtest, reporting "elapsed time":

    
    
      1x1  5.8s
      2x1  6.0s
      4x1  6.0s
      8x1  6.1s
      16x1 6.1s
    
      1x2  5.5s
      2x2  3.5s
      4x2  3.4s
      8x2  3.5s
      16x2 3.5s
    
      1x4  5.4s
      2x4  3.6s
      4x4  3.3s
      8x4  3.4s
      16x4 3.7s
    

same thing but for loadtest, reported value is "messages/sec":

    
    
      1x1  75521
      2x1  76523
      4x1  77477
      8x1  74613
      16x1 73678
    
      1x2   71439
      2x2  100571
      4x2   93426
      8x2  108938
      16x2  97593
    
      1x4   71390
      2x4   87426
      4x4   98746
      8x4   99320
      16x4 119692
    
      1x8   64413
      2x8   87445
      4x8  102953
      8x8  120720
      16x8 106842
    

(note that the loadtest doesn't look like it takes long enough to be
meaningful in its default)

as always with benchmarks, caveat emptor, but in my opinion go does just fine.

~~~
yelnatz
Are your numbers in line with the blog's table?

So you get those instead of these?

Loadtest:

Go: 90.9 +/\- 11 100.1 +/\- 0.1 99.3 +/\- 0.2 98.8 +/\- 0.3

If not can you make it relevant to the table so people can compare and
contrast?

~~~
f2f
loadtest for 100 500 750 1000, X*Y below means X = loadtest GOMAXPROCS and Y =
mqttsrv GOMAXPROCS:

    
    
      1x1 80735 67696 67068 66585
      2x1 64380 76531 31346 43268
      4x1 77613 73794 72759 43263
      8x1 77813 64197 32899 46960
    
      1x2 71641 72521 72715 72695
      2x2 103711 102944 104210 39619
      4x2 73171 23803 90537 46204
      8x2 96624 49698 33714 45729
    
      1x4 69418 74088 34364 38935
      2x4 88805 102847 113740 108829
      4x4 102406 108110 34672 120161
      8x4 109022 124726 121642 46765
    
      1x8 62336 68238 71297 68754 
      2x8 104833 115276 118170 44617
      4x8 122292 126075 126101 121306
      8x8 108826 125098 124765 47713
    

edit: s/pingtest/loadtest/ in the explanation above.

~~~
yelnatz
Still not following. For Loadtest:

D: -- [121.7 +/\- 1.5] -- [166.9 +/\- 1.5] -- [171.1 +/\- 3.3] -- [167.9 +/\-
1.3]

C: -- [106.1 +/\- 0.8] -- [122.4 +/\- 0.4] -- [95.2 +/\- 1.3] -- [74.7 +/\-
0.4]

E: -- [104.1 +/\- 2.2] -- [124.2 +/\- 5.9] -- [117.6 +/\- 4.6 -- [117.7 +/\-
3.2]

G: ---- [90.9 +/\- 11] -- [100.1 +/\- 0.1] -- [99.3 +/\- 0.2] -- [98.8 +/\-
0.3]

G(2): ? ? ? ? ?

G(4): ? ? ? ? ?

G(8): ? ? ? ? ?

------
colanderman
"the Erlang version I’m not sure"

At least try to find out if it's HiPE (native compilation) or not?

------
lpgauth
"an excuse to learn Erlang, and wrote his own implementation"

Why even included Erlang if you're not going to share the source...

~~~
atilaneves
He wasn't in the office. I'm going to ask him about it today. Like it says at
the end of the blog post, if he puts it up on bitbucket I'll include the link.

------
ader277
"the Go implementation was executed with go run ". I think it would be much
more appropriate to build the GO script with "go build" and use the resultant
binary. Because the D program was compiled.

~~~
colanderman
[http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript](http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript)

"Run _compiles_ and runs the main package comprising the named Go source
files."

~~~
he_the_great
So he should have run the D program with dmd -run :D

~~~
jeremiep
or #!/usr/bin/rdmd

------
chrismorgan
I found this article a little hard to read;

\- #555 on #fff is distinctly too light for a web page's text color (I
generally recommend around #202020);

\- the font-size of 13px is too small—remove it and let the default of 16px
show.

------
karcass
"Go is an opinionated language, which would be fine if I agreed with its
creators’ opinions." Spot on.

------
eerr
Would have been cool to see how many lines of code each implementation took
including the erlang one.

~~~
atilaneves
It's mentioned. It's not easy to measure but they're all in the 800-1000 lines
region.

------
drakaal
Summary:

I hate Go, So Go Lost this race.

I love D, So it won or tied with C which I don't like but everyone knows is
fast.

I wanted to learn Erlang, so I made sure it did better than Go.

I didn't share my SourceCode so you can't see if I was stupid or cheated.

Did I miss anything?

~~~
andralex
Quite a few things.

The Go version was written by a person enthusiastic and presumably competent
in Go. See their perspective here where he concedes defeat on the performance
front: [http://blog.nella.org/mqtt-code-golf/](http://blog.nella.org/mqtt-
code-golf/)

Adding C as a baseline is entirely appropriate.

The source code for C, Go, and D is available. The Erlang version may or may
not at the discretion of its author.

Probably a more sensible hypothesis is that you like Go and saw a result that
is unfavorable to it, so you had an emotional response.

~~~
azth
> As usual, Go programs compare quite favorably to C programs in terms of
> memory safety...

That's not saying much :)

------
codygman
> Go implementation was executed with go run

Why didn't they just use "go build program.go; ./program"?

If you're compiling the other programs it doesn't seem fair to use the
interpreter for go.

~~~
colanderman
I thought "go run" did exactly that (compile then run)?

~~~
codygman
Perhaps I'm mistaken, looking at the source it does seem to do just that:
[http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript](http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript)

------
chrismorgan
I'd rather like to see Rust in here.

~~~
atilaneves
Me too, but I've decided to not even look at Rust until it hits 1.0.

~~~
kibwen
Good, because we're certainly not ready to compete in serious benchmarks yet.
:)

------
dscrd
The results of the latency benchmark say that a larger latency is better. Is
that a mistake?

~~~
atilaneves
No, it was measured in thousands of messages per second.

------
emrekzd
While your "benchmark" (if we should call it that way) obviously make no
sense, I would suggest you guys look into how concurrency and parallelism are
different concepts.

------
static_typed
The key takeaway in the benchmarks seem is that Go is significantly slower
than the rest, much much slower.

Apart from the regular 30-sec interval postings on HN extolling the virtues of
Go, it seems to me, people should rekindle an interest, or discover a new
interest in D, C or Erlang where performance is a consideration, and maybe Go
where a need to feel like part of the post-Ruby crowd.

~~~
CoffeeDregs

        >Go is significantly slower than the rest, much much slower.
    

Everyone knows that Go is a new-ish language and has room to improve on
performance. That said, Go seems perfectly fast since all of the languages
seem to perform fairly well.

Further, were Go "much much slower", CloudFlare, IronIO, Google, etc would
probably not be using it...

    
    
        >discover a new interest in D, C or Erlang
        >where performance is a consideration
    

Erlang on The Game:
[http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...](http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=hipe&data=u64q)

Go looks to be about ~50% the performance of C++, but, from lots of peoples'
comments/writings, is much nicer to write so they use it when productivity is
more important than absolute performance.

Note: I'm interested in Go (as I am about lots of languages), but am not a
user of it right now.

~~~
colanderman
Ya, Erlang doesn't do well on the Benchmarks Game because the Benchmarks Game
doesn't measure communication, which is Erlang's strong point. No-one writes
algorithms like those in the Game in Erlang; you code them as a server in C or
Java or anything and farm work out to them from Erlang.

------
Thaxll
I don't understand those benchmarks, 'insert any languages' can achieve
performances than 99% of companies don't even need. And for the other 1%
you're probably smarter than those posts on HN.

