
tfbvis: Visualizing TechEmpower benchmarks beyond requests per second - emmanueloga_
https://ajdust.github.io/tfbvis/
======
emmanueloga_
Found tfbvis because I wanted to see more than RPS: specially memory
footprint. The headers sortable and can be filtered.

The most memory efficient results belong to C, C++, Rust and Go, usually using
around 1,1.5GB of RAM max. Java programs in the same ballpark RPS use maybe up
to 4 times that much.

Not sure how much weight to put in these benchmarks though. After all, they
measure performance under stress... How many web apps receive > 200,000 RPS
during sustained time?

Is there a clever way to extrapolate an estimation of RAM usage under normal
load?. Would be cool to see how the memory usage grows as the RPS grow, for
different platforms, but I don't think the raw data has that info.

~~~
emmanueloga_
Hmm was just talking with the author of jooby and its benchmark, he pointed
out he just copied one of the docker configs [1] from someone else, which
explains why a lot of the benchmarks have similar memory profiles.

The jooby author also said he deployed this particular framework with as
little as 512MB, which sounds reasonable. So a bunch of JVM implementations
may be using "java -Xms4g -Xmx4g", so perhaps the results are not so useful,
at least when it comes to the JVM.

1:
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/jooby/jooby.dockerfile#L12)

------
emmanueloga_
TechEmpower wants to "provide representative performance measures across a
wide field of web application frameworks." Do you think it succeeded in this
regard? It seems like the results are pretty well received by the community in
general, or at least I haven't seen a lot of people disregard these benchmarks
or give them the CLBG treatment [1].

1: [https://benchmarksgame-
team.pages.debian.net/benchmarksgame/...](https://benchmarksgame-
team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html)

~~~
igouy
> at least I haven't seen a lot of people disregard these benchmarks

[https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_ph...](https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_phoenix_compare_to_fastest_frameworks/g3mdkru/)

> the CLBG treatment

Please say what you mean ?

~~~
emmanueloga_
The "The Computer Language Benchmarks Game", if I remember it right, did not
start with that title, and the very authors have progressively made more and
more emphasis in the fact that the programs are "toy benchmarks" and not
useful to compare which language is "fastest".

TechEmpower benchmarks, on the other hand, wants to "provide representative
performance measures across a wide field of web application frameworks" (and
proglang is a prominent part of the framework) so it seems away from the CLBG
stance.

~~~
igouy
> more and more emphasis

Well, more and more literal / less and less sarcastic.

Consider —

"Some people have suggested that I summarize the results, or declare a winner.
In order to satisfy this request, I have come up with a unique and subtle
quantification system to score languages on their overall performance, which I
call the Completely Random and Arbitrary Point System!, or CRAPS![TM], for
short.

… you can link this page (with properly selected CGI parameters) from your
language advocacy page as the final proof of your language's supremacy! Think
of the glory."

[https://web.archive.org/web/20010302143046/http://www.bagley...](https://web.archive.org/web/20010302143046/http://www.bagley.org/~doug/shootout/craps.shtml)

