
Framework Benchmarks Round 19 - bhauer
https://www.techempower.com/blog/2020/05/28/framework-benchmarks-round-19/
======
ssijak
People like bashing java, but java based frameworks (and other JVM based ones)
are always at the top. Solid foundations, great virtual machine, epic tooling,
great and reliable libraries (which do not change every 3 months like in
node), and in the recent time much faster evolution of the language. And if
you do not like java as a language (which has its downfalls and problems) you
can use Kotlin or Scala, or even Haskell on the JVM if you fancy so.

Funny to see rails, django, laravel and other most popular frameworks from
dynamic languages at the bottom of the list with such bad performance. Even
the Java heavy weight Spring framework is much closer to the top.

Not to mention that JVM can support all kinds of concurrent, parallelism and
scaling models.

~~~
hombre_fatal
I'd say performance in this benchmark is inversely related with how nice it is
to express business logic in the respective language/framework, and that's a
pretty big deal.

For example, [https://github.com/an-tao/drogon](https://github.com/an-
tao/drogon) (benchmark winner) or [https://oci-
pronghorn.gitbook.io/greenlightning/](https://oci-
pronghorn.gitbook.io/greenlightning/) just aren't pleasant to work with. For
example, compare Green Lightning with
[http://sparkjava.com/](http://sparkjava.com/) and look at how much slower
Spark is while clearly being nicer to use.

I worked with Netty at work for two years and it was a dream to use something
like Express or Flask on the side, not having to worry sharing/disposing
ByteBufs, not dealing with the weird in-channel/out-channel abstraction (vs
simple middleware), not investigating how async-/thread-safe a given library
was, etc.

It just all goes back to trade-offs. You're going to have different
considerations if you're building a Caddy competitor vs a complex game server.
Another consideration is how horizontally scalable you can be which is going
to be different for a forum vs a websocket server.

~~~
ssijak
Netty is network IO framework and is almost never used as is to develop web
applications. You usually plug it in your framework or http library which
abstract over it (or Jetty, or Apache, or Undertow, or ...).

Look at Composite Scores in Techempower benchmarks. Jooby, Micronaut, Ktor,
Dropwizard, Http4k.. are all nice to work with and are all pluggable with
different network IO frameworks like Netty for example. And they are not
barebones http layers only optimised for speed, they come with a bit of
batteries included but not that much that you battle the framework. They are
less magic and more composable with other libraries where you can more easily
know what is happening. Similar to Express and Flask. But even a heavyweight
Spring framework is at the top. Flask is stuck at the bottom. Even express has
bad results.

And, this is a personal preference, but having built production apps in both
dynamically typed and statically typed languages, I prefer the latter for
various reasons. Using Kotlin to write backend apps is really enjoyable for
me.

------
tfparty2
The addition of the "realistic" vs "synthetic" benchmarks is very welcome,
with "realistic" being the default. Hopefully this could help get things to a
better state.

Still questionable as to what is allowed in "realistic" though. Aspnet core
using a "rawdb" adapter doesn't seem like what I would interperet as
"realistic".

Perhaps there needs to be a "marketed" category, for use as the docs describe.
For aspnet core, this would include using entity framework.

Not much point in splitting the categories if it isn't a uniformly split
thing.

It'd be nice to see benchmarks where: • Everything was implemented as
documented in the framework's tutorials. • Realistic things were included end-
to-end (payload validation, csrf (if applicable), etc).

Still seems like a whole basket of fruit comparisons here.

~~~
nodamage
> Aspnet core using a "rawdb" adapter doesn't seem like what I would
> interperet as "realistic".

Why not? I don't think using raw SQL instead of an ORM is particularly
uncommon.

The "platform" benchmarks (aspcore-rhtx-pg, aspcore-ado-pg) that appear to
skip the ASP.NET Core framework entirely and write raw HTTP responses are
definitely not realistic though. I'm going to assume that these are mislabeled
because this filter is new and not all the benchmarks have been updated yet.

~~~
tfparty2
I'll agree that raw database queries aren't unheard of, but EF and Dapper are
much more suggested (and IMO "realistic") database paths for "asp.net core"

additionally, I'm not quite sure that this is the recommended way to return
database results:
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/Benchmarks/Middleware/SingleQueryRawMiddleware.cs#L33)

What I believe is more "realistic" is to:

• Use the framework as described: [https://docs.microsoft.com/en-
us/aspnet/core/tutorials/first...](https://docs.microsoft.com/en-
us/aspnet/core/tutorials/first-web-api?view=aspnetcore-3.1&tabs=visual-
studio#return-values)

To be fair, I'm picking on aspnet core because I like it so much (via F#). The
framework is great, but this gaming of public benchmarks (techempower,
benchmarksgame) by "breaking the same rules as everyone else" rather than
investing time into making these results more useful is disappointing.

~~~
darrenkopp
I'd say I do about 40% raw ado and 60% ORM overall. I think raw ado is
completely valid in a performance benchmark because you usually step down to
raw ado for that very reason: optimizing performance.

------
minimaxir
Interesting that drogon is relatively new (first release a year ago; new to
Round 19) and is utterly destroying the benchmarks. [https://github.com/an-
tao/drogon](https://github.com/an-tao/drogon)

Show HN:
[https://news.ycombinator.com/item?id=19427332](https://news.ycombinator.com/item?id=19427332)

~~~
marricks
I heard some old guard person at Mozilla talk about these benchmarks last year
when Rust's ACTIX was on top, something like "whatever is the latest framework
can take advantage of all the latest methods and libraries to become the
fastest", at the time I took it to mean ACTIX being the fastest has more to do
with it being new then it being written in rust.

Given a newcomer is at the top again that seems to ring true? Does seem odd
older frameworks wouldn't keep up with that game, but perhaps once they get
the publicity and user base they can worry more about features than having the
highest benchmark?

~~~
Klonoar
It's worth noting that actix-web is at the top by doing some insane tricks,
which you wouldn't normally do in writing a web application.

e.g at one point they used Askama as their templating engine (may have
switched), which isn't really fair as using Askama means you force a recompile
of your project for each template change - and I can tell you from experience
that your compile times will shoot through the roof as well.

Nobody is going to bother doing that.

I say all of this as an actix-web user. It's my go-to. I just view these
benchmarks as "if I ever needed to, I know I can wrangle performance out of
it... but I won't assume it by default".

~~~
chrismorgan
I don’t think recompiling for template changes is as unreasonable as you
depict. It’s a standard technique in various languages, though I acknowledge
that they’re mostly not languages that compile to native code, but rather an
IL (e.g. ASP.NET with Razor) where the compilation time is somewhat lower.

For a language like Rust, compilation times and tooling support mean that the
technique is perhaps not ready for prime time, but it’s fundamentally a
perfectly sound technique, and can produce _wondrous_ developer ergonomics
(again I cite Razor).

There’s something quite incongruous about using dynamically-typed and
basically uncheckable template language when you’re doing the rest of the code
in Rust.

~~~
Klonoar
I didn't say that recompiling template changes is bad - e.g, you can
accomplish the same result with much faster turnaround time using Tera, and
for most projects the speed is in no way a bottleneck.

------
kumarvvr
I wish there was an additional set of data that uses this set, and then
computes the average cost of using each framework on various cloud providers.

Its amazing to see python based stuff so low in the benchmarks, yet so high in
general popularity.

In cloud based servers, where requests per second directly translates to
inverse USD, its imperative costs are incorporated into the selection process
for the software stack.

~~~
p_l
There are some writings by people from Stackoverflow on how much they saved
going with native COMPILING NET stack.

------
gremlinsinc
Damn, I just settled on a high performance framework for my next reddit killer
meets slack clone + SaaS, and now I need to rethink that and add more months
to actually launching, on the off-chance my app takes off and needs to quickly
support millions of connections at once.

~~~
deathtrader666
Stop reading my mind /s

~~~
gremlinsinc
haha, I do this every few months on a side project, then usually fall back to
laravel+vue just cause it's what I know. Though, at least now swoole gives me
a little more 'oomph' w/ php.

------
jph
Rocket (Rust) is the framework on my new project, and the benchmarks clearly
show that speed improvements would be very welcome. If anyone here wants to
work on this area, please contact me and I'll be glad to boost the Rocket
numbers together.

~~~
plainOldText
I actually benchmarked it yesterday and I would like to point out that the
_async_ branch of Rocket will get you improved performance, as the current
stable implementation v0.4.4 appears to be synchronous. I think Rocket v0.5
will change that.

~~~
lordnaikon
What was the result of your benchmark – just a ballpark number would be great
on what to expect from the async version.

~~~
plainOldText
Absolute numbers vary with each machine, to the point they’re irrelevant in
isolation.

However, in terms of relative performance the async version was 10x to 20x
better. Bear in mind it was a simple hello world benchmark. I also disabled
keep-alive. Reusing a connection gets you even better performance.

Rocket is a well designed framework, so I’d expect it to perform similarly to
other rust frameworks once the async version hits master.

------
viig99
Modern c++ is surprisingly easy to pickup especially for developers coming
from other languages, very easy to focus on the good parts and totally
disregard the messy bits, working with older libraries is still a pain though.

------
Spiritus
What does the new "<T>" icon represent on the graphs (only present on some of
the frameworks)?

~~~
Spiritus
Answering my own question:

> TPR-3 is a composite hardware environment score for a three-machine
> configuration, derived from all test types for TPR-tagged <T> frameworks.
> (More about TPR frameworks and TPR-1 versus TPR-3)[1]

[1]
[https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Tech...](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/TechEmpower-
Performance-Rating-\(TPR\)#two-flavors-tpr-1-and-tpr-3)

~~~
bhauer
Indeed, this is related to our (experimental) addition of a performance rating
for the hardware environment we're testing on. This is rendered as the big
number on the composite scores page. Theoretically, our benchmarks covering a
diverse spectrum of technologies could be useful for measuring hardware and
network performance. But a full run of all test implementations takes 111
hours, and I suspect few people who want to measure hardware performance would
be willing to wait that long. So we selected a subset to use for hardware
measurement, hence the icon you see.

------
barbarbar
Just came to say that it is an amazing and extremely thorough job they have
done in all these runs. And also still mind-blowing how much work the fastest
frameworks can do.

------
matt42
I wrote a new C++ framework (lithium) last december ranked #4 in this
benchmark and while this benchmark does a great job comparing raw performances
of many frameworks on a set of test, here are my thoughts: 1/ Best Frameworks
are all using Non blocking network IO (based on linux epoll or equivalents) 2/
Very few companies need such performances. Don't look for those number if you
have less than 10M visitor/months 3/ Tuning the number of sql connections can
do x5 on performances and all framework did not tune this parameter. 4/ Even
if you a low ranked framework, it may not be the bottleneck of your stack so
changing to a top performer may not change your thoughput overall.

------
bsaul
I was surprised to see the performance of php frameworks. Has there been
something special happening with this language ? It looks completely on top of
all the other dynamic languages on most benchmarks, which i really don’t
understand.

~~~
jorams
It's important to note that all the PHP results that are high up on the lists
are using Workerman[1] with the event module, Swoole[2] or ngx_php[3]. They
don't use the traditional PHP runtime model, which boots up and tears down the
world for every request. Global variables, which are no problem in "normal"
PHP code, are going to cause serious trouble in these environments. They are
essentially gluing a very fast async IO framework to a fairly minimal amount
of PHP application code.

The vast, vast majority of PHP code you will find in the wild is running a
more traditional stack, and those results are much lower down the list.

The results are still impressive though. PHP is very good at gluing fast C
code together without slowing it down much. Especially Workerman is
impressive, since it appears to still run a significant amount of PHP code.

[1]:
[https://github.com/walkor/Workerman](https://github.com/walkor/Workerman)
[2]: [https://www.swoole.co.uk/](https://www.swoole.co.uk/) [3]:
[https://github.com/rryqszq4/ngx_php7](https://github.com/rryqszq4/ngx_php7)

~~~
gremlinsinc
Part of that's also that php 7.4 is much faster, so of course that alone would
improve it more, I wonder what JIT will do when 8.0 launches in December.

I use swoole w/ laravel, still trying to figure out best ways to milk it for
more performance, and also grok the way it handles async and passes variables
around using swoole_Table.

~~~
melchebo
For the moment (well, february) the improvement with PHP 8.0 JIT is minimal:
[https://www.phoronix.com/scan.php?page=news_item&px=PHP-8.0-...](https://www.phoronix.com/scan.php?page=news_item&px=PHP-8.0-Performance-
Bench-2020)

But usually a JIT is made in a way that you can then move forward with your
optimizations in small tested steps. So it makes sense for JITv1 to have a
very similar performance as the non-JITed version. It probably generates
highly familiar code. First make it do the same, then optimize.

------
dom96
If you're thinking of evaluating language performance by looking at these
benchmarks then think again. A lot of the benchmarks here (h2o.cr and pico.v
being prime examples) are thin wrappers over libraries written in C/C++.

~~~
giulianob
That's what's actually fairly impressive about C# being #1 on plaintext. Both
ASP.NET and Kestrel (the frontend server) are both fully in managed code.

~~~
jonathanoliver
Various teams at Microsoft should be congratulated for their efforts. A few
years back, ASP.NET was last (or near last) on this benchmark for simple,
plaintext requests. They've come a long way and put some serious engineering
into their system.

~~~
sebastienros
You can't even imagine. A goal has been set by the management to get specific
results on some benchmarks, by improving the framework. We have weekly
meetings to follow up on improvements and next tasks to keep track on these
goals. And these meetings involve super qualified engineers and also brilliant
open source contributors. I can t describe the level of technicity at each of
these meetings. The impact on the framework is tremendous, you can follow it
as our charts are public. And the goals are already reached, which is not yet
reflected on TechEmpower as we only submit release bits. This is good for PR,
but as much for also for any developper using dotnet.

~~~
moi2388
Link to those charts please? :)

~~~
giulianob
[https://msit.powerbi.com/view?r=eyJrIjoiYTZjMTk3YjEtMzQ3Yi00...](https://msit.powerbi.com/view?r=eyJrIjoiYTZjMTk3YjEtMzQ3Yi00NTI5LTg5ZDItNmUyMGRlOTkwMGRlIiwidCI6IjcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0NyIsImMiOjV9)

------
Zingler
A fun diversion, but largely meaningless.

