
Lithium ranked the highest performance web application platform - matt42
https://www.techempower.com/benchmarks/#section=test&runid=57b25c85-082a-4013-b572-b0939006eaff&hw=ph&test=composite&a=2
======
bhauer
Brian from TechEmpower here. I don't want to take away from these results, but
I do want to provide an important piece of context. This link is to a
rendering of data from a continuous run [1] that hasn't received the type of
sanity checking we do for an official TFB round. You can tell from the title
("Test") and the UUID of the run visible in the gray box under the navigation.
Based on this link's attention, we'll make a more prominent warning for
renderings of continuous runs so that they are more obvious to readers not
familiar with the project.

We execute runs like this continuously to allow maintainers of test
implementations to observe the results of their contributions. Given the
performance seen here, it is _very likely_ that Lithium will be well ranked in
the next official round. But we feel the ranking seen in a continuous run such
as this should be taken with a grain of salt until that next official round is
available.

[1] [https://tfb-
status.techempower.com/results/57b25c85-082a-401...](https://tfb-
status.techempower.com/results/57b25c85-082a-4013-b572-b0939006eaff)

~~~
AtlasBarfed
Is that why it didn't vary when I selected cloud vs physical?

~~~
matt42
Yes, this is only physical. The cloud env (Azure) only run once every 1 or 2
months I think. On the official rounds they do both.

------
The_rationalist
I had investigated what explained the huge gap between the few fastest
frameworks vs the rest. The answer is a deceptive one, they didn't achieve
revolutionary optimizations. The thing is: on many of those benchmarcks, the
bottleneck obviously is the DB. The ability to do DB queries asynchronously
and with batching is the differentiating factor. Only postrgresql can support
such a feature but you need support on the postrgresql client too. The
official C postrgresql client used everywhere does not support said feature
except with a patch from 2016. Yes the secret of drogon (and probably of
Lithium) is that they use a fork of libpq from 2016 because upstream can't
agree on merging the patch and nobody is working on upstreaming it. Actix web
benefited from the feature because their client tokio-postgres is a
reimplementation and does not use libpq.

The industry grade server ecosystem that is the JVM use the jdbc which sadly
has a blocking socket thus not allowing asynchronicty. But when loom arrive
every jdbc existing code will magically, automatically become truly
asynchronous such spring should come on the top 4 place.

There is also a wrapper of the jdbc through kotlin coroutines and there are
reactive jdbc implementations such as R2DBC. It is unclear as of today if such
solution enable postrgresql async queries and batch processing. It seems that
nobody has tried those on TechEmpowerUp which is sad. Finally one could use
libpq over JNI. Edit: I have read that the next release of pgjdbc (43) will
switch from std socket to the NIO non blocking socket.

What should be heuristically the fastest HTTP framework (H2O, in C) has
refused to use the old libpq fork because the api is not stable and thus not
production grade.

~~~
matt42
Indeed, sql query batching is the key (and I personally think that the
techempower website should make it explicit), async communication with the db
and the http client is also the key.

FYI lithium is not using a libpq fork from 2016, it has been rebased on master
1 month ago. but yes Lithium (the sql-pipeline branch) is using it. (I mailed
the drogon maintainer to update it aswell).

About H2O, the non batched version of lithium is as fast (slighly faster on
some tests, slighly slower on others), while being much simpler to use
(implementation of TFB of H20 is 4400 lines vs 250 lines for lithium...)

~~~
The_rationalist
Interesting, thanks for the answer. What I would really like to see is the
obligation to have in the name "Lithium-with_batch" and a version "Lithium"
that would be without it. It would show the impact of other optimizations
which are currently hidden by the paradigm shift.

What would to you the other major optimizations?

~~~
matt42
I added -pipeline but I guess -with-batch is more explicit. I'll change it
then. But anyway the techempower team will eventually add a special tag for
batching. There is actually a version of lithium not using batching, check for
'lithium-postgres' in the benchmark tabs other than 'composite score'

Other big optimizations are \- non blocking communications between the
database and the framework \- non blocking communication between the http
client and the framework

This is for C++, for other slower interpreted languages: calling C bindings is
a big optimization aswell (and this is how some php frameworks get good
performances for example).

~~~
The_rationalist
Thanks. Asynchronocity is one thing but there is one thing that in theory can
achieve better performance than asynchronicity alone: I'm referring to the
reactive programming paradigm which benefit from the concept of backpressure.
If you've heard of it, do you think it could achieve even better performance?
[https://medium.com/@jayphelps/backpressure-explained-the-
flo...](https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-
through-software-2350b3e77ce7)

~~~
matt42
I had a look at the video, but I don't think there is backpresure in this
benchmark since there is only 512 connections max and each connections wait
for the server response before sending a new request. So in other words the
load generator never send more request/s that the server is able to handle.
(tell me if I missunderstood backpressure)

~~~
The_rationalist
Mmh then I wonder if the techempowerUp benchmark suite would benefit from a
new benchmarck that is backpressure sensitive, thus increasing its coverage of
real world workloads?

~~~
matt42
yes, it has been discused already to increase the number of connections. But
still, several thousands of simultaneous connections is still ok for the best
performing frameworks...

------
andybak
Is this a criteria that's important enough to warrant this much effort?

I can think of several metrics that would come first - even for sites with
fairly high performance requirements.

How often is the web framework genuinely the bottleneck (at least in a way
that's not easily fixable with caching or any other similar sledgehammer)?

~~~
Alex3917
Django is already _vastly_ faster than human perception. An entire roundtrip
to my Django app takes 70 - 110ms. The entire roundtrip is already faster than
human perception even in the worst case, and of that Django itself probably
contributes something like 2 or 3ms.

~~~
w_t_payne
110 ms is _not_ faster than human perception. It's over a tenth of a second,
and is very noticeable. The flicker fusion threshold for human vision lies in
the 60Hz-90Hz range, corresponding to between 11 and 17ms. If I recall
correctly, EEG experiments show differences up to around 100-120Hz, so on the
order of 8ms. In any case, network latency itself adds on the order of 10ms
per 1,000km of distance; 35ms or so for a London to New York round-trip, so
the point about framework speed being a bit of a moot point other than in
exceptionally high traffic scenarios still stands.

~~~
Alex3917
> 110 ms is not faster than human perception.

OK yes, it a literal sense you can see things that happen in less than 100ms.
What I mean is that you can't tell whether the nominal amount of flicker on
page load is from the time it takes to paint the DOM or from the network
request. Unless you were asking a professional front end developer, I don't
think most people would even have any intuition about what was happening.

------
matt42
Link to the lithium homepage:
[https://github.com/matt-42/lithium](https://github.com/matt-42/lithium)

~~~
ponytech
Thanks for the link.

A web search for "lithium web framework" gave [https://li3.me](https://li3.me)
as the first result, which is a PHP framework.

Why did you pick such a name if already taken by a similar product?

~~~
matt42
I was not aware of li3.me and when I realized it, I was too lazy to find a new
name..

------
supermatt
Incredibly frustrating the benchmarks not linking to the actual frameworks -
searching for "lithium web framework" gives me some PHP framework...

Edit: Comment lower down suggests:
[https://github.com/matt-42/lithium](https://github.com/matt-42/lithium) is
the correct one.

~~~
AtlasBarfed
Re: obscurity: This has been true for a long time. I remember seeing these a
long time ago and it wasn't until about 80% down the list that a "real"
framework showed up.

"Lies, Damn Lies, and Benchmarks" is somewhat true in this like any benchmark,
but as others have pointed out, important techniques gain exposure using this
benchmark over the years that trickle into mainstream use.

It would be nice if a summary of what techniques were used by new "frameworks"
to gain leadership. All too often discussions of this benchmark turn into
pissing matches by language zealots.

Actix's turn at the top was very interesting for the, uh, discussion, it
prompted in the rust community about unsafe, although it was stressful for the
original author.

In javaland, Vert.X has been high on the list for quite a while, but as I
understand it, it uses "arena" allocation of buffers to avoid GC and other
tricks to get the JVM up to par with non-GC languages, but isn't really
typical of JVM development.

I would also like to know if HTTP protocol "hacks" are used, or networking
tricks like userland network stacks.

But alas, it's just a huge wall of github projects and not a lot of use to a
run of the mill developer like me.

~~~
matt42
I'll write a post later explaining how I reached those performances, several
people already asked for it.

------
9wzYQbTYsAIc
Interesting to see asp.net at the very bottom of the list of 100+ and asp.net
core in the top 10.

~~~
e12e
Indeed, quite impressive for a presumably "fully production grade" stack to
score so hight. But is it me, or are there no fsharp listings in round 19?

~~~
Multicomp
Zebra is on there. Giraffe may be in there.

~~~
e12e
Strange, I can find zebra in the filter list, but not in the result list. (ff
and chrome on Android).

------
paultopia
I'm more interested in the fact that something called asp.net core is one of
the five fastest, while asp.net not 'core' is the single slowest. What's the
deal there? I dimly know that asp.net is a Microsoft thing, and remember it
being used by a bunch of enterprise sites a decade or so ago...

~~~
mythz
ASP.NET Web Framework is the Windows only 18 yo Web Framework based on .NET
Framework
[https://en.wikipedia.org/wiki/ASP.NET](https://en.wikipedia.org/wiki/ASP.NET)
It could run on Mono/Linux, but did so very slowly & buggy and possibly what
the slowest benchmark was run on.

ASP.NET Core is the newer (complete rewrite, inc runtime), leaner cross-
platform (Win,Linux,macOS) Web Framework that's built on top of the
performance focused .NET Core runtime released in 2016
[https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-
cor...](https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-core) They're
only the same by name which is unfortunate, because it's completely new. It's
the future runtime/platform for .NET and will be renamed to .NET 5 whilst the
older .NET Framework will stay at v4.x (currently at v4.8).

~~~
qes
> They're only the same by name which is unfortunate, because it's completely
> new.

I certainly wouldn't say that. Full Framework .Net and .Net Core are _far_
more similar than they are different - they share a very large common base
class library API surface in Net Standard.

A huge percentage of .Net MVC 3-5 (full framework) applications could be
migrated to .Net Core MVC in a matter of hours to days.

You don't even have to specifically target .Net Core in a lot of cases - v2
included a compatibility shim so that libraries targeting full framework just
work as long as they don't call any of the API's that were in full framework
but are not in the Core runtime.

The main differences are in startup configuration of your web app and how you
go about implementing cross-cutting concerns that might inspect or intercept
every request - but even for a lot of common things like authentication there
are similar extension points to what existed previously.

There are some particular technologies - like WCF Server or Linq2Sql - that
are hard blockers if you were relying on them - but by and large moving from
older full framework .Net to .Net Core is not that difficult and certainly
doesn't require learning much new, especially fundamentals - it's mostly just
some details of the web app framework.

~~~
mythz
ASP.NET Core was a rewrite. None of the previous .NET Framework or
HttpListener HTTP classes or Abstractions are available in .NET Core which
were rewritten to use newer HTTP Abstractions from scratch.

> The main differences are in startup configuration

It's definitely not the main difference, it may be what's immediately visible
as a dev using it, but the entire Web Framework your App depends on is new.
It's not a fork of an existing code base, it's a completely different one with
the goal to retain the same development MVC & Web API model to preserve
knowledge reuse & ease porting efforts.

The compatibility layers are just that, retrofitted to preserve compatibility
& ease porting which is able to work through impl-free Reference assemblies
with APIs they both share. Of course none of ASP.NET Framework System.Web is
in .NET Standard, because it was rewritten.

The OSS code base for ASP.NET Core is at
[https://github.com/aspnet](https://github.com/aspnet) whilst you can view
source code of .NET Framework (which isn't OSS) is at:
[http://sourceof.net](http://sourceof.net)

------
karmakaze
I really do like these TechEmpower Fortunes benchmarks. It has quite an
extensive list of frameworks, languages, and configurations with throughput,
latency (w/ SD), and errors.

I always check any lesser known framework to see which ballpark it falls into.
I'm always surprised to see that so many of the popular frameworks are ~10x
worse than the best--although many of those 'best' don't do as much
processing. I'm much more likely to pay attention to the error counts, max
latency, or SD (σ).

------
dathinab
I wonder how realistic/fair the code of some benchmarks is.

I just took some small peak at atix (because I happen to know it) and while
first all looked fine it wasn't quite that realistic.

Mainly:

\- It uses a fork of tokio-postgres specific for the test (which differs in
that Client is no longer send and it has a Unsafely soundness hole by wrongly
using Unsafe Cell, and no Issue tracking enabled on, through replacing that
cell with RefCell probably yield very similar performance)

\- Instead of using the default web::Json responder it uses simd-json (EDIT:
explicitly encoding the data into a buffer instead of returning it wrapped in
web::Json)

\- Uses snmalloc instead of the default allocator
([https://github.com/microsoft/snmalloc](https://github.com/microsoft/snmalloc))

Just to be clear besides the first point all of this is not unrealistic for a
context where you want to highly optimize your server. But not how actix is
used most of the time.

\---

Edit: Also just to be clear I didn't nit pick intentional on actix, it just
happen to be the framework I'm more familiar with and especially using
snmalloc for actix seems to be quite a reasonable idea.

EDIT2: The Unsefety unsoundness isn't triggered given the way this library is
used, it's still there and would prevent this change from ever been merged
upstream.

------
ex_amazon_sde
Most of the time the framework is not in the top 10 bottlenecks of your
application.

Optimize your webapp before hunting for the fastest frameworks.

~~~
ratww
I know this is a popular line of thought, but as someone who really measures
things, it doesn't match my experience when it comes to _web_ server
frameworks.

In web applications, apart from glaring mistakes such as N+1 queries, pretty
much every bottleneck I had in production was due to slow serialisation, slow
database abstraction or excessive CPU/memory usage due to language/framework.
Also had issues caused by initialisation time. Saving 200ms or 300ms on each
request can give you better user experience.

For growing startups, having a fast web framework gives you the choice to
postpone doing complex horizontal-scaling for months, or even years. This
saves money on the sort term and allows you to keep growing their products
without having to worry about other things. The database can scale vertically
in the meantime.

As for large tech companies: at my previous workplace there were two
migrations in the span of two years because of performance issues. Hundreds of
developers collectively agreed on changing the tech twice because of framework
bottlenecks.

Of course you shouldn't go straight into a C++ framework when Rails becomes
too slow for you, but people should know the tradeoffs of the framework
they're using.

~~~
ex_amazon_sde
_Most of the time_ the framework is not in the top 10 bottlenecks of your
application.

Believe me (or not) when I say that Amazon, among all FAANGs, could be the
most data-driven, at least in the teams I knew, almost to an excess.

------
montroser
On lithium's JSON parser, they describe:

"Only cases where the structure of the object is known at compile time are
covered."

Numbers are impressive, but there are some big trade-offs that come with.

~~~
Yoric
That depends exactly on what that sentence means. It is fairly unusual that
you need to parse a JSON with unknown structure during any kind of speed-
critical section.

My personal criticism is that C++ is usually (not always, though) the wrong
language for implementing web-facing applications or APIs. There is a
considerable advantage in terms of speed (when using the right framework) but
there are so many things that can subtly go wrong in terms of safety and
security that you need to be very, very careful when picking C++.

And I write this as a C++ developer.

~~~
regularfry
It's fairly unusual that _today_ I don't know the structure of the JSON I'm
parsing. It's not at all uncommon to want the server to do the right thing if
I add keys to the object which it didn't previously care about, though.

------
foolinaround
Link to the project :
[https://github.com/matt-42/lithium](https://github.com/matt-42/lithium)

not obvious from a google search

------
tluyben2
Offtopic: matt42 (or others); where do I learn modern optimization like this?
I can optimize for cycles on embedded MCU's and older computers and I can
optimize in .NET/JVM (and others) but are there any good sources for OS on
modern metal optimizations? Besides reading the Lithium sourcecode ofcourse.

~~~
matt42
I would say the profiler output is the best source of information. Lithium is
not using black magic at all. Actually on large framework like lithium,
performance problems come more from obvious errors that get hiden in the
quantity of code than in code that you did not crazy optimize.

Only the profiler will tell you if there is a hidden malloc that is slowing
down all the rest, or a inefficient datastructure, or ...

There is probably lot of non optimized code in lithium, it just has no impact
(or if it has I still has to find it).

That said, for other kind of software (usually more CPU bound), going down to
assembly or using SIMD can be the only way to get good perfs.

------
rapfaria
Anybody else's laptop activates their fans when visiting this?

~~~
matt42
Me, the benchmark data is pretty slow to process.

------
pknerd
Thanks to Google, The PHP Framework, Lithium will get some unwanted praise
here.

------
matt42
I'm wondering why this post got suddenly removed from the front page...

~~~
giancarlostoro
People probably flagged it cause people hate benchmarks / dont trust them. I
think they at least give us some indication about things on the other hand.

~~~
matt42
arf. ok. It's probably the mods who unflagged it.

~~~
ricardobeat
There is a comment by one of TechEmpower guys - this benchmark is not the
official one and results should be taken with a handful of salt.

~~~
matt42
This is the official continuous techempower benchmark, run by techempower on
the exact same environment than the techempower "official rounds". The only
diff I'm aware of is that there is more manual review and check for rounds
than for continuous benchmarks. Here is more info about it here:

From [https://www.techempower.com/blog/2018/06/06/framework-
benchm...](https://www.techempower.com/blog/2018/06/06/framework-benchmarks-
round-16/):

We have already seen tremendous social adoption of the continuous benchmarking
results. For selfish reasons, we want to continue creating and posting
official rounds such as today's Round 16 periodically. (Mostly so that we can
use the opportunity to write a blog entry and generate hype!) We ask that you
humor us and treat official rounds as the super interesting and meaningful
events that they are. Jokes aside, the continuous results are intended for
contributors to the project. The official rounds are less-frequent snapshots
suitable for everyone else who may find the data interesting.

------
bsaul
has there ever been a code created just specifically for this benchmark, in
assembly language with everything hard-coded, just to see what the upper bound
is ?

~~~
matt42
Some framework are using raw sql requests to skip the overhead of the ORM but
they are still slower (lithium's ORM has no cost at runtime anyway). Coding
everything in ASM would be insane and increadibly hard to ensure that this is
the optimal version.

~~~
e12e
Brings to mind:

"Rapid Web Application Server (in Assembler)":

[https://2ton.com.au/rwasa/](https://2ton.com.au/rwasa/)

Discussed on hn a few times as I recall:

[https://news.ycombinator.com/item?id=9948749](https://news.ycombinator.com/item?id=9948749)

(see also sshtalk):

[https://2ton.com.au/sshtalk/](https://2ton.com.au/sshtalk/)

[https://news.ycombinator.com/item?id=15829206](https://news.ycombinator.com/item?id=15829206)

~~~
matt42
omg...

------
thdrdt
Wow, the difference between .Net core and .Net!

------
RocketSyntax
just ahead of jooby and kooby

------
fluffy87
2x faster than the Actix Rust framework.

Rust, C++ just said “hi” to you again, it’s on. Time to make Actix fast.

------
polote
Lithium is the fastest web framework among the frameworks that nobody use

Seriously why is this reaching the front page, who really care about such
benchmark ?

~~~
fluffy87
Those who work on these frameworks, and those interested in http framework
benchmarking.

Some of the ideas developed in these frameworks percolate down to the PLs that
use them and to the libraries they use, which benefits everyone. These
frameworks also test new language features.

~~~
aioprisan
Particularly the case with PHP and Zend from 5.0+ days. Are there other
examples across other stacks?

