
Web Framework Benchmarks Round 9 - osener
http://www.techempower.com/benchmarks/#section=data-r9
======
idlewan
Disclaimer: I'm the author of Nawak, so I'm pretty excited about this.

The Nimrod programming language is finally featured for the db tests (at least
on i7. Not sure what happened with ec2 and peak, as neither jester nor nawak
seem to appears in the results).

It fares pretty well when the database is involved.

Look for the nawak micro-framework, in the top 10 both for fortunes:

[http://www.techempower.com/benchmarks/#section=data-r9&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r9&hw=i7&test=fortune)

and updates:

[http://www.techempower.com/benchmarks/#section=data-r9&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r9&hw=i7&test=update)
)

And there is room to grow. That micro-framework will not be the best for the
json or plaintext tests, but once the database needs to be involved, it is
trivial to add more concurrency: firing up more workers (1-3meg each in RAM)
acts as an effective database connection pool (1 database connection per
worker).

edit: Why should you care? Nawak
([https://github.com/idlewan/nawak](https://github.com/idlewan/nawak)) is a
clean micro-framework:

    
    
      import nawak_mongrel, strutils
    
      get "/":
          return response("Hello World!")
    
      get "/user/@username/?":
          return response("Hello $1!" % url_params.username)
      
      run()
    

Benchmark implementation in 100 lines here:
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nawak/nawak_app.nim)

~~~
dom96
(Author of Jester here)

It's nice to see Nimrod making it high up on the list of the fortunes
benchmark. I still have not had the time to properly optimise my framework, or
to even write code for the other tests, but I am currently working on a new
async module which will be a lot more competitive (an experimental version of
it already made it into the latest Nimrod release, see [http://nimrod-
lang.org/news.html](http://nimrod-lang.org/news.html)). In any case it's nice
to see that others are creating their own Nimrod frameworks to compete with
mine (thanks idlewan :) ).

------
bhauer
We're very happy to have Round 9 completed. The 10-gigabit 40-HT core hardware
has been quite amusing to work with. In most respects, these servers utterly
humble our in-house workstations that we've been using for previous rounds.

If anyone has any questions about this round or thoughts for future rounds,
please let me know!

~~~
nostrademons
I'm curious, would you be interested in microbenchmarks for things like
concurrency models? I'm thinking about preforking vs. thread-per-request vs.
events for C++, NIO vs InputStreams vs. Fibers for Java, etc. The idea would
be to measure how much overhead is coming from the OS vs. the language vs. the
framework so that framework authors could make informed choices for their
framework architecture. Or would that be out of scope?

~~~
bhauer
We are comfortable accepting and including what we call _permutations_.

Clearly, our project is about testing web application frameworks and platforms
(we just say "frameworks" for conservation of typing). Testing NIO vs.
InputStreams, for example, in isolation is a different project.

But as part of a web framework—let's say you had a framework that provided
support for both NIO and InputStreams—we'd be happy to include the multiple
permutations.

Obviously, this is within reason. If you submitted 512 permutations, I might
have a small heart attack. Also, a general principal that we ask all
participants to bear in mind is that each test should represent a viable
production-grade configuration. This guideline might rule out a great number
of wilder permutations.

------
eggbrain
These web framework tests have been really interesting to look at, and each
time I've been saddened to see that Rails/Ruby, the framework/language I
program with most days, is consistently near the bottom. With Adequate Record
now being merged into master, I'm hoping we start climbing the speed test
records.

But a question that keeps coming up in my mind is that there are metrics that
would be much harder to compare, but might be more useful in my book.

For example, I'd love to see a "framework olympics" where different developers
build an agreed upon application/website on an agreed upon server using their
favorite framework. The application has to be of some decent complexity, and
using tools that an average developer using the framework might use.

In the end, you could compare the complexity of the code, the average page
response time, maintainability/flexibility, and the time it took to actually
develop the app and the results could let developers know what they sacrifice
or what they gain by using one framework over the other. I know a lot of these
metrics could reflect the developer themselves vs the actual framework, but it
might also be a tool to let you know what an average developer, given a
weekend, might be able to produce. It would also help me to see an application
written a ton of different ways -- so I can make good decisions about what
framework to choose based on my needs.

In the end, speed only tells us so much -- and speed is not the only metric
that we consider when we write applications -- otherwise it looks like most
developers would be coding their web apps in Gemini.

~~~
bhauer
Readers should definitely evaluate the complexity of the code necessary to
implement the tests. To that end, I plan to eventually enhance the results web
site to allow you to readily view the related source for each test
implementation (perhaps using an iframe pointing at GitHub).

For the very first round, we included the relevant code snippets in the blog
entry, and I think that added a lot to the context. With nearly 100
frameworks, the volume of code has become too large to simply embed all of
that code directly into the blog entry, but we've put too much burden on the
reader to sift through the GitHub repo to compare code complexity.

Things we aim to do:

* Pop-up iframe with relevant code from GitHub.

* Use a source lines of code (sloc) counter to render sloc alongside each result in the charts.

* Render the number of GitHub commits each test implementation has seen at our repository. At the very least, this would show whether a test has seen a lot of review.

* Introduce more complex test types [1].

And as the other reply has mentioned, we have also discussed the possibility
of a larger test type that might include a multi-step process. I'd love to
eventually get to that point.

[1]
[https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...](https://github.com/TechEmpower/FrameworkBenchmarks/issues/133)

~~~
programmer431
a matrix rank calculator might be helpful as well to filters results for
users, though I am impressed with the present and new(ish) filtering
available.

------
nemothekid
Interesting to see Go benchmarks fall right out of the top 10 on the 10GbE
machines while Java/JVM and even Javascript do amazingly well. My guess at
10GbE you are now testing how much the framework spends on the CPU.

~~~
sandGorgon
there's something funny - for the multiply queries benchmark [1], Go is
performing worse than codeigniter, kohana. But cpoll_cppsp-postgres is
performing worse than codeigniter-raw .

anybody know what's going on .. is a full PHP framework performing better than
raw c++ under certain conditions?

[1]
[http://www.techempower.com/benchmarks/#section=data-r9&hw=pe...](http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=query)

~~~
Loic
PHP is for these tests, nginx and 256 to 512 PHP processes under PHP-FPM. Not
a single process. If you put 40 Go processes behind nginx, each one allowed to
use only 2 to 4 processors, you will get notably different results.

Anyway, the tests show how hard it is to scale to many processors with a
single process. It shows that the Java VM is really good at it (to be
expected, Sun always had systems with insane numbers of cores).

------
RyanZAG
Couple notes:

They have a blog post about the results here:
[http://www.techempower.com/blog/2014/05/01/framework-
benchma...](http://www.techempower.com/blog/2014/05/01/framework-benchmarks-
round-9/)

If you're running on EC2 and not dedicated hardware (probably most people
reading this), be sure to toggle to the EC2 results at the top right of the
benchmark.

------
nobullet
It is great to see my beloved Ninja framework (fullstack, Java) in standalone
mode to be the one of the best performers in multiple queries benchmark
(better than 81% of 93 known frameworks) and data updates (better than 77% of
93 known frameworks).

These are the most realistic scenarios for the web app in my opinion.

------
ksec
And Ruby on Rails is...... at the bottom of the chart again.

It is quite disheartening to see it being 20 - 50x slower on the best. Or 2 -
5x Slower to other similar framework.

------
manishsharan
How is this data useful to someone building a web application? I have used
several of these frameworks ,alteast the jvm based ones, and I can tell you
that it is like comparing apples to oranges. Case in point, Wicket, which I
have been using for several years, is a component oriented framework with a
rich ready to use pre-built components. If on the other hand you are using
netty, you are left to reinvent pretty much everything . Based on your
configuration, it may be that wicket is returning the response from cache.
Compojure and wicket serve different business use cases.

~~~
kbenson
Like any set of benchmarks, using the raw number without taking into account
externalities such as those you listed would be a poor indicator of the best
framework for your needs.

That said, if you need serving power, the fact that some some solutions can
literally serve 100x more requests than others, and for likely less than 100x
slowdown in development time and effort, may matter.

------
ldrndll
I was hoping that Snap & Yesod would be run on GHC 7.8. It'll be nice to see
what sort of improvements MIO will make, especially on the 40 core machine.

~~~
vdaniuk
Yeah, I am currently learning Haskell and was surprised by the apparent low
performance of Yesod and Snap in comparison to the faster frameworks.

I am wondering if I should spend more time on Clojure instead, as it seems to
be significantly higher in the ratings.

~~~
cgag
The next version of GHC should improve those numbers significantly, but I
would also note that there are much more significant reasons to consider using
Haskell than the difference between 37k and 43k req/s.

------
jsnk
I am surprised that NodeJS on Mysql is much faster than on MongoDB. Is this
expected?

[http://www.techempower.com/benchmarks/#section=data-r9&hw=pe...](http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=query)

------
cwufbt08
There are many frameworks showing as "did not complete". I was interested to
see the results for Spray since it did really well in previous rounds, but
there are no results for Spray in the latest round.

~~~
bhauer
"Did not complete" indicates that a framework's test implementation does not
pass validation. Spray is somewhat exceptional because the Spray community
asked for it to be pulled until they have the time necessary to re-implement
its test case. See the PR below:

[https://github.com/TechEmpower/FrameworkBenchmarks/pull/842](https://github.com/TechEmpower/FrameworkBenchmarks/pull/842)

------
rschmitty
Could you make the error log accessible? Some of the frameworks appear to just
bleed errors left and right. Be interesting to see if they are real errors or
just misconfiguration

~~~
bhauer
We still need to post the final logs, but you can find logs from the preview
of Round 9 at the following GitHub repository:

[https://github.com/TechEmpower/TFB-
Round-9/tree/master/peak/...](https://github.com/TechEmpower/TFB-
Round-9/tree/master/peak/linux/preview-1/latest/logs)

~~~
bhauer
I can't edit the above message. The final logs are posted now:

[https://github.com/TechEmpower/TFB-
Round-9/tree/master/peak/...](https://github.com/TechEmpower/TFB-
Round-9/tree/master/peak/linux/final/latest/logs)

~~~
rschmitty
thank you!

------
faizshah
Does anyone know why Ur/Web has far better performance on EC2 than other
platforms?

Also can anyone share their experience using cpoll_cppsp?

------
megaman821
I would love to so Varnish in here for some of the tests.

For a typically webpage with multiple queries there appears to be around 5-10x
performance disadvantage between slow and fast languages.Things like serving a
plaintext or json response, where the slow languages are much much slower,
Varnish is a good match for.

~~~
jerf
I would be intrigued just in general to compare against nginx or something
serving a static file of the same size. Obviously that's "cheating" but in
some sense it's a better benchmark for me to judge against than "what happened
to be the fastest framework this time".

~~~
bhauer
We are focused on testing without reverse proxy caches such as Varnish and the
caches provided by nginx and Apache HTTPD. This is because we want to measure
the performance of dynamic requests—requests that for whatever reason are not
cached.

That said, we have considered including a measure of, say, nginx delivering
the same content statically as a high-water mark. That could be interesting to
see how well a framework compares to what might be an ideal case.

~~~
jerf
I mean your second paragraph, and for that reason.

------
sergiotapia
I may have overlooked it but I cannot find ASP.Net MVC5 there. Is that
framework version being tested?

~~~
epaladin
I think they were trying to run it under Mono, and it failed to complete most
of the tests. Setting up IIS for a "real" test of ASP.NET MVC might be out of
their intended scope.

~~~
bhauer
We do have Windows tests (run exclusively on the Peak environment for Round
9), but most of the ASP.NET tests are not starting and passing validation
correctly as-is. We'd like to get these fixed up along with all of the other
tests that are not passing validation and running correctly. As you might
expect, we have a never-ending game of whack-a-mole with so many frameworks in
play, so we lean heavily on the community to assist. We'd love to get pull
requests to help validate the existing .NET tests.

------
edgarvaldes
It's disappointing to see Laravel at the bottom, showing 47,135 errors and a
51% succes rate.

~~~
JonoBB
Given that the Laravel tests completed fine in all previous benchmarks, I'm
guessing that there is something else amiss here.

------
itry
Nice to see how great PHP is doing. It is still my favorite language.

One thing is strange: The HHVM result on the "plaintext" test. How can HHVM
only do 938 requests/sec if it can do 70,471 in the much more complicated
"singly query" test?

~~~
bhauer
Yes, something is presumably wrong with the hhvm plaintext test. Some test
implementations enter a failure state after the warm-up and never recover.
That type of failure is distinct from the "did not complete" failures where
the validation checks do not pass.

------
bigdubs
Still scratching my head at the C#/httplistener results. Ostensibly it should
be pretty close to what a native C++ implementation should look like
performance-wise, as a good chuck of the work on the raw text results is done
by http.sys.

~~~
RyanZAG
Just because it's written in native c++ does not mean all native c++ code will
perform the same. Just look at the difference between the top java framework
and bottom java framework - huge difference, even though it is the same
language. Same likely applies to the difference between http.sys and something
like cpoll

~~~
bigdubs
So normally, I agree that there is a difference between projects in the same
language.

However, http.sys is kernel mode in windows, and should have advantages that
other programs don't have.

------
necrobrit
Interesting how the various Scala frameworks have been slipping down the
charts from round to round -- an effect of changing methodology?

~~~
cwufbt08
Spray was one of the leading contenders in previous rounds, but didn't
complete in the latest round. I wonder why that is.

~~~
kclay
May be because Spray was merged into the playframework,also I think it plays a
bit in the akka http layer as well, not to sure.

~~~
saryant
Spray has not been merged into Play, it's just a Typesafe project and will
eventually be merged into Akka.

------
rcarmo
Hmmm. No Python-gevent? Still? Good showing of http-kit, though. Love that
lib.

~~~
rschmitty
It's open source benchmarking. There are instructions on how to submit a
language-framework of your choosing.

[https://github.com/TechEmpower/FrameworkBenchmarks](https://github.com/TechEmpower/FrameworkBenchmarks)

------
joeblau
Wow HHVM on top for Data Updates! Congrats to Facebook!

