
Web Framework Benchmarks - pfalls
http://www.techempower.com/blog/2013/03/28/framework-benchmarks/
======
goodwink
One of the most interesting things this comparison brings out to me is not so
much the differences between the various frameworks (although the differences
between options on the same platform is definitely very useful information),
but also the issue that few of us seem to think about these days: the cost of
any of the frameworks above the bare standard library of the platform its
hosted on.

Theres a consistant, considerable gap between their "raw" benchmarks (things
like netty, node, plain php, etc.) and frameworks hosted on those same
platforms. I think this is something we should keep in mind when we're tuning
performance-sensitive portions of APIs and the like. We may actually need to
revisit our framework choice and implement selected portions outside of it
(just like ruby developers sometimes write performance-critical pieces of gems
in C etc.) or optimize the framework further.

I'd like to crunch these numbers further to get a "framework optimization
index" which would be the percentage slowdown or ratio of performance between
the host platform and the performance of the framework on top of it. I might
do this later if I get a chance.

~~~
jondot
I think this is a much needed and excellent point to make. Just take a look at
how Go dips down when using Webgo.

~~~
bilbo0s
I'm not sure many people would use webgo in real life. I don't know... maybe
some people... certainly not pros.

Also, the 1.0.3 thing is probably dragging on the numbers a bit. 1.1 would
boost it a little. Not enough to get it into the top tier... but a little.

Also, for Vert.x, they seem to be only running one verticle. Which would never
happen in real life.

Play could be optimized a bit... but not much. What they have is, to my mind,
a fair ranking for it.

Small issues with a few of the others but nothing major. I think Go and Vert.x
are the ones that would get the biggest jumps if experts looked at them. And
let's be frank... does Vert.x really need a jump?

So what they have here is pretty accurate... I mean... just based on looking
through the code. But Go might fare better if it used tip. And Vert.x would
DEFINITELY fair better with proper worker-nonworker verticles running.

~~~
smandou
The Play app uses MySql in a blocking way, while Nodejs uses mongo. It's not
comparable.

~~~
bhauer
We have a pull request that changes the Play benchmark (thank you!) so we will
be including that in a follow-up soon.

We tested node.js with both Mongo and MySQL. Mongo strikes us the more
canonical data-store for node, but wanted to include the MySQL test out of
curiosity.

------
vanderZwan
_"Among the many factors to consider when choosing a web development
framework, raw performance is easy to objectively measure."_

Oh really? Then why did Zed write such an angry rant about how you are doing
it wrong?

<http://zedshaw.com/essays/programmer_stats.html>

Can we please see some standard deviations, at least?

~~~
k__
What's wrong with that guy?

~~~
sergiotapia
He's a fantastic software engineer but he is _very_ abrasive. I've read some
of his posts claiming he would fight some other developer in person at a
conference if he steps up. He would rent out a ring and he would put his
yellow belt to practice.

Really, I'm not making this up. He sounds like a jerk to work with.

~~~
coolsunglasses
He's actually a really nice person. He helped me out (to him I was just some
stranger on the phone) when I was trying to decide what to do with my career
when I was in NYC.

Zed Shaw is probably one the best people you can know in the developer
community, a very good guy.

Your sensationalism based on some of the stuff he says on Twitter and Blogs is
amusing though.

~~~
k__
> Your sensationalism

well he doesn't try to be a nice guy in public...

------
atonse
I'm torn about this. On the one hand, while I've known my framework of choice
(Rails) is slow, I didn't know how much slower it could be in the grand scheme
of things. But on the other hand, I'm more shocked by the difference between
EC2 and dedicated hardware (10x improvement with rails), and even 89 requests
per second (20 query benchmark on EC2) is still a decent amount of traffic.
(Plus this doesn't count any optimizations I would make anyway, like caching).

Either way good architectures usually optimize the high traffic or high CPU
areas anyway away from a scripted language.

Thanks for the really informative post! Go seems to be a good balance as a
high performance language without having to go back to my traumatic Java days.

~~~
jondot
Don't be too disappointed about Rails.

As a rule I like to divide this world into "Featureless" and "Featurefull"
products.

When you use Rails, you're aiming to pile up features. You want to react to
Product managers, to users, you want to work fast and satisfy the needs of
customers - or else you won't have anyone to build to.

In this reality, the fact that you're doing 20req/s is OK. In fact, I'm
betting that even when you take Go or Node.js - pile up all of the
infrastructure and features that exist in Rails, and pile up a ton of your
code - buggy and not buggy - you'll get around the same kind of satisfaction
index from users.

This is because your product can be perceived as slow even though your servers
are blazingly fast.

On the other side of the spectrum there are "Featureless" products. These are
infrastructural products. A logging service. An analytics service. A full-text
search. A classification and recommendation engine.

These you don't want to build in Rails. I'm sure you haven't even considered
it. These you want to build with one of the top-notch libraries that this
survey indicate.

~~~
film42
Also, there are certain features about rails like thin or unicorn that can
drastically increase your overall performance. So in that sense, I think it's
a lot more complicated to determine.

------
bhauer
I worked with Pat (pfalls) on this effort. He pulled the benchmarks together
and built the script to automate the tests. We aimed to deploy each
framework/platform according to best-practices for a production environment
and then stress test common operations: JSON serialization of objects and
database connectivity. We were surprised by the wide spectrum of performance
we observed and hope that this interesting to you as well. Four orders of
magnitude in one of our tests!

If you have any questions or see something we stupid we did, please let us
know. We'd like to correct any mistakes straight away, especially since we're
certainly not experts on all of these frameworks and platforms.

~~~
goodwink
I'd be interested to see performance for Vert.x on its other hosts (this is
the JVM version, I believe).

~~~
bhauer
Interesting point. You're correct, we've only tested Vert.x as a Java/JVM
platform.

~~~
proland
I agree, seeing vert.x with it's other language options would be interesting.

------
sergiotapia
Where is ASP.Net MVC? Odd that you list obscure frameworks like Wicket and
leave out one of The Big Four frameworks.

(The big four in my book are: ASP.Net MVC, Rails, Django and CakePHP)

~~~
bhauer
We'd love to have ASP.Net MVC included. One minor gotcha is that to do it
justice, we'd need to spin up a Windows EC2 instance and figure out how to
script that. It's on our to-do list!

We did briefly test ASP.Net on Mono (see another comment in this thread) but
didn't include it since we didn't believe that qualifies as a "production"
grade ASP.Net MVC deployment.

~~~
voidlogic
You should include it; Lots of shops are deploying their products on .NET MVC
with Mono.

~~~
Revex
I agree... Let's see what the numbers show for Mono, and on Windows.

------
akmiller
I'd like to see how .Net MVC would compare. I realize you'd have to spin it up
on a Windows EC2 instance and there would definitely be some variance in the
performance of that box vs. the nix EC2 instances but I'd still be interested
in seeing how it fares in comparison.

~~~
bhauer
We agree. We aim to provide a .NET MVC test soon. We did briefly test that on
our i7 hardware and if memory serves me correctly, it clocked in at around the
position as Spring.

But don't quote me on that! :)

~~~
hhudolet
I would very much love to see:

(win | mono) + (httphandler | asp.net mvc | webapi | servicestack | nancyfx)

it would hopefully compare with java stacks!

------
sequoia
> This exercise aims to provide a "baseline" for performance across the
> variety of frameworks. By baseline we mean the starting point, from which
> any real-world application's performance can only get worse.

I disagree with the implication here (that this is a good point for comparison
because "real-world application's performance can only get worse."). Yes it
can only get worse but _how much_ worse (per unit of "features") is both
significant and unaddressed.

This isn't the best example but look at the gap between the top and bottom of
the scale in the _Database access test (single query)_ and _Database access
test (multiple queries)_ charts: In the first, Gemini is ~340x faster than
Cake, in the second, only ~23x faster. There is still a big gap but it closed
by an order of magnitude once you stepped past the most trivial possible DB
access test.

So nodejs or php-raw is faster than cake at a single DB access, but what about
when you create a real world scenario with authentication, requirement to be
able to update features faster (i.e. use an ORM), env. portability
requirement, etc.? It seems to me this would look like _a little slower, a
little slower, a little slower_ in the {raw} versions, and _already included,
already included, already included_ in Rails or Cake. The full featured
frameworks take a lot of their performance penalty up-front, with less of a
hit as features are added (maybe? :P).

My point is that it's not reasonable to assume that hackernews-benchmarks will
actually reflect production use. That said I think the article is cool, and
agree that it's good to keep framework authors' feet to the fire regarding
performance!

~~~
campnic
This is completely loaded. Your implication is that the only viable test is a
test which exercises all of the functionality of the most feature rich
framework. How would that be a)viable and b) meaningful?

We know that there is a set of common features and the benchmarks goal is to
test least common denominator stuff on the networks. Authentication and
portability are not LCD. The argument that they are is capricious. What if we
made the requirement be that the framework is a lisp? Now we've completely
changed the intent.

~~~
sequoia
I meant to suggest that comparing php-raw to Rails is apples & oranges- not
"you must benchmark in a way that benefits larger frameworks", just "please
acknowledge that LCD tests like this inherently cast Railsy frameworks in a
bad light."

It's like condemning a swiss army knife because it's not as efficient as a
fixed blade at cutting apples. Well yeah that's true, but what about when you
need to screw a screw or pull a cork? One is a multitool, it doesn't make
sense to compare it to a specialized tool unless all you plan to do is cut
apples.

------
mythz
Would've loved to see <http://servicestack.net> on this list which has great
performance on .NET and Mono:
[https://github.com/ServiceStack/ServiceStack/wiki/Real-
world...](https://github.com/ServiceStack/ServiceStack/wiki/Real-world-
performance)

And also maintains .NET's fastest JSON and Text Serializers:
[http://theburningmonk.com/2011/11/performance-test-json-
seri...](http://theburningmonk.com/2011/11/performance-test-json-serializers-
part-ii/)

~~~
bhauer
Thanks for the tips on these. We'll add ServiceStack to our to-do list. As you
might imagine, that list is getting long as a result of some great community
feedback. Pat (pfalls) is diving into the pull requests this morning.

------
jondot
pfalls - amazingly, I spent the last 2 days of my holiday doing the same thing
for a future open source project. I was just stumped when I saw you guys did
the same (could have saved me a couple of days!)

I wanted to find the leanest Web framework on any kind of platform; but the
difference from your approach - I already knew the kind of code that would run
on it.

I tested: Go, Java (servlet, dropwizard), Scala (scalatra), Ruby, Node.js
(connect).

For me it was:

* Scala

* Java

* Clojure (equal to Java - big surprise here)

* Node.js

* Go (almost equal to Node.js)

* Ruby (far far down)

Scala took the lead with amazing results. More over, a good metric was latency
which Scala was the only one to take micro-second resolution.

I'm not a fan of Scala because of its surrounding tools, which is why I'm
still considering going for either Clojure or Node.js.

I think the most surprising positively was Clojure, being that it is a dynamic
language. And most surprising negatively was Go - by itself is impressive, but
when given real work (Web handling, Redis/mongodb) goes bad quickly. Happy to
see this correlates with your findings too; I'm assuming this is a symptom of
library maturity..?

I'd be happy to see how Scala fares on your tests.

You've done an awesome job!

~~~
pfalls
Thanks for the comment!

This started out as a small exercise, that quickly ballooned because we were
curious about every framework and platform. Obviously we had to stop
somewhere, but we're very interested in adding more tests in the future. In
fact, we're hoping the community will help us out as well!

~~~
jondot
Yes, I know the feeling :)

What started as a couple of hours of exercise for myself ended up as 2 days of
hacking and barely sleeping, as surprises in my assumptions kept unfolding,
and as I wrote and rewrote POCs just to validate that Clojure is as fast as
the number say, and that Scala is faster than Java, etc.

------
xenonflash
1) The Python version has some basic newbie coding errors. This sort of code
is what Python programmers call "Java written in Python". It may be a valid
algorithm in Java, but it's the wrong way to do it in Python. Code like this
will work, but it will be slow. Depending on the size of "queries", you are
potentially allocating gobs of memory in two different places for no reason,
and then throwing it away without using it. I wouldn't be surprised if the
examples in other languages had similar problems.

2) The JSON serializer in Django 1.4 uses a method which is known to be very
slow, but which is easily portable across different platforms and works with
older versions of Python. They no doubt included for easy bundling. In a real
application you would probably want to simply use the normal JSON serializer
from the standard library (which is many times faster).

3) The examples are little more than "hello world". I did some benchmark tests
with several Python async frameworks, Pypy, and Node.js for an application I
was working on. With small JSON objects there wasn't much difference in
performance. Once you started using large JSON objects the performance lines
for all versions were indistinguishable from each other. The performance
bottlenecks were in libraries, and those standard libraries were all written
in 'C', so interpreter versus compiler versus JIT made little difference.

4) The problem with "toy" examples is that in real life there are two
performance factors which must be taken into account. Think of as y = mx + b.
With a toy example you are probably only measuring "b". With most real life
applications it's "m" that matters. There are often different optimization
approaches that are best for varying ratios of "b" and "m". You have to know
your application intimately and benchmark using data which is realistic for
that application.

Python has a reputation for being "easy to learn". However, it is "easy" in
the sense of being able to hack something together that works without knowing
very much. There can be several different ways of doing things and doing it
one way versus another way can mean a difference in performance of several
orders of magnitude. The same may be true for some of the other languages, but
I haven't examined them in enough detail to say.

------
pekk
Numerous irregularities plus a strong vested interest in the JVM make me doubt
they have given adequate shrift to Go, here.

Given the amount of interest in Haskell and Yesod around here, it is strange
that it is missing.

~~~
papsosouid
>Given the amount of interest in Haskell

I see more people making uninformed "haskell sucks" posts than expressing
interest in it.

>and Yesod

Really? Yesod is the anti-haskell haskell framework.

~~~
markokocic
> Really? Yesod is the anti-haskell haskell framework.

Can you please elaborate. Being interested in Haskel web development and
trying to choose web framework makes me wish for more information.

~~~
pilgrim689
I wouldn't call Yesod "anti-haskell". By default, it relies on QuasiQuotes and
TemplateHaskell a lot [1], which are extensions to the GHC. So by default,
you'd have a hard time running Yesod applications on anything else but GHC
(the Glasgow Haskell Compiler). These extensions allow you to write in an EDSL
that generates Haskell for you. IMO, Yesod's use of these extensions are a
benefit, as it allows the user to get stuff like type-safe URLs in HTML for
free (e.g. you put href=@{Home} on your HTML element and Yesod will ensure
that the value interpolates to a route that exists _at compile time_ ).

Haskell libraries often depend on language extensions, whether it is
overloaded strings or type families or whatnot... so I think it's strange that
Yesod gets picked on for doing the same: taking advantage of the tools
provided by GHC to create a better environment for the developer.

[1] <http://www.yesodweb.com/book/haskell#template-haskell-14>

~~~
bjterry
Hugs is now defunct (last release in September 2006, doesn't even support the
2010 language standard), so there is no reason that being GHC-only should be a
consideration in selecting a Haskell framework. It's the only real option.

------
akg
Would love to see how these results compare to some of the web frameworks for
concurrent functional languages like Erlang/Haskell: Nitrogen, Chicago Boss,
Snap, Yesod, etc.

~~~
pilgrim689
ditto. I hear Warp (the server behind Yesod) is a beast.

~~~
bhauer
I'm not familiar with Warp. Would one of you guys be willing to help us put
together a test for Yesod?

~~~
josh-j
#haskell, #yesod, #snapframework on freenode are very helpful.

------
tnash
This is exactly why I decided to use PHP for my startup. I have something
along these lines that I hope to blog about in the coming weeks (I tested php-
fpm on nginx/go/node.js/silk.js and php won by a landslide when it came to
speed).

I would love to see php-fpm on nginx included in this test.

~~~
TylerE
The problem with php is that it looks great on (some) micro-benchmarks, but on
real apps under real sustained load it certainly turns to cold dog shit from
time to time for no apparent reason.

~~~
xd
What are you basing this on? I've been using PHP for well over a decade in
high load environments and never experienced it turn "to cold dog shit" .. any
issues I have experienced had a good reason, not "no apparent reason".

But then, I've never used a PHP framework in all the time I've used it ..
maybe that has something to do with me never having had negative issue with
PHP.

------
jalopy
As a Rails developer and admirer, this is eye-opening. I love the framework
(and Ruby especially), but these numbers bear some serious consideration.

30-50x performance difference gets really... real, no? The standard refrain of
"throw more hardware at it" must reconcile with the fact that a factor of
30-50x means real dollars for the same amount of load. Is the developer
productivity really that much greater?

~~~
bradleyland
Preface: This post is going to come across as a Rails apologist piece, but
please read the entire thing before you reach a conclusion. Please also
consider that you could apply these same arguments to just about any of the
high-level language based frameworks on the list. I use Ruby on Rails in my
comparisons, but I'm a huge fan of Node.js, Python/Django, and Go.

I fully respect the JVM family of languages as well. I just think that Mark
Twain said it best when he said: "There are three kinds of lies: lies, damned
lies, and statistics." It's not that the numbers aren't true, it's that they
may not matter as much, and in the way, that we initially perceive them.

Performance is certainly something you should consider when selecting a
language/framework, but it is not the only thing.

========================

You should undertake a detailed examination of these statistics before making
any decisions.

Issue #1) The 30-50x performance difference only exists in a very limited
scenario that you're unlikely to encounter in the real world.

Look carefully at the tests performed. The first test is an extraordinarily
simple operation: take this string, serialize it, and send it to the client.
This is the test in which we see massive differences:

Gemini vs Rails

25,264/687 (gemini/rails-ruby) = 36.774

25,264/871 (gemini/rails-jruby) = 29.000

Node.js vs Rails

10,541/687 (nodejs/rails-ruby) = 15.343

10,541/871 (nodejs/rails-jruby) = 12.102

That's a 37x performance win for Gemini, and 15x for Node.js.

Side note: You might be wondering why I didn't compare to the top performer,
Netty. Netty is more like Rack. You build frameworks on top of Netty, not
_with_ Netty. As a Ruby dev, you could think of this in the same context of
comparing Ruby on Rails with Rack; not a good comparison. Hence We won't
compare Rails to Netty.

The error would be in extrapolating that a move to Gemeni or Node.js would
give you a 37x or 15x performance increase in _your_ application. To
understand why this is an error, we jump down to the "Database access test
(multiple queries)" benchmark.

Issue #2) Performance differences for one task doesn't always correlate
proportionally with performance differences for all tasks.

In the multi-query database access test, we start to see the top JSON
performers slow down significantly when compared to the slow down for Rails:

Gemini vs Rails

663/89 (gemini/rails-ruby) = 7.449

663/108 (gemini/rails-jruby) = 6.138

Node.js vs Rails

116/108 (nodejs-mysql-raw/rails-ruby) = 1.077

60/108 (nodejs-mysql/rails-ruby) = 0.555

In this scenario -- which is arguably much closer to the real world -- Ruby on
Rails closes the gap and even beats some of the hip new kids.

But why? The in-depth answer to this question would require a lot of space,
but the really, really short version is kind of a "what's the sound of one
hand clapping" response: Ruby isn't actually all that slow.

To understand what the hell that means, check out this presentation from Alex
Gaynor (of rdio/Topaz fame):

[https://speakerdeck.com/alex/why-python-ruby-and-
javascript-...](https://speakerdeck.com/alex/why-python-ruby-and-javascript-
are-slow)

Ruby is just about as fast as C, provided you're comparing it to C that does
_exactly the same_ operations on the hardware as the Ruby code. Don't get me
wrong, that's a HUGE provision. But it warrants close examination.

The real benefit of lower level languages like C is that they give you the
flexibility to drill down in to your actual bare-metal operations and optimize
the way the program executes on the hardware. As Alex points out, we don't
currently have that level of flexibility in languages like Ruby (without
dropping down to inline C), so we suffer a performance penalty.

This penalty is _huge_ for simple tasks because they involve only a handful of
operations that execute extremely quickly. As you add complexity, however, the
benefits of micro-optimizations get lost in the vastness of the overall
execution time.

Look at it like this. When Gemini hits 36,717 req/s in the JSON test, each
request only lasts about 1.6 ms. This is only possible because of the
simplicity of the operations being done on the hardware. Ruby loses big here
because there is a lower boundary to the way you can optimize without dropping
down to C.

gemini: 1.6 ms per request

rails-ruby: 87.3 ms per request

When we look at the multi-query database access test, we can see how the
optimization at the low level gets lost in the sea of time taken to process
the request.

gemini: 90.5 ms per request

rails-ruby: 674.2 ms per request

Granted, that is still over a 7x performance win for Gemini, but this is where
the Ruby arguments about programmer efficiency come in to play. I don't know
Gemini, so it may very well beat Rails in that comparison too. Ruby is getting
more performant with every release though, so it's easier to justify on the
basis of preference alone when we're this close.

~~~
chrisdone
Don't conflate Ruby with Rails. Ruby _is_ slow:

[http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...](http://benchmarksgame.alioth.debian.org/u32/benchmark.php?test=all&lang=ruby&lang2=v8)

[http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...](http://benchmarksgame.alioth.debian.org/u32/benchmark.php?test=all&lang=jruby&lang2=v8)

and so is Python:

[http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...](http://benchmarksgame.alioth.debian.org/u32/benchmark.php?test=all&lang=python3&lang2=v8)

The only time Ruby or Python are fast is when the program is not running Ruby
or Python but running some C code underneath. If your programs only consist of
that, you can very well say "this ‘Ruby/Python’ code is fast". But as soon as
you have something that isn't in your standard library, welcome to the actual
language, and welcome to performance problems.

Elaborating on the implications of this: whenever you actually _use_ the
language to do some abstraction, you pay heavily for it:
[http://bos.github.com/reaktor-dev-day-2012/reaktor-talk-
slid...](http://bos.github.com/reaktor-dev-day-2012/reaktor-talk-
slides.html#\(31\))

~~~
bradleyland
You should really check out Alex Gaynor's slide deck. Nothing I said disagrees
with what you've said here, provided you take the entire thing in context.

------
neya
How about Lift? Btw, the play framework you tested is Java or Scala based?

Either way, I'm shocked to see Play perform so slow comparatively. Although
it's easily 10x faster than rails on most tests, I'm shocked to see Node.js
faster than Play! (by 2x in most cases) Wow!!

Maybe Node.js critics should start appreciating it after all..

~~~
kainsavage
It is probably worth noting that while we strive to make the tests as fair as
possible, we followed the official tutorials for each framework when building
out the tests but we fully expect there to be small instances where minor
tweaks improve a given test. Given that I am no Play expert, it would be of
great value to have one who is (and it sounds like you could lend a hand
there) to check out the code on the github page. If we did anything wrong with
the setup or in general, we would rerun the tests. Again, we followed all the
official 'getting started' posts for each framework, so we believe we have
best practices used.

Disclaimer: I am a colleague of the author of the linked article.

~~~
mark242
There are a few problems with your Play code that are causing it to be
unnecessarily slow.

First-- what you're really testing here is the Jackson library. A majority of
the cycles used in your application are being burned in that toJson call of an
array of objects. This isn't a fair test compared to the servlet
implementation because you're calling Jackson against a map in the Play
example, versus against a simple String in the servlet example.

Second-- you are running database calls serially, and those are blocking.
Considering that you're using the more-or-less default Play/Akka
configuration, there are only enough threads as you have available processor
cores. I would start by increasing the parallelism-factor and parallelism-max
to be higher, so you'll have more available threads. More importantly though,
the database access should be wrapped in a Future, and you should be returning
asynchronously. This should speed up the application by a huge amount.

~~~
neya
Do you think such a configuration could outrun the Vert.x configuration
they've posted? I'm not challenging you, I'm just genuinely curious! Because
if Play+Akka can outrun Vert.x, then it would be an interesting game
altogether...

~~~
mark242
I think Play, with well-written asynchronous code, could approach the
Netty/Vert.x speed. In other words, I'd be willing to trade the ease-of-use of
Play for the slight speed impairment vs. writing directly to Netty/Vert.x/etc.

~~~
bhauer
We'd love to test that theory. Can you or any Play expert rewrite our Play
code and submit a pull request?

------
vph
This seems to be a nice benchmark. For the Python group, I would suggest two
things: (1) include a lightweight framework like Bottle, and (2) Try pypy.

~~~
bhauer
Thanks. Agreed, we'd love to get a Python micro-framework in the test. If you
have some free time and feel like putting together a test for Bottle as a
Github pull request, we'd really appreciate that.

------
shanelja
I'm most surprised that PHP seems to be around an order of magnitude faster
than Ruby on Rails, I knew it was faster, but didn't think it would be that
much.

~~~
justinsteele
The comparison would be php-raw vs ruby. The actual "php" benchmark did just
as poorly as rails.

------
xal
What kind of server did you guys use for your rails test? Thin, Puma, Unicorn?
Are you sure you ran it in production environment?

Update:

Looks like passenger in development mode. Good job you benchmarked a web
server that no one uses wile reloading all code between requests.

Update2:

Ok it seems to run in production mode but still, passenger is not an idiomatic
choice.

~~~
daenz
I'm not familiar with Rails. Could you explain what you mean in layman terms?
Why is code being reloaded on every request by default?

~~~
awj
In development mode Rails will reload code on each request to pick up on
changes you have made to your application. That way you can interact with the
application to help verify that your code is working properly.

As of Rails 3.2 development mode watches for file changes and attempts to only
reload those files, but it's still a significant performance issue.

By default all Rails applications start in development mode, so one gotcha of
benchmarking Rails is that some people will forget to set the mode correctly.
That said, from the setup code[1] (line 14) it looks like they were running
passenger in production mode. The max pool size seems excessive, especially
when running on large ec2 instances, but I'm not fully convinced that it's out
of line.

[1]
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rails/setup_ruby.py)

------
anuraj
Interesting - C# should have also been added. Java still rules the roost. If
you should absolutely have a scripting language - old php with warts is
better. More magic in the framework - more abstraction and indirection and
code inefficiency - that can be the spoil port when it comes to performance.

~~~
bhauer
We agree and want to get a C# test in there. It's among the top priorities for
us.

------
desireco42
What I can take from this is that when you use ORM, it slows things down
considerably. Also, looking at rails example, you didn't use active record,
which is really wrong.

I think you should tweak your tests to use more real world like examples. I
realize it would be hard to do this across frameworks.

Like let's have a database query pull user record from 100,000 users by
username. And maybe do md5 on password.

------
plainOldText
How can you write a web framework benchmark and not include some of the non
mainstream languages with probably the most performant frameworks like Erlang
(Cowboy, Mochiweb), Haskell (Yesod, Snap Framework)? That's just wrong;
anyway.

~~~
brass9
Oh... and you think Erlang or Haskell frameworks are mainstream while you fail
to mention ASP.net?

------
krg
There is a huge difference between raw PHP and CakePHP. I'd be curious to see
other PHP frameworks (such as Zend, or Slim) in there-- is Cake just
particularly slow, or is that simply what happens when you have a PHP
framework?

~~~
Joeri
I'm curious if they configured php with apc or zend optimizer. The huge
difference is easy to explain as parse overhead for the framework's code,
which happens on each request if you're not using a bytecode cache.

~~~
beryllium
Yep, this is exactly what I was wondering. Opcode caching is a key part of
production PHP environments - in this case, since the code isn't changing,
they should even disable the "change check" (apc.stat=0) as one would do in a
production environment.

And if this is the case with their configuration of PHP, it makes me wonder
what other platforms are not configured for production in this benchmark :)

------
knappador
Utter crap.

"Sadly Django provides no connection pooling and in fact closes and re-opens a
connection for every request. All the other tests use pooling."

But it's free, open-source software and we provide asynchronous database
connection pooling for Postgres SQL:

<https://github.com/iiilx/django-psycopg2-pool>

~~~
andybak
Your comment is valid and would have been vastly improved without the first
two words.

~~~
knappador
I'll elaborate on the first two words:

They're shilling for their company with a config and tools I wouldn't be
caught dead using. No idea what other craziness lurks in the other daemon
configs. It's irresponsible and misleading. They're misrepresenting the
framework I use to do my work and probably others while hoping I or someone
else is going to do their work for them. "Outsourced CTO services?" Their
trash. My lawn.

Someone's going to eventually ask me to develop the rest of xyz in node and
I'll have to repeat myself about articles like this. Bad enough when it's
bloggers. Worse when it's self-shilling company that's obviously not willing
to put the time in to be what they claim to offer.

------
cridenour
I'm curious to see Django results when using the gevent worker for gunicorn.
For these type of quick JSON calls, you can see huge performance increases.

~~~
jaddison
Along with gevent, it would be good to throw in psycogreen to improve DB
(well, postgresql) evented connections.

~~~
bhauer
It looks as if we've got a Github pull request including these changes, so
we'll be able to revise the Python-Django numbers soon.

------
adad95
I Will love to see dotNet (C#) incluid in this test. Asp.Net WebAPI
(Synchronous and Asycnhrounous) Asp.Net MVC (Synchronous and Asycnhrounous)
Asp.Net HTTP Handlers (Synchronous and Asycnhrounous)

~~~
Throwadev
Also ServiceStack vs ASP.NET MVC vs NancyFX vs Fubu, Mono vs IIS/windows,
default serializer vs JSON.NET vs ServiceStack's, etc. There are tons of
variations that could be done.

~~~
platz
ServiceStack vs WCF would probably be a more apt comparison, but I suppose
there are people using SerivceStack for web apps as well.

------
hnriot
haha, all those hipster developers using rails can now eat the php guys shorts
:)

Seriously though, this isn't news to anyone that does this professionally. The
further up the abstraction curve you climb, the less performant the code will
be. Ease of development vs run-time performance.

~~~
oneeyedpigeon
Exactly. And since human time is much more expensive than CPU time, very few
people are writing their web apps in machine code.

------
saym
What I'd love to see paired with this data is a cost comparison. At what scale
does performance of ruby/python/php become cost prohibitive? Twitter made the
move from Ruby to Java some years back, did they ever post a comparison of
their numbers before and after?

Also, the difference between EC2 and local i7 hardware is glaringly obvious.
At what scale does owning the server hardware become imperative?

I know these questions are beyond the scope of a performance review, but
inquiring minds would like to know.

~~~
bhauer
As you might imagine, during this exercise, we've had a lot of conversations
about the points you raise. We have our own opinions, but we ultimately
removed most of that content from the blog post because we didn't want it to
be too editorial. We will be posting follow ups with some of our opinions.

Some things are really difficult to answer in a vacuum. If you already have a
competent devops staff, hosting your own hardware is probably beneficial. The
increased performance per "server" is substantial. But no devops staff? Then
it's either very risky or cost-prohibitive to own hardware.

~~~
ochoseis
I too was shocked at the deltas between Amazon and dedicated hardware. I think
AWS runs on Xen, so I wonder if you could strike a more optimal
performance/flexibility balance using lighter virtualization (LXC or OpenVZ)
with either in-house hardware or another VPS provider.

~~~
spartango
While it is true that EC2 runs on a customized version of Xen, it's very
unlikely that latency is being introduced by the hypervisor itself. With
paravirtualized kernel and drivers, Xen introduces negligible overhead, and
thus LXC would likely be no better.

The reason that the virtualized setup performs more poorly than the dedicated
setup is that you are fighting for CPU time with other AWS customers, so those
other customers are introducing latency into your application. Any
shared/virtualized host will have this problem.

Interestingly, where Netflix really wants to squeeze CPU performance out of
EC2 instances, they allocate the largest instance type so that they know that
there's nobody else on the underlying machine.

~~~
e12e
Regarding performance of Amazon, this post from 2009 is (still) very
interesting:

<http://uggedal.com/journal/vps-performance-comparison/>

Amazon performance is surprisingly low, but also surprisingly consistent. For
some use cases, consistency (knowing what you get you for what you pay) might
be worth a considerable hit in performance.

------
jfarmer
What would be much more interesting than "peak responses per second," which is
a weird metric to begin with, is the actual histogram of sampled throughputs.
Or at the very least a box-and-whisker plot
(<http://en.wikipedia.org/wiki/Box_plot>).

Most folks who have run Rails at scale, for example, find that the untuned
garbage collector in MRI (Ruby's default interpreter) introduces a large
amount of variance, for example.

~~~
bhauer
We don't have a box plot, but we have line charts for all of the data
(performance at multiple client concurrency levels). Just click the "All
samples (line chart)" tab on the data panels.

(Note that the very first data panel in the intro is an image and doesn't have
tabs.)

------
knes
No love for Flask? I would have love to see how it compare to Django and RoR

------
lux
I'd love to see how PHP 5.4 compares. In my own app, I saw a noticeable
speedup and RAM usage per request dropped by half.

------
codenerdz
One glaring ommision: DOS on Dope [1]

[1] <http://dod.codeplex.com/>

------
cloudwizard
It would be nice to see raw Python like you have raw PHP. I would expect
Django to perform very poorly unless you optimize it's caching.

------
goodwink
What was the parallelization like across these tests? Were they all running
single thread/process mode or did you try to take advantage of
threading/multiple-processes/etc. to optimize a production-like performance
across all the different frameworks? If the latter, that's a really impressive
amount of work!

~~~
bhauer
The short answer is that we attempted to use the CPU as fully as possible
across all frameworks. So for those that had tunable parallelism, we used the
settings that seemed best given the number of cores available on each platform
(2 for EC2 Large, 8 for i7).

We posted the deployment approach for each framework to the Github page.

------
saosebastiao
This is super awesome. I wish Yesod/Warp was available though :)

~~~
raphaelj
I've just send a pull request[1] with an Yesod implementation.

It runs pretty well, scoring similar to webgo for the JSON pong benchmark, is
almost at the same level as grails for the 1 query benchmark and is slightly
faster that Play for the last benchmark.

So, Yesod is in the same performance gap as Play or Grails, and is 3~4x faster
that Django or Rails.

But I've tested those on a mono-core Virtual Box, and I know Yesod scales
pretty well on a multi-threaded environment.

Also, keep in mind that most of the top performing frameworks are not fully
featured web framework but asynchronous I/O libraries (netty, go, nodejs,
vertx, ...) which implementations just write mindlessly the raw response
directly on the socket whatever the HTTP request was.

[1] <https://github.com/TechEmpower/FrameworkBenchmarks/pull/39>

~~~
bhauer
Thanks, Raphaelj! Pat (pfalls) will be in touch if he has any questions.
Really appreciate the contribution!

------
chasing
What's the difference between "php" and "php-raw" in some of the data? Maybe
I'm still in my morning fog, but having trouble thinking of what "php-raw"
might mean. Sigh.

~~~
jtreminio
The code says php-raw refers to using PDO (which all project should be using)
vs using an ORM or ActiveRecord.

I am completely stunned by the performance cost of using ORM/AR, and will be
using this to shame our team lead into giving it up and going for raw queries.

~~~
datasage
Some of the ORM cost can be mitigated by using caching. In most cases this is
essential in a production deployment.

~~~
bhauer
Agreed. However, for our database tests we expressly wanted to stress-test the
ORM and, where configurable, disabled caching.

We plan a subsequent test, time permitting, that enables caching.

------
sabe__
Isn’t it that django and rails has flat curves because they are by nature
blocking. So it wouldn’t matter how many request you throw at them as they are
limited in how much they can handle at once.

And to on top of this compare database requests is meaningless as the blocking
nature of the framework itself is the major bottleneck, and not the database.

Now, for a normal web application, the largest amount of request would come
from static content, or cached content within the web app so the real gain
would be a tiny fraction between python/ruby and go/java-based frameworks.

That said, if you want to handle static content (images and such) from within
your app, or build a java script-centric application with lots of tiny
requests, or even persistent once...nor rails or django would do.

------
larkinrichards
I roughed up a flask benchmark but I haven't been able to test locally.

If you want to help, check out this pull request:
[https://github.com/TechEmpower/FrameworkBenchmarks/pull/14/f...](https://github.com/TechEmpower/FrameworkBenchmarks/pull/14/files)

~~~
bhauer
Awesome, thank you! We'll aim to get this incorporated into a follow up post
soon.

------
thatthatis
A few additions/tweaks that would be great to see:

1) Add a raw http test -- no template compilation or html, just return "OK".
Would give a relative idea of the cost to "just turn the thing on"

2) Don't json encode after the database tests. By json encoding you're doing
two things but saying you're only testing one.

I come from the python/django world, and I know that different python json
packages have orders of magnitude differences in their performance. From that
I can infer that there are probably similar or greater cross-language
differences -- I suspect node.js having json as a native object type helps
immensely.

------
ryanslade
Why are there no Go results for the DB access benchmarks?

~~~
laumars
I'd be interested to know this as well. I can understand time constraints in
compiling these kind of comparisons, but as I'm about to embark on a web
project in Go, I'd be interested to know if there was any technical
constraints that prevented benchmarking Go.

~~~
bhauer
We recognize that deficiency and we do aim to add database tests for Go. If
you could quickly draft that up and submit a pull request, we'd love to add
it.

~~~
laumars
Sorry, but as much as I'd love to contribute I'm still learning how to write
database apps in Go myself (that's actually part of the reason I wanted to see
your tests). So I'd be the wrong person to to write code for that particular
test.

------
dmoreno
Very nice benchmark. As developer of a toy "framework", but powerful http
server library, I wanted to check on my project. I did a fast blog entry at
<http://bit.ly/10l9Smj>. I got ~58288.10 for simple json, 8594.39 on db using
sqlite, 503.23 for 20 requests.

Anyway for me its more important speed of development than performance on the
server. Maybe my servers do not get that many visits.

~~~
bhauer
Oh this is awesome! Very nice performance for a mobile CPU.

Pull request, maybe? :)

------
eCa
Would love to see how modern Perl frameworks (Mojolicious/Dancer) ranks in
such a test.

~~~
bhauer
Interesting thought. Would you be able to put together a test for one of
those? pfalls can give you the details for how we expect the test for a new
framework to work.

------
wldlyinaccurate
Cake was an interesting choice for a PHP framework to test. I wonder why they
didn't choose Symfony, which is arguably the leading PHP framework.

~~~
jasonhanley
Zend and Symfony are probably the leading OO MVC frameworks for PHP.

I'd also like to see a "light" framework meant for building APIs, like Slim,
for instance.

~~~
wldlyinaccurate
Yes, I'd definitely like to see Slim tested as a light framework. People keep
mentioning Silex as a "light" framework, but I've found it to be almost as
heavy as Symfony2 (which isn't much of a surprise considering it's just
Symfony2 components strapped together).

------
waxjar
Im surprised to see that Sinatra on JRuby often performs worse than Sinatra on
MRI, while Rack on the other hand performs much better.

~~~
bhauer
We were surprised by the sinatra-jruby tests as well. If you read our
"expected questions" section, it's not clear to us why Sinatra's performance
on JRuby was weak. We'd love to hear from a JRuby expert about how to address
Sinatra's "wrong" looking numbers.

~~~
ssmoot
Maybe the web-server choice (I didn't look to see what that was). But last I
looked at it, the Sinatra router really is very naive/awful.

For the size of the framework, that may be the right choice ultimately.

On the other end of the spectrum is Rails, who've written their own Regexp
engine for routing basically. It seems to be generally quite a bit faster.

All major Ruby frameworks have just awful routers though really.

------
gruseom
It would be very interesting to see OpenResty (Nginx + Lua) in there, since
it's so different from other approaches.

~~~
xt
I would love to see Openresty tested, for simple json and databases Openresty
is a magnificent tool.

------
jtreminio
I would love to contribute a code for a specific PHP framework: Silex. Do you
have requirements you'd like to hit?

~~~
bhauer
That would be fantastic. Get in touch with pfalls on Github and he can give
you the information. It's fairly simple.

~~~
wyck
+1 this would be interesting given that there are a lot of PHP frameworks
besides cake, namely.

Silex/Symfony2

Yii

Zend

Kohana

Fuel

Laravel

~~~
johnx123-up
I'll be happy to see benchmarks with PHP frameworks written as a C extension:
1. Yaf PHP <http://www.yafdev.com/> , 2. PHP-ActiveRecord++
<https://github.com/roeitell/php-activerecord-plusplus>

------
rombdn
Very interesting! But isn't there a disavantage for node.js considering it's
single threaded? Did you cluster it to use every core on the box ?
(<http://stackoverflow.com/a/8685968/1909827> \- "Scaling throughput on a
webservice")

~~~
saym
From the OP's response to a similar question:

>We attempted to take advantage of threading/multiple-processes as best we
could (see the nodejs code for an example of using the cluster module). But we
suspect there are additional areas of improvement here.

~~~
rombdn
Thanks

------
egonschiele
Since a few people have asked about Haskell, here are some Haskell benchmarks
in comparison: [http://www.yesodweb.com/blog/2011/03/preliminary-warp-
cross-...](http://www.yesodweb.com/blog/2011/03/preliminary-warp-cross-
language-benchmarks)

~~~
nickknw
Still, it would be nice to have these results independently verified and put
alongside all the rest.

What do you think, bhauer? :)

~~~
bhauer
Our to-do list is getting unwieldy. :) We would love to get this added,
though.

Any chance a Haskell expert could create a test and submit it as a pull
request?

~~~
nickknw
It sure sounds like it is!

I'm a far cry from a Haskell expert unfortunately, so I'll have to leave it to
someone else.

------
olegp
What I've also found is that Node (V8) tends to have much higher variance in
response times when compared to Java Servlets (JVM):
<https://github.com/olegp/common-node#benchmarks>

------
kenyee
In the FAQ for the benchmarks, could you also add a bit about how you
configured the JVM (OpenJDK?). Curious about heap settings, etc.? Did you just
use the default install? e.g.
<http://planet.jboss.org/post/rhel_openjdk_performance_tuning>
[http://www.jaspersoft.com/sunopenjdk-jvm-garbage-
collection-...](http://www.jaspersoft.com/sunopenjdk-jvm-garbage-collection-
tuning-tutorial)

------
codebeard
>And let us simply draw the curtain of charity over the Cake PHP results.

Something about that phrase "Let us simply draw the curtain of charity..."
resulted in an immediate spittake. Coffee everywhere.

------
neya
Thank you for such an informative comparison. You've helped shed light beyond
the frameworks that are known to most of us - Ruby on Rails, Node.js, Django,
Express, etc...

Thanks again! :)

------
bhauer
In case it's not obvious, you can hide frameworks/platforms of no interest to
you to narrow the view. For example, here's Node.js versus Express:

[http://www.techempower.com/blog/2013/03/28/framework-
benchma...](http://www.techempower.com/blog/2013/03/28/framework-
benchmarks/#hide=1,2,3,0,27,7,30,11,12,13,15,14,17,16,18,19,21,20,22,23,24,25,26,29)

(Note the initial chart in the introduction section is an image and it won't
be affected.)

------
just2n
Quickly taking a glance -- these benchmarks (like most benchmarks) seem like
they might be highly misleading.

For instance, in the Express example code, they're sending JS objects rather
than serializing them to raw data that the socket can just send. Instead,
serialization/copying are happening on each request, which is a significant
overhead.

I don't have domain knowledge over many others, but I suspect a similar
problem might exist with others.

------
vinay_ys
For json serialization, this benchmark seems to indicate Netty is twice as
fast as Golang. Don't think that's right. The golang benchmark code is not
exactly equivalent to netty benchmark code.

The netty code creates the ObjectMapper once and uses it for all requests,
whereas golang code creates the json encoder for every request (enc :=
json.NewEncoder(w)). Just getting rid of that would make this trivial code so
much faster.

------
HallofFamer
Also I dont think evaluating json based on a simple 'Hello World' is a valid
test. In fact, 'hello world' should never be the standard to test the speed of
any processes. If all your script does is to output 'hello world' to the
screen, why bother using a server-side programming language at all? You are
better off just writing it in HTML, its faster than using any server-side and
client-side languages lol.

------
asbjornenge
It just goes to show that your framework of choice is pretty unimportant. The
developers (and their skills), who are actually developing on and configuring
the framework, outweigh the "baseline" for performance on multiple orders of
magnitude. From that I'd say that developer friendliness (easily approachable
concepts) and sensible defaults is by far the most important thing when
choosing a framework.

------
iamjem
These sort of "benchmarks" are so subjective that they really don't provide
any value. Looking at the framework lineup, a lot of the frameworks at the
bottom have so many (tightly coupled) features (which also lead to bloat, like
excessive middleware layers) that they're obviously not going to perform
anywhere near as well as some of the barebones frameworks at the top. Neat
post, but yeah, zero value.

------
pacomerh
Go's performance is great, and hearing good things about it, does any one have
any experience on the type of applications this language is meant for?.

------
baghali
We need a project similar to TodoMVC [1] for benchmarking different languages
and frameworks.

<http://todomvc.com/>

------
bioinfo
I think this benchmark is not that useful.

If you seriously thought that the difference between a netty and django would
be 4 times then you simply don't understand what these frameworks do in the
first place.

I would have guessed something like less than 100 times slower and that would
have been still fine since the cost of all kinds of latencies in the system
usually far outweigh the speed of the framework itself.

------
systematical
These are some brutal results for CakePHP (which I use). However, in practice
no intermediate developer would be issuing queries like that in a loop. They
would loop through a set of data, building a IN statement example: WHERE field
IN ('x','y','z'). Thus only sending a single query to the database. Still, the
Cake developers really need to improve the speed of their framework.

~~~
pekk
I keep seeing numbers like this in benchmarks over the last few years, it
would be great to see CakePHP's numbers "rehabilitated" using techniques like
you discuss.

~~~
systematical
It's a fair way of testing, its just not the way a developer would write it in
practice. Clearly disregards the "Big O".

------
programminggeek
I would like to point out that there is no reason you would benchmark a raw
SQL query in PHP, and then only use ActiveRecord on the ruby side. At the very
least you could have used the Sequel gem to just build a MySQL query for the
Sinatra app. ActiveRecord is going to give you the same performance failure
that it did in PHP too.

------
HallofFamer
Regarding PHP, well, isnt CakePHP by far the slowest major PHP framework? Why
dont you test the other PHP frameworks? How about Yii, Codeigniter, Lithium
and Symfony? Id say the result is kinda biased for PHP frameworks since you
pick the slowest PHP framework available, instead of the fastest or at least
the average one.

------
fatbat
I see alot of people saying how certain configs are missing/better/not tested.
So what about a "crowdsourced benchmark" instead?

Maybe a few elaborate scenarios are posted and people can simply submit their
best setup/config/code to be benchmarked. I imagine devs would improve on it
over time and eventually the most optimized would surface?

~~~
kainsavage
We are doing precisely that: if there are any mistakes or misunderstandings
about the production-level best practices, we are accepting pull requests on
our github (set up with all these tests, in case you want to run them
yourself) to fix them, rerun them, and then report our updated findings.

------
reactor
Not to nitpick, but that (and another popular one
[http://www.techempower.com/blog/2013/03/26/everything-
about-...](http://www.techempower.com/blog/2013/03/26/everything-about-
java-8/)) was a clever way to get people to techempower.com :)

Useful comparison anyway. Seems Go struck a good balance.

------
Kiro
So this means I should stop using CakePHP?

~~~
asadkn
Not really. Take this benchmark with a grain of salt unless they benchmark it
again with APC at minimum. Nobody serious about a web app would run a PHP app
without APC.

Similarly, relatively modern deployment standards (not really cutting-edge)
like nginx and php-fpm OR apache 2.4 with worker mpm + php-fpm should be added
to the mix.

~~~
bhauer
Asadkn, we'll try to get things revised based on your feedback. Admittedly, we
have some learning to do with tuning production PHP environments.

~~~
beryllium
I posted a note above that might help:

apc.enable=1 turns on Opcode caching (when php-apc is installed), and
apc.stat=0 turns off "stat" checks - this means that once a file is opcode
cached, PHP won't even have to touch the file on disk to execute it. The I/O
gains from this, as well as the execution gains from not having to parse the
file, should help quite a bit.

------
lobster_johnson
Some of the choices are unfair in an apples/oranges sort of way: They are
testing Rails and Sinatra against Java servlets, but the Ruby equivalent of
servlets is not a high-level framework like Rails, but Rack. (Also, the Ruby
tests use Passenger, which is probably not a great choice for performance.)

------
grannyg00se
Probably not a major factor but I'm curious to know why mongoose was thrown
into the mix for the node.js test rather than going with the native mongo
driver.

It might be more of a real world test to include mongoose with the
node+express test, but for the node-only test the native driver might be more
appropriate.

~~~
pfalls
We wanted to use an ORM in all cases, it was only recently that we started
working on native mysql access. We hope to add native mongodb results as well.

------
mmcclellan
The wsgi benchmark uses gunicorn, which is fine. But if you are surprised by
the so-so performance, know that the worker class can be changed via the -k
flag (-k gevent), and may improve performance. Of course, other wsgi servers,
like uwsgi, are also available.

~~~
knappador
Are they not even using gevent? Someone please get these guys off the
internet. The are worse than some personal blogs at everything except the
comprehensiveness in their low-quality testing.

------
codegeek
Nothing for Flask ?

~~~
stevenleeg
Yeah I was disappointed to see a lack of flash as well, especially since it
advertises itself as a microframework and all.

~~~
bhauer
We received a Github pull request with a Flask test! So we'll aim to include
that in a follow up post soon.

------
superuser2
It's interesting to note the time per request when you invert these. Even some
of the most drastic differences (like Spring vs. Django) turn out to be the
difference of a millisecond or two. That's pocket change in terms of user
experience.

~~~
jonathanhefner
It is a good and interesting observation, but as the benchmarks approach real-
world scenarios, the difference is more than a millisecond or two.

For instance, if you're running on EC2 hardware (which is common enough) and
you're executing ~20 DB queries/request (which is probably, unfortunately,
common enough), the difference between Java servlets and Rails is more like
10ms.

Then what happens when more than 89 real-world users start to hit your Rails
server each second?

(Note: I really like Ruby and Rails. Much more so than Java and its
offerings.)

------
cogwheel
How much of the difference is due to the random number generator calls? They
aren't free or equal cost across platforms by any stretch. I really have a
hard time trusting these numbers with how casually calls to RNGs are bandied
about...

------
neya
For anyone else curious on how Vert.x stacks up against the famous Node.js:

[http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-
node-...](http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-
simple-http-benchmarks/)

~~~
nyam
you haven't read discussion, right?

~~~
neya
I posted this way before the discussion happened.

------
lennydizzy
How much does 'asynchronous' matters for the performance of a web framework? I
mean netty, node.js are all async frameworks as I understand, but the source
code for Go and Compojure seems was using a thread/process per request model?

------
NicoJuicy
I kinda miss asp.net web api (okay, it's on Windows, but it's also a framework
:))

------
greenlaw110
Why Play benchmark is running on Resin? Shouldn't it be directly on top of
Netty?

~~~
bhauer
Play is not running on Resin. Did we accidentally say that somewhere? If so,
we'll correct it.

~~~
sharedliberty
Here:
[https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/play-
java)

The tests were run with:

Java OpenJDK 1.7.0_09 Resin 4.0.34 Play 2.1.0

~~~
bhauer
Whoops! Fixing that up presently.

Thanks for the catch.

------
33degrees
I'd be curious to see how ruby fares using a setup other than apache +
passenger; I have a feeling that nginx with unicorn or puma would provide
better results, and might give it a go when I have a bit more time.

------
eamodeorubio
Hi, great work! I only have two suggestions:

a) Please, publish not only the requests per second, but the memory and CPU
usage of the machine for each framework. b) For JAVA systems, can you publish
de heap configuration of the JVM?

Cheers!

------
jrussbowman
wow, these results and the performance of go is making me consider rewriting
my tornado app again. I just put off doing it in order to build more stuff in
to the python version. Maybe I'll reprioritize.

------
obsurvey
If I wanted to return a bunch of _ids from a mongo database i would do a
_id:$in[array of ObjectIDs] and then stream the resulting cursor to res.
Streams are core to node. You should use them.

------
richyzhang
JVM can use many cores at the same time node/rails only use one core. The test
is totally nonsense.

You should show us the cpu usage. If you had done so, you would have found how
absurd the mistake you had make.

------
Yaggo
The raw json output merely tells about the performance of the language itself.
Nice, but not that interesting if you need to get the job done and don't
happen to know 10+ languages/frameworks.

------
purplefox
I haven't looked at this detail yet, but it should be noted that we haven't
even optimised Vert.x yet so there should be plenty of scope for further
improvement :)

(Disclaimer: I'm the Vert.x project lead)

------
bhanks
I guess it is a good thing we are finally migrating away from Cake :/

------
dugmartin
I wonder how a cgi binary run from xinetd would perform?

------
suyash
Java (JVM) > JavaScript (Node.JS) > Ruby-Rails

~~~
chc
If all you're doing is trying to serve JSON as fast as possible.

------
rbanffy
I still believe launching a usable web application next week is preferable to
launching a really fast web application 3 months from now, if ever.

~~~
arkitaip
That's a straw man. Competent programmers should be able to code fast code in
a fast manner.

~~~
rbanffy
Different languages and platforms yield very different productivity levels.

------
mariust
I am 99% sure that you haven't used contain=>false for cake php this way all
related tables are loaded to serve the related data.

------
niteshade
Can't believe you didn't include Zend in there :(

~~~
craigching
Just read this above:

"We've got Zend as a next target for PHP frameworks."

------
sGrabber
Thanks. We are reconsidering our stack and are in middle of evaluation, I am
sure this post is going to be of great help.

------
mschuster91
You're using PDO in the PHP raw test - use direct mysqli_* API, it's lot
faster than the PDO abstraction layer.

------
redwolfb14
Wow, this benchmark is like comparing apples, oranges, mangos and cherries.
Just because they are all fruit.

~~~
ldng
It's fine as long as people take it for what it is. I still find the fruit
salad interesting :-)

------
donebizkit
Very useful info. Thanks for sharing.

------
eweise
Was Play 1 or Play 2 tested? I thought Play ran on Netty so I'm surprised it
didn't fair better.

~~~
jfim
They're using Play 2.1 with Java, without using futures for database queries.
Not sure how representative of real usage that would be, though.

~~~
bhauer
We've got a bunch of feedback that the Play database test needs to be
asynchronous, so we'll make that a priority for our next run. Perhaps I can
entice you to submit a pull request to improve the Play configuration?

------
gary4gar
Woah....PHP is faster than Ruby in lot of results? I wonder why is this the
case?

------
mixmastamyk
Wasn't it agreed a long time ago that benchmarking against a VM was a bad
idea?

~~~
bhauer
Possibly, although we believe it represents precisely what we wanted to test:
a realistic production environment.

However, we also tested on our physical i7 hardware. Did you happen to scroll
down? :)

~~~
mixmastamyk
I did, but didn't see a pretty graph for it. Full production results are
interesting, but benchmarks should test on a single variable. A cloud VM adds
several to the mix.

------
JOnAgain
This is awesome. Thank you.

------
bjouhier
node.js is single threaded and EC2 large instance has 4 CPUs. So to be fair
you must either:

a) bench on a single core processor (a small EC2 instance)

b) configure node.js as a cluster with as many instances as processors.

------
philliphaydon
Needs more NancyFX + Mono! Nice post tho, very interesting.

------
dthakur
Could you kindly add error bars to your test results?

------
a-b
I would love to see Dropwizard in this benchmarks!

------
cdi
What about C++ Web frameworks?

~~~
bhauer
Don't think we didn't consider it! :) If you are willing to craft a comparable
test for a C++ framework, we'd definitely like to get one into the next set of
numbers.

We especially don't feel confident in our C++ chops to do a C++ framework
justice.

------
kbar13
...flask?

------
abimaelmartell
im not a PHP nazi, but you can't compare PHP and a framework

------
zeeg
You guys should feel bad about generating such awful benchmarks.

How the hell can you compare accessing a MySQL database to accessing a MongoDB
database?

Its like comparing apples to piles of poop.

Also when you're testing things like Django in web requests, you're testing
gunicorn, not Django.

