
Web Framework Benchmarks Round 2 - pfalls
http://www.techempower.com/blog/2013/04/05/frameworks-round-2/
======
bhauer
This is our first follow up to last week's web framework benchmarks. Since
last week, we have received dozens of comments, thoughts, questions,
criticisms, and most importantly pull requests. This post shows data collected
from a second run on EC2 and i7 hardware that started on Tuesday of this week.
A third round with even more community contribution is already underway.

Thanks especially to those who have contributed! We hope this is useful
information.

~~~
giulianob
Did you guys turn on byte code caching for all the PHP frameworks? If not,
then I recommend everyone ignore these benchmarks until that is completely
done.

~~~
kainsavage
Yes, we are using APC for all the PHP tests in this round of benchmarking.

~~~
keammo1
Is there somewhere we can see the settings for APC?

Thanks for the great work!

~~~
krg
[https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/config)

See php.ini, php-fpm.conf.

~~~
keammo1
I could be missing something, but it looks like you are using the default
settings. Have you tried tweaking it all? Specifically setting apc.stat=0,
which will stop it from checking the mtime. You'll need to clear the cache
with apc_clear_cache() when you make code changes though. You may also want to
look at apc.php to check for fragmentation and adjust apc.shm_size if
necessary

~~~
ebiester
fork, benchmark, and create a pull request. :)

~~~
keammo1
I'm hesitant to just change the config to apc.stat=0, since I can't ensure
apc_clear_cache() will be called during deployment

I'll add in apc.php to help make sure other things are tuned properly though.
different settings could be appropriate for different servers...

------
thatthatis
This is still a json API comparison, not a full framework comparison.

In the first test it sees how fast frameworks can serialize json and in the
second test it sees how fast frameworks can access the database AND serialize
json.

An inefficient json library and a framework is doomed across all tests. There
are a lot of things frameworks do beyond serving json. These benchmarks are
presented as representative of broad use when really they are only
representative of the frameworks if used as a json api.

A raw HTML hello world, and removing the json encoding step in the db test
would go a long way towards getting results that can mean what the authors
seem to want them to mean.

There's nothing wrong with a json API benchmark, in fact its quite valuable in
a lot of cases. But if that's what this is intended to be, the authors should
say so.

~~~
logic
Agreed; for example, in our (in-house) use case, switching a django app from
using either stdlib's json or simplejson to using ujson[0] was a significant
performance increase when serializing/deserializing large-ish (~100MB) JSON
datasets.

[0]: <https://pypi.python.org/pypi/ujson>

------
mdlthree
As an amateur hacker and after 275 days of reading Hacker News I feel I can
now navigate in the sea of client side Javascript frameworks. With the
introduction of this data on web framework performance, I now have another set
of choices that I am completely unqualified to comprehend.

My first impression is that this data shows about 3 levels of web frameworks.
At the bottom (slowest) we have Django's and Rails and many other introductory
app server frameworks. Lets say after I was able to build an initial product
successfully, would I then consider re-building the product in a higher
performance framework as Go, Node, etc?

The third level, netty and gemini and servlet etc, I am not familiar with.
Googling "netty" I get --"Netty is an asynchronous event-driven network
application framework"-- I thought that is what Node is (in js) and Go does
with gophers.

What are the use cases for these faster frameworks and do they follow an
evolution of performance options that an app might go through?

~~~
Joeri
Netty, gemini and servlet are all java server solutions without a thick
framework layer getting in-between. What you can tell from these benchmarks in
my opinion is that java (or rather the JVM) has the best "raw" performance.
Frameworks introduce a slow-down factor on top of the raw platform, and
different frameworks have different slow-down factors.

You have to trade off developer productivity for performance when it comes to
choosing frameworks. The slower the framework the more it does for you.

~~~
cmsimike
Gemini is a full-featured web framework. We've built it and maintain it
internally. It offers caching (though not enabled for these tests),
lightweight ORM, bunch of default handling methods and plenty of other
features.

~~~
kolev
So you did all this to tell us indirectly that you have the best web framework
in existence? ;)

------
mythz
Are C#/Mono web frameworks an option? Because I would love to see how
<http://servicestack.net> fares:

[https://github.com/ServiceStack/ServiceStack/wiki/Real-
world...](https://github.com/ServiceStack/ServiceStack/wiki/Real-world-
performance)

~~~
voidlogic
I agree, I would like to see C# Mono included as well. People might also want
to see C# .NET, but I don't think that is applicable here because we are
talking about 64-bit Linux (the dominate internet platform) here.

~~~
mythz
Sorry I don't understand why it's not applicable? Mono is how you run
C#/F#/.NET on Linux?

~~~
voidlogic
I wasn't saying Mono wasn't applicable, I WAS saying .NET was not
applicable....

~~~
mythz
ok cool, was just a little vague, you should've mentioned 'Windows' as C#
(even F# <http://www.servicestack.net/mythz_blog/?p=785>) is cross-platform
and runs on OSX/Linux with Mono.

------
just2n
Once again, the Node examples are wrong.

\- Unnecessary parsing.

\- Writing strings instead of buffers (they're copied, they aren't sent as-
is).

\- Using async. It does a lot of really nasty things, most of which break V8
optimization best practices. This is a perf benchmark, not a comparison of how
concise your code can be.

\- Having the main request handler in a gigantic function that will never be
properly optimized by V8.

\- Not getting helper functions that are clearly monomorphic warm before
accepting requests.

While the code itself is what I would consider fine, when benchmarking against
strongly typed compiled languages, performance concerns become important, even
if you have to write ugly code.

~~~
kaoD
On the one hand, you're completely right.

On the other hand, shouldn't real code be tested? How accurate would a ugly-
optimized-code be when you SHOULD write reusable code in your real life
projects?

~~~
just2n
I wouldn't use async myself, so what's "real" code varies from person to
person. This is why benchmarks like this are highly misleading.

But ideally, yes. We'd be better informed to see how real world code performs
rather than some idyllic perf-oriented creature.

------
voidlogic
It might be interesting to track average and peak server side memory usage
during these tests. For example, if framework A is only 80% the "speed" of
framework B, but uses 1/4 as much memory, for some users this might be a win
for A.

~~~
kainsavage
This is definitely something we want to include in the tests, but it is
difficult to manage a fair way of monitoring. Outside of the actual
benchmarks, where we kept everything as fair as possible, we ran individual
tests to make sure they worked prior to the benchmarking and would routinely
have htop running at the same time to get an idea of what that sort of data
looks like.

It IS very interesting and is definitely something we have discussed methods
for measuring for the sake of these benchmarks. Other areas that we discussed
were cpu utilization, disk utilization, and network saturation.

~~~
kalmar
One idea that jumps to mind is putting the running frameworks in an LXC
container (or similar) and monitoring the memory usage of that. Not sure how
accurate that is, but it's one avenue. It also still might not be fair because
some runtimes can let their heap get pretty big despite not actually needing
the memory. You'd want a way to differentiate those from the ones with big
heaps that would choke with some memory pressure.

~~~
voidlogic
Just make sure you measure resident memory not virtual memory :)

------
Zak
A note about Compojure: they're using Korma for database access. Korma is an
abstraction layer that's more Clojure-friendly than using SQL directly, but
lighter than an ORM. Using clojure.java.jdbc might produce higher performance.

~~~
krg
Where possible on the tests, we tried to use something other than raw SQL
because we wouldn't expect someone to use raw SQL everywhere when writing a
real application. We chose Korma because it seemed decently popular for
Clojure, but we're certainly open to a "compojure-raw" test that uses
clojure/java.jdbc. Feel free to submit a pull request. :)

~~~
Zak
I'm not saying Korma isn't a good or realistic choice. I use it myself. There
did seem to be some disagreement in freenode #clojure about whether that was
the best way to go, or JDBC and prepared statements. Evidently, a lot of
people actually do _like_ SQL.

It's just important for people to be aware that "Compojure" actually
represents "one possible production-ready stack that uses Compojure as the
routing component", as do several other items on the list. I don't mean that
as a criticism - just clarification.

------
milesf
It's a useful data point to know the speed of frameworks, but what matters
most to me is speed of development and whether or not I enjoy the process.

Wish there was a way to benchmark those points.

~~~
columbo
>what matters most to me is speed of development and whether or not I enjoy
the process.

Matters most? There are 100x differences here. Take two companies writing the
same application, one in Spring the other in Rails. Both have easy access to
knowledgable people, both are industry standard. However the Spring
application would be several orders of magnitude more scalable[1]. I'd
consider that to be a more important consideration over enjoying the language.

[1] All benchmarks are suspect until proven otherwise. You can write slow
software in Java/C/C++, you can write fast software in Python/Ruby etc etc etc

~~~
wyqueshocec
From my experience, the Spring one would take 10x as long to make, which means
more developer cost and more feedback reaction time. Certainly not a deal
breaker, but a cost/tradeoff one must consider.

Edit: clarity.

~~~
rufugee
As someone who recently came back to Java from a 5 year stint with Rails, I
can safely say this is not true by any stretch of the imagination. If you
stick to those tools in the Java ecosystem which meet your requirements, you
can be incredibly productive.

The problem is that the Java world is full of over-engineered solutions which
are really targeted at extremely large enterprise applications surrounded by a
community of folks who like to menturbate about them. As a startup, you _don't
need to focus on those solutions_. You'll not need 99% of them.

However, you can choose bit and pieces. I'm using Vaadin, JPA and Guice and
finding it incredibly productive and am able to deliver functionality in far
less time than it took me in Rails. I'm not using 90% of the Java EE stack,
but it's available to me if/when I need it.

Rails was a great thing for the software development world. It drove
convention over configuration and caused a lot of the competing technologies
to pause and question why things were the way they were. But IMHO, managing
the whole Rails stack (rails + coffeescript + css in our case) across a group
of people became a chore. YMMV, but the statically-typed nature of Java along
with the very capable technology stacks you can assemble have eased our daily
jobs tremendously.

~~~
elechi
How are you finding Vaadin work out for you? What are you using it for? What
version? The 6 series or the new 7 series?

~~~
rufugee
So far, so good. It's very different from your typical template-driven
approach. We have a decent amount of Swing experience, and developing with
Vaadin feels similar (but much easier) than Swing.

If you are developing business-facing applications and not just consumer web
sites, as we are, I think Vaadin makes a lot of sense. It might even make
sense for consumer sites, but that's not our current target.

~~~
rufugee
I should also say: putting together a new screen (form or otherwise) in Vaadin
is _extremely_ fast. It's the fastest I've ever been able to develop web-based
UIs, simply because all I'm forced to think about is Java.

~~~
elechi
I see. I found it to be a little difficult in terms of dealing with data,
especially messing around with the Table object. Best of luck to your project.

~~~
rufugee
The JPAContainer and SQLContainer it provides makes this reasonably easy.

------
kevinburke
It seems like you're still using Mongo DB as the datastore for some languages
and Mysql as the datastore for other languages. This seems like it would bias
the results.

~~~
pfalls
The only framework that uses MongoDB exclusively is the vert.x test, the
nodejs tests have tests for both MySQL and MongoDB

------
alberth
I wish Lua was tested in the benchmarks.

In particular, Lua-CJSON [1]

[1] [http://www.kyne.com.au/~mark/software/lua-json-
performance.h...](http://www.kyne.com.au/~mark/software/lua-json-
performance.html)

(I know, everyone wishes their favorite language was represented and not all
can be tested. I'm just posting this in case the OP does a "Round 3")

EDIT: fix typo

~~~
pfalls
There will most certainly be a Round 3, and 4 and on and on as long as we
continue to have feedback from the community. I can guarantee that we'll have
a Lua test in there, but if we receive a pull request, we'll include it.

------
nlh
This is an awesome project, and thank you for the follow-up.

In particular, this is awesome because it's introduced me to some new
frameworks that I hadn't even heard of (ie vert.x) that seem extremely
interesting.

Keep it up and many thanks for this contribution!

~~~
pfalls
Thanks for the kind words. Glad to hear that you've been introduced to some
new frameworks, we feel the same way, and we're excited to see what else the
community has up it's sleeve.

~~~
diminish
Pls could u include some C/C++ web frameworks... comparison with Java, could
be interesting..

~~~
krg
We're interested in that, too. In progress is the onion http library pull
request from davidmoreno:

<https://github.com/TechEmpower/FrameworkBenchmarks/pull/43>

We hope to get more as well.

~~~
spyder
It would be interesting to see G-WAN too because it seems controversial.
<http://gwan.ch/>

------
voidlogic
It is worth pointing out Go 1.0.3 not Go 1.1 beta is being used. I tested the
code locally and on my machine 1.1 beta was about 20% "faster" (req/sec).

~~~
cmsimike
Really? We're definitely going to try that out!

~~~
voidlogic
In addition to the improvements from the improved scheduler, use of accept4 on
Linux, better GC and code generation, Brad Fitzpatrick has been giving a lot
of love to the "net/http" package. Here is a small example:
[https://plus.google.com/u/0/115863474911002159675/posts/L3o9...](https://plus.google.com/u/0/115863474911002159675/posts/L3o9hEs8SAe)

------
superuser2
This is definitely motivation to learn another framework besides Rails and
Django.

Compojure's syntax and design seem really attractive... is anyone using that
in production? Would it be a reasonable choice for a "serious" web
application?

~~~
Zak
I've used it for client work that was in production and for a side-project
that's in production, but private with few users. It has been reliable for me.

Note that Compojure is more of a routing library than a framework. It's
similar to Flask or Sinatra. Clojure's philosophy includes aggressive
separation of concerns such that full-stack frameworks aren't really "the
Clojure way".

------
saosebastiao
Yesod disappoints! Unfortunate for how much it is hyped. This is definitely
good to know, even if it appears that the Warp server was not used.

~~~
pilgrim689
That's the first thing I noticed. What was used instead of Warp?

I also see that Snap got pulled in a couple days ago, so it will probably get
tested in the next round. Should be interesting to see how it fares against
Yesod.

~~~
mightybyte
The way the benchmark is constructed, it's really more about JSON and DB
performance. Snap makes no decisions for you about those things, so the
numbers really won't reflect the performance of Snap all that much. I don't
have great expectations for the "Snap" code that is in there now because it is
not using very fast JSON and DB libraries.

------
Uchikoma
Very interesting - again.

What I - again - find most interesting, is the large discrepancy in the
20-query test. Common wisdom seems to be that language performance is not
important b/c every language hangs in the same IO/DB.

~~~
malachismith
probably a combo of mysql drivers and connection pooling.

~~~
Uchikoma
I wonder why Play is so bad then.

------
tantaman
Does anyone know what on earth is Play! doing that makes it is so slow?

I'd like to see Lift in there if you guys do another round.

~~~
maybenot
The Play JSON controller is using Futures to do JSON serialization.

[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-
java/app/controllers/Application.java)

Compare this to the Java Servlet, which just serializes directly in the
Servlet callback.

[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/servlet/src/main/java/hello/JsonServlet.java)

The former seems like overkill.

~~~
redtuesday
This is not the version which was used in the tests. The tests used a earlier
version.

------
VeejayRampay
Gemini conveniently leads the pack all the time. So either OP and his/her
colleagues have revolutionized web programming or the benchmark is nothing but
a PR piece.

~~~
krg
Just ignore Gemini. In fact, you can click the "hide" link on the charts to
remove it.

From our first benchmarks post:
[http://www.techempower.com/blog/2013/03/28/framework-
benchma...](http://www.techempower.com/blog/2013/03/28/framework-benchmarks/)
"Why include this Gemini framework I've never heard of?" We have included our
in-house Java web framework, Gemini, in our tests. We've done so because it's
of interest to us. You can consider it a stand-in for any relatively
lightweight minimal-locking Java framework. While we're proud of how it
performs among the well-established field, this exercise is not about Gemini.
We routinely use other frameworks on client projects and we want this data to
inform our recommendations for new projects.

------
jimmytucson
Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other
Python/Ruby frameworks?

If not, can anyone explain in layman's terms why that should be the case?

~~~
jcroll
Why shouldn't that be the case? PHP has been designed from the get go to be
super fast and 5.4 introduced a bunch of optimizations that have sped it up
even more so. Ruby was designed for programmer happiness, PHP was designed for
performance and getting sh*t done.

~~~
pdeuchler
PHP technically wasn't designed

------
boundlessdreamz
This is superb.

1\. More sorting options please :) I really want to sort by standard
deviation.

2\. Also in case of rails and other single threaded frameworks are you
spinning up multiple processes?

3\. Why is Go not being tested in the Database access tests?

4\. Can you test rails with unicorn too?

~~~
bhauer
1\. Blame me. I put together the charts and tables. I'll try to get some
sorting added for the next round! :)

2\. Pat (pfalls) should know the answer to that more definitively, but we are
definitely trying to use the CPU cores as fully as possible. I think you'll
see some comments by Pat elsewhere in this thread suggesting some things
coming in round 3.

3\. Good question! Mostly because we haven't had the time to add it. I would
really like to see it added, though. If you know Go and can write the code,
might I entice you to submit a pull request?

4\. I believe that is the plan, yes.

------
BryantD
I apologize if this question has been asked and answered, but why is netty
missing from the database tests? No ORM?

~~~
pfalls
The short answer is that at some point, we kept adding more and more tests on
our own, which kept the blog from going out, and we finally decided to have a
cutoff date and let the community continue adding tests that they were
interested in. I think adding a database test to Netty (as well as Go for that
matter) would be a great addition.

~~~
BryantD
Thanks! And thank you very much for this work -- it's tremendously useful.

------
redtuesday
Thanks for the tests. Especially the latency is interesting.

I cant't wait for the next round, when the updates of the last two days are
included (elli, grizzly, play-java, play-scala, snap etc).

~~~
pfalls
Since we moved to a different benchmarking tool for round 2, we had to re-run
all the tests, which is a time consuming process, especially on the EC2
hardware. Now that we've gotten that out of the way, we have an easier time
running and processing results for individual tests. So stay tuned!

------
jeltz
I wonder where the bottleneck for the ruby benchmarks is. Is it in passenger,
the framework, or somwhere else? I am sure I have gotten higher requests per
second on my workstation when I tuned Ramaze (which should not be faster than
using raw rack). My guess is that it is either due to a different method of
measuring or the fact that I was using thin rather than passenger.

------
gourneau
As a Django guy with hurt feeling right now, I just wanted to say that this is
a great project. I look forward to the next versions.

------
polskibus
I would love to see ASP .NET and ASP .NET MVC included in the benchmark.

------
apendleton
I think I said something similar on the first round, but for the Python tests,
I'd be curious to see alternate runtimes and/or concurrency strategies: pypy,
gevent, and probably the confluence of the two (though it requires quite a bit
more hacking, since gevent doesn't officially support pypy).

~~~
pfalls
We're very curious about PyPy as well, and have it on our list
<https://github.com/TechEmpower/FrameworkBenchmarks/issues/15>

------
bsnyder788
I think that the play-java one would likely benefit greatly from using raw
jdbc rather than the ebean ORM. In fact my best guess is this is why the
scala/java versions are so far apart. In past projects, scrapping ebean always
led to large performance improvements in throughput, etc.

~~~
redtuesday
<https://news.ycombinator.com/item?id=5499839>

ps: i agree that raw jdbc would enhance the performance.

------
kriro
Cool stuff. Only looked over it quickly but it seems JAVA stuff is dominating.

Kind of curious why jruby doesn't provide more of a boost (especially given
compojure doing relatively well, too) and would like to see Django+Jython in
version 3 (and +1 to the .NET/Mono stuff)

------
abalone
Interesting that the Play code optimizations had virtually no effect. Almost
identical absolute scores between tests. Clearly something heavy's going on
within the Play request handling framework to slow things down, not the code
we see.

I was also surprised at the difference between the Java and Scala play test,
since I thought they were supposed to be similar. But it looks like the
approaches are quite different. The Java DB test uses the ebean ORM while
Scala does not. The JSON code also looks a bit different, with the Java
version surfacing more Jackson internals. I don't know if these are
necessarily "wrong", but perhaps it's not as apples-to-apples a comparison as
it could be.

~~~
bhauer
Agreed, and I would certainly like to see the Play tests (both Scala and Java)
improve versus what we have measured so far. I would not rule out a
configuration glitch in our deployment either. But on that front, I'm really
hoping the Play experts can lend a hand. I think we've received a couple tips
about the database connection pool size.

If the contributions we've seen so far are any indicator, we're going to need
to get more clever with how to show and hide rows in our results tables! :)
But I'd like to see a few more rows added to cover the various permutations of
the ORM and JSON options, as you point out.

~~~
abalone
Thank you Brian for this so much. Really just a fantastic effort and a great
contribution to the web dev community.

------
cheriot
As one would expect, as more data access is added, the scale of differences
between web frameworks goes down. It would be interesting if you added 5 and
10 db query test runs as that starts to get closer to what a functional
website would be doing.

~~~
bhauer
Hi Cheriot,

We do in fact have 5 and 10-query runs available in the data. Switch the views
to "Data table" or "All samples (line chart)" and you should see the data for
1, 5, 10, 15, and 20 queries per request.

------
coolsunglasses
Go 1.1 beta is up, _really_ need to get some results with tip instead of
1.0.3.

~~~
pfalls
Go 1.1 is very high on our list, we'd love to get that into the tests as soon
as we can. Of course, anyone in the Go community is welcome to issue a pull
request that updated the version to 1.1.

Edit: Just saw this pull request come in, so expect some Go 1.1 love in the
next round. <https://github.com/TechEmpower/FrameworkBenchmarks/pull/64>

~~~
TylerE
Don't think you'd need to change any code, just pull mercurial tip and
recompile go and the app.

~~~
cmsimike
Is 1.1 production ready? I'd feel weird about incorporating something still in
a "beta" phase though.

~~~
coolsunglasses
People have been working off of tip for months.

------
tracker1
Where's the source for the implementations? Of interest to me is the NodeJS
instances.. was this single-process, or multi-process via cluster (which is
recommended for higher performance). The only source I saw was for the testing
framework.

\--- edit: found it. <https://github.com/TechEmpower/FrameworkBenchmarks>

Seems to be using cluster. I'm not surprised that Go and servlets were
faster... just surprised to see NodeJS in the middle of the pack position.
Though performance isn't the only reason I really like NodeJS

------
mojomama
No love for Mojolicious? It would be great to see a little Perl here!

~~~
voidlogic
Submit your pull request to add it?

------
njharman
I find the lack of Erlang frameworks disappointing.

~~~
taeric
Instead, find it motivational. :) They are accepting pull requests.

~~~
krg
In fact we've already got a pull request that includes Cowboy and Elli:
<https://github.com/TechEmpower/FrameworkBenchmarks/pull/50>

We'll be including this in the next round.

~~~
chops
If I can find the time, I'll try to put together one for Nitrogen
(<http://nitrogenproject.com>) as well.

------
programminggeek
One of the variations that is being tested here, that I'm not sure if it is
legitimate is that for PHP specifically, Apache is likely spinning out many
PHP threads, I'm not sure if the same behavior is happening on the ruby side.
Thus, if you are comparing performance of 10 PHP threads to 1 ruby or python
thread, your benchmark is not going to be terribly accurate.

~~~
krg
PHP was tested behind nginx this time due to community feedback we received,
not Apache.

~~~
programminggeek
Awesome, I did realize that. I stand corrected. Love the work you're doing!

------
portmanteaufu
This is great stuff. I'd love to see some more frameworks written in compiled
native languages like Haskell and C!

------
kbd
What about Puma (<http://puma.io>), Goliath (<http://goliath.io>) or Unicorn
(<http://unicorn.bogomips.org/>) instead of Passenger in the Ruby stack?

~~~
pfalls
We're looking into moving to unicorn as the community has suggested. If anyone
is interested in setting up Puma or Goliath, we'd be interested in testing
those out as well.

~~~
cheald
For something like the query tests where you're IO-bound, Puma is going to
annihilate the competition. For CPU-bound tests like JSON generation, unicorn
with multiple workers is going to perform better.

It's worth noting that Rails' default JSON solution is the compatible-
everywhere pure-Ruby JSON generator. Using a drop-in replacement like the oj
gem will drastically improve throughput there.

I didn't get to the pull request for this round, but tweaking the GC
parameters for the Ruby install should dramatically improve the general
performance of the non-JRuby Ruby tests, as well. I'll see if I can get a PR
in. :)

~~~
ksec
Well yes please do this, not sure if it is allowed though since Oj gem is
written in C ( I think )

~~~
cheald
So is the mysql2 driver (at least the slow parts), and Passenger is C++. C
extensions are a common part of your standard Ruby deployment stack.

------
don_draper
I code in Ruby, Python and Groovy/Grails. I'm amazed that Grails performs so
much better than the others.

~~~
pfg
I guess that's one of the big advantages JVM languages have. Using stuff like
Spring or Hibernate (like Grails does), you benefit from thousands of person-
hours spent on tuning those libraries and you still get native Java
performance.

Obviously there's a price you have to pay for all the things Grails built on
top of Spring/Hibernate (in the benchmark it looks like Grails' performance =
Spring / 2), but in general, it's still faster than just writing everything in
Groovy (Ruby, Python, ...).

~~~
msgilligan
It would be interesting to see the results for Grails 2.2 (which uses Groovy 2
and Invoke Dynamic)

One of the nice things about Grails is that you can easily drop down to the
Spring and/or JVM level when needed for performance.

~~~
don_draper
Yup. I manage a Grails app where I wanted to speed up performance on a popular
part of the app. So I replace the Grails controller with a servlet. I saw a
bump in performance.

------
wheaties
Is it against the rules to use different workers for Django or Flask? In other
words, there's a big difference when I run things with
Gunicorn+Gevent+psychopg (PostGres async driver) than when I run it just with
Gunicorn alone.

~~~
pfalls
Each test can be modified in isolation, if both tests see an improvement in
that setup, we would like to include the changes for both though.

------
seunosewa
I wish you would have tested minimalist web frameworks in Python: CherryPy and
Bottle. Flask is not really a micro-framework in the same way that CherryPy
and Bottle are micro-frameworks because of its numerous dependencies.

------
apunic
Sequelize (used for the node-mysql tests) is the slowest ORM out for Node,
others are much faster (i.e. the node-mysql module).

And I would love to see node-mongodb tests with the native Mongo driver and
not with Mongoose (which is slower).

~~~
bhauer
I know this will sound like a refrain, but would you be interested in
preparing a test for Node that uses a better ORM? Or if you can demonstrate
that another ORM would be clearly superior, perhaps it would make more sense
to simply modify the existing Node test code, replacing Sequelize?

------
infinia
I recently started transitioning from PHP to Rails. Should I be reconsidering?

~~~
voidlogic
Maybe, maybe not, that depends on your priorities- If your concern is (or a
major concern) is performance, then yes, you should reconsider (maybe look at
Go?). If that is not a major concern, maybe not. (I personally am not a Ruby
fan due to performance and security issues, but it works well for many people)

------
cakeface
I like to see that Tapestry is rising high in this latest round. I have a lot
of production sites running Tapestry and I'm always impressed with how fast it
runs.

------
eibrahim
I would love to see how asp.net mvc compares to all this...

------
andybak
Wow. I really need to use connection pooling with Django.

------
fsiefken
Again no mono with ServiceStack! It boggles my mind...

------
talloaktrees
It should be mentioned that Go 1.1 is about to be released, which includes
really big speed increases (sometimes over 30%)

------
charlesjshort
Any thoughts of putting some Erlang in there?

~~~
kainsavage
We got a pull request for an Erlang framework (my brain is not letting me pull
up the name) yesterday that we accepted, but sadly it was too late for this
round of benchmarks. We will include it in the next round.

------
jjm
Hm... we'll need to get spray.io on this list

~~~
bhauer
Yes, please! Spray.io folks, can we entice a pull request? :)

------
saadazzz
Any plans for ASP.NET?

------
jebblue
Servlets are super fast, this shows that.

------
apunic
Why are there no DB tests with Go?

~~~
bhauer
Disappointing answer incoming: we haven't found the time. If you know Go
sufficiently to write those tests (they should be fairly easy to write for
someone who works with Go regularly), we'd be very happy to accept a pull
request.

Also, just to reiterate something I've said elsewhere. We've done some spot
testing with Go 1.1 and its performance is super. We really look forward to
showing Round 3 numbers next week!

------
mike--
migth be interesting to add Erlang.

~~~
kainsavage
We have at least one Erlang test that came in yesterday and will be included
in the next round of benchmarks.

~~~
mike--
thanks.

~~~
kainsavage
Don't thank us, we didn't write the code for it - one of you (the community)
wrote it and submitted a pull request. We just ensured that the test would
work in our benchmarks, then ran it! We are happy to do it, and super
appreciative of the support from you guys!

------
escaped_hn
Is gemini really that awesome? Why arent more people talking about it?

~~~
saym
Gemini is closed source. The company running these comparisons is the only
team that is qualified to test/develop with it.

~~~
kainsavage
While it is true that our source code is closed at the moment, you are still
free to check out the benchmark code from github and run it yourself to verify
our results.

It wouldn't really be a fair test if we put all these frameworks up for
testing, suggested Gemini was better than sliced bread with auto-applying jam,
and didn't back it up in a testable way ;-)

