
Web Framework Benchmarks Round 4 - pfalls
http://www.techempower.com/blog/2013/05/02/frameworks-round-4/
======
saosebastiao
I love what you guys are doing. This is by far the most comprehensive (in
terms of number of frameworks) comparison of web frameworks. I also am a big
fan of the new filtering metadata.

However, I'm starting to think that all of the advocates of various frameworks
are now conspiring independently to make this comparison meaningless...any
framework (except Cake for some reason) can be superoptimized towards a small
set of tasks. If you do another round, could you increase the number of
different tasks? Some examples could be:

1) Mixed bag of queries of various complexity 2) Static file serving 3) A few
computation/memory-intense benchmarks (such as those in the Language
Benchmarks Game) 4) Templating

~~~
bhauer
Hi saosebastiao,

As Pat points out, we definitely look forward to implementing some more
computationally-intense request types in the future. This round does include
the first server-side template test. We'd like to hear the community's
opinions about more tests.

That said, I feel most of the frameworks' implementations of the existing
tests are _not_ cheating. Our objective in this project is to measure every
framework with realistic production-style implementation of the tests. No
doubt there is temptation to trim out unnecessary functionality and focus on
the benchmark's particular behavior. We have attempted to identify any such
tests that remove framework features to target the benchmark as "Stripped" and
those can now be filtered out from the list.

In other words, our aim is that the implementation of each framework's test is
idiomatic to that framework and platform. And if that's not the case for a
test, we want to correct it.

Your concern could be clarified by pointing out that framework authors may be
tuning up their JSON serialization, database connection pools, and template
processing in order to improve their position on these charts. And, to be
clear, I have already seen evidence of that in my interaction with framework
authors. To that concern, however, I would say: _That is awesome_. _I want
those features to be fast_.

~~~
jakejake
I would like to pile my thanks onto this list as well. I'm the author of
Phreeze and I can say that I'm grateful that fairness is being encouraged.
There is certainly glory in ranking well on any benchmark and I have to admit,
as I was implementing the tests in Phreeze, I saw many opportunities to
"cheat." For example, skipping the framework routing, not using the "proper"
way to communicate between the layers, etc and substituting things with "raw"
code would have potential to skew the result. I feel that would be missing the
entire point of a benchmark, so I'm glad that is being considered.

I can also say that this benchmark inspired me to take a hard look at class
loading and I was able to make some improvements to the framework's efficiency
in general. So, in a way, I did some tuning - not _for_ the benchmark, but
rather _as a result_ of the benchmark. Thanks to this benchmark all Phreeze
users will gain a little performance.

I would also like to suggest a test idea. I think the biggest challenge for
frameworks comes into play when you have to do table joins. Something like
looping through all purchase orders and displaying the customer name from a
2nd table - that would be a very real-world type of test. I think foreign key
type of queries are more telling about an ORM than a single table query.

Thanks again!

~~~
camus
Can you share with us the tuning you did with class loading for instance ?
thanks for your comment.

~~~
jakejake
Oh sure, nothing to complicated. Basically I just happened to notice that I
loaded several classes that were not always needed. I was able to tune up the
framework to load some of them on-demand instead.

One example is that the framework loaded an lot of MySQL classes whether or
not you do a DB query. So, now I wait to initialize the DB stuff until after
you make a call that requires it. Phreeze has always been lazy about opening
the DB connection, but now it's even lazier and doesn't even load the classes
until you need them!

There were some other utility-type classes like XML parsing and such that
probably don't even get used much. So that is lazy loaded now too.

For a non-DB request I was able to get it down from about 37 files that loaded
to around 20. For a DB request I think it's still around 30 files, but I
definitely consider that a performance improvement. The benchmark led me to
scrutinize what is being loaded so I think it has already improved the
framework.

------
bhauer
This is the most recent update to our ongoing project measuring the
performance of web application platforms and frameworks. In this round we've
received several more community-contributed tests in Perl, PHP, Python, Java,
and JavaScript. Go is a comeback champion thanks to changes made by Brad
Fitzpatrick [1] and others in the Go community.

A new "Fortunes" test was also added (implemented in 17 of the frameworks)
that exercises server-side templates and collections.

With 57 total frameworks being tested, we have implemented some filtering to
allow you to narrow your view to only those you care about.

As always, we'd really like to hear your questions, suggestions, and
criticisms. And we hope you enjoy this latest round of data.

[1] <https://code.google.com/p/go/source/detail?r=45c12efb46>

~~~
nopal
Any reason you didn't test ASP.NET MVC or ASP Web Forms?

~~~
apalmer
ASP.Net kind of put themselves out of the benchmark game here:

Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows
a Mono performance test even if accurate is going to be of dubious value.

Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net
implementation the numbers wont really be meaningful.

Windows Issue #1, if you do the test on a different OS than every other test
implementation, the results really wont be comparable in any fair way.

Microsoft Issue #1, dont know if it still holds now a days but in past
official EULA for .Net prohibited publishing benchmark results. PERIOD.

I am a .net developer and as much as I like ASP.Net I dont think the effort of
adding a .Net implementation really would pay off.

~~~
bsaul
I completely understand your point, but I think it's fair to say that most
.Net code will run on Windows server, and that pretty much everything else
will run on some kind of linux flavor. Just like you have a "keep the default
framework setting" approach to help compare very different frameworks because
that's how the majority of people will use them, you may very well assume that
comparing frameworks on their preferred OS is fair enough.

I know that i wouldn't mind switching to a windows+.Net environment if it
proved to be much much faster than what i'm using right now.

~~~
DennisP
Exactly, and if there is such a performance boost, having the numbers would
help you figure out whether hardware savings would be worth the licensing
cost.

------
Periodic
First off, I love the work you're doing, keep it up.

Benchmarks like this are designed to be the starting-point of a discussion an
investigation, and not as anything meaningful in their own right. Boiling it
down a framework to one performance number ignores the many, many nuances of a
framework.

What surprises me most is the difference between different frameworks. A few
years ago the mantra seemed to be "Use Rails, Django or a similar full-stack
framework. Speed of deployment trumps everything!" Over the last few years
I've seen a shift as people are trying to get more performance from limited
hardware. Personally I'm intrigued by how a fairly innocent decision early in
the project (of what language/framework) may have profound performance
implications in the long run.

For myself, I've been looking for a good functional-programming framework.
Just looking at this gives me a good list of frameworks to start looking at.
It feels to me that a framework that performs well is likely well engineered,
so the ones that perform better will go at the front of my queue for
investigations.

~~~
bhauer
Thanks, Periodic. It's especially rewarding to hear that people have gleaned
value from the project.

You're precisely right about how to put this data to use: as one point in a
holistic decision making process. We address that in the Questions section of
the site, in fact. That said, we are not reducing each framework to a single
performance number. Our goal is to measure the performance of several key
components of modern frameworks: database abstraction and connection pool
performance, JSON serialization, list and collection functions, and server-
side templates. We'd like to add even more computationally-intensive request
types in future rounds.

So, no, we're not testing your (or anyone else's) specific application on each
framework. But we are testing functions that your application is likely to
use. You're still better off measuring the performance of your use-case on
candidate frameworks before you start work, but perhaps you can first trim the
field to a manageable number.

In the first round, we echoed your surprise at the spread--four orders of
magnitude! I think the shifting winds of opinion come from the fact that
today's high-performance languages, platforms, frameworks are _not
necessarily_ more cumbersome to use for development than the old guard. As
others have pointed out elsewhere in this thread, Go is not a terribly verbose
language, and yet its performance is fantastic.

Has the era of sacrificing performance at the altar of developer efficiency
ended? I'm not sure. But we have some data to add to the conversation.

------
just2n
Before looking at the benchmark results, I took a glance at the Node source
and I expected it to perform worse than it did previously. It does almost
universally. Not only haven't the glaring perf issues remained since round 1,
it's added more. In the real world, when you look at a metric that says your
req/s is a bottleneck, which is what this benchmark is loosely simulating,
you'd fix it. You wouldn't just say "nope, that's what this framework does,
sorry boss."

I still don't find these benchmarks very useful. From the looks of the
comments, a lot of you don't really either (even if you don't realize it).

For example, a lot of people in these comments want to correlate language
speed with performance in these benchmarks, by arguing specific examples, but
comparing almost any two frameworks/platforms in this "benchmark" is an apples
to non-apples comparison, and the result is actually full of counter examples
(faster languages performing more poorly). That should instantly tell you that
this benchmark isn't telling you what you think it's telling you, and that you
haven't really derived any value from it.

Perhaps the biggest reason I don't find value here is that every product here
does wildly different things. It's like comparing wrenches to hammers to
screwdrivers to 3D printers.

I also want to point out to people who say that this is a "comparison" of
frameworks that it is emphatically _not_ a comparison. What is the value of a
framework? Is it speed? Atypically. And this "benchmark" tends to point at
such cases as "being better" because they do better in this specific task. A
framework/platform's value lies in features and abstractions. This does not
compare those.

I will gladly build a "framework" in NodeJS that is only capable of doing the
tasks in this benchmark as fast and with as little overhead as possible. You
would _NEVER_ use it in the real world, but it would be a beast at serializing
JSON and making repeated database queries in an insecure fashion. But score
here is the important factor, right?

~~~
matt2000
In my opinion you've missed the point almost entirely:

1) If you see problems with a language you're an expert in, submit a pull
request. I've never seen a benchmark done like this before, it gives everyone
a chance to fix problems in their favorite framework/language. 2) It is a
little bit of a unfair comparison between very low feature frameworks to
higher ones, but it gives you a good idea of what you're trading off on basic
performance. For example, I thought our use of play1-java wasn't far off of
servlet on basic tasks, but boy was I wrong, perhaps by 10x.

Should you read this list and pick the top thing on the chart? No. However,
hard to argue this isn't interesting and useful information.

------
coldtea
It's impressive how well PHP holds up with many queries per request (which is
the most common CRUD/webapp scenario).

While for no or just one query it's slower than a lot of the other frameworks
(due to PHP being slow to parse, startup etc), as soon as we have a lot of DB
queries, the C interface to MySQL leaves the other frameworks in the dust.

The well known PHP shortcomings aside, that's a nice example of optimizing for
the things that matter most, especially for it's common use cases (Wordpress,
Drupal, etc).

~~~
EGreg
In really scalable sites, you need sharding. Unless your database itself is
doing the scaling (such as with Riak), you're going to sometimes hit multiple
shards. With PHP and other languages that can't do async, you're going to have
to query the DB sequentially, increasing latency proportionally to the number
of shards you have to hit. With Node.js and other asynchronous apps, you
don't.

Disclaimer: mysqli does have async capabilities, but most people such as
myself use PDO for its other benefits. And mysqli only works with MySQL.

~~~
bhauer
Some of the fastest implementations you see in these tests are _not_
asynchronous.

With Servlet for example, a worker thread is chosen from Resin's thread pool
and used to handle a request. The Servlet then executes 20 queries
sequentially and returns the resulting list data structure. This is Servlet
3.0 but not using Servlet 3.0 async.

Async isn't making the top performers fast. Being fast is making them fast.

~~~
EGreg
I agree, but what about the special case of hitting multiple shards and
aggregating the results? Shouldn't the non-blocking win over the blocking?

~~~
cmircea
Well depends exactly on the implementation.

Some may issue queries in parallel and aggregate the results, blocking until
everything is done. Others may run them sequentially, which is the simplest
but slowest way.

~~~
kbenson
Or if your data supports it, you shard based on an algorithm that is
repeatable and cheap, and the client can compute where to look for data, if it
exists.

------
zinxq
There should be no surprise that interpreted, dynamic languages are utterly
out-gunned as compared to compiled (JIT'd or otherwise) languages. It's
inherent to the system - every little thing you do costs more.

Many people choose Ruby and figure, given that premature optimization is the
root of all evil, they'll optimize later if needed.

That's like choosing between a farm tractor or a ferrari - and figuring if the
tractor doesn't perform up to snuff, we'll add a spoiler (and given the 10x
disparity between Java and Ruby in some of those graphs, if we throw out a
20mph top speed for a farm tractor, the ferrari analogy is actually rather
spot on).

There are many good reasons to choose dynamic/interpreted languages - but
always know you're giving up performance in exchange.

~~~
dragonwriter
> Many people choose Ruby and figure, given that premature optimization is the
> root of all evil, they'll optimize later if needed.

True.

> That's like choosing between a farm tractor or a ferrari - and figuring if
> the tractor doesn't perform up to snuff, we'll add a spoiler

Its really not like that at all, because programming languages aren't like
vehicles. Particularly, with Ruby, on typical method of optimization is
finding which bits of code are bottlenecks, and then optimizing those
bottlenecks, often by replacing them with C (or, if the Ruby runtime being
used in JRuby, Java).

Which I guess is like having your tractor _turn into a Ferrari_ for the parts
of work that involve going long distances on a road without towing something,
but I think that kind of points out how bad even using the tractor/Ferrari
analogy is.

~~~
candybar
I keep hearing about this, but do people really rewrite performance-critical
parts of their web apps in C? Even if it happens to be part of some third-
party library? And maintain a fork? What if that performance-critical part is
dependent on other parts in a non-trivial way? It seems that an unanticipated
replacement of some core functionality with a C library may involve a major
rewrite and most Ruby teams may not have the expertise to do a good job
maintaining a C code base any way.

~~~
dragonwriter
> I keep hearing about this, but do people really rewrite performance-critical
> parts of their web apps in C?

Certainly they do it for Ruby apps in general. I don't think its all that
common for it to be a high-value proposition for web apps.

> Even if it happens to be part of some third-party library? And maintain a
> fork?

If its an open-source third-party library that tends to get used in a way that
is performance-critical, upstream will probably accept moving bottlenecks to
(portable) C and maintaining the API, so its unlikely that you'll need to take
up responsibility for a fork.

> What if that performance-critical part is dependent on other parts in a non-
> trivial way?

If the call pattern is such that they are not part of the performance critical
part themsellves, then the performance critical part calls them through the
regular conventions for calling Ruby from C.

If the call patter is such that they are part of the performance critical
piece, well, I think the answer is obvious.

> It seems that an unanticipated replacement of some core functionality with a
> C library may involve a major rewrite

It might, but in the meantime you've got working code.

> and most Ruby teams may not have the expertise to do a good job maintaining
> a C code base any way.

If the team determines it needs expertise in a particular area that it doesn't
currently have, then it should either develop that expertise or bring in
people that have it. That's true whether its particular domain expertise
(e.g., building messaging systems) or particular technology expertise (e.g.,
C). That's part of the normal development of a team.

~~~
candybar
I guess my point is that it seems disingenuous to point out "you can write
that bit in C" as a way to mitigate performance problems with Ruby, when in
practice it's so costly compared to available alternatives (throw more
hardware, write manually optimized Ruby, switch to a faster language/runtime)
that almost no one does it. How much of Rails is written in C? It's like
proposing compiler extensions/patches as a way of dealing with performance
problems. And if you have a complex application that utilizes many of Ruby's
idioms to deal with the complexity, it's extremely unlikely that you can
simply replace parts of it with C libraries without reorganizing in such a way
to increase complexity.

~~~
dragonwriter
> I guess my point is that it seems disingenuous to point out "you can write
> that bit in C" as a way to mitigate performance problems with Ruby, when in
> practice it's so costly compared to available alternatives

I don't think its costly compared to available alternatives; I think its
generally an efficient alternative for the type of bottleneck that is actually
related to implementation language efficieny. I think, for most typical _web_
apps, the bottlenecks are only rarely of that type, so that's generally _not_
where the effort is going to be spent, but for the ones that do have
bottlenecks of that type, its quite appropriate a way of solving it.

> throw more hardware, write manually optimized Ruby, switch to a faster
> language/runtime

If writing manually optimized Ruby is an effective and cheaper solution, you
aren't experiencing the class of bottlenecks that are related to
implementation language efficiency. Switching languages or runtimes for a
component is a proper subset of the work of switching languages or runtimes
for a project, so the latter isn't going to be less costly than the former (it
may, if language-related bottlenecks are pervasive, or if you have non-
performance interests in the alternative language, have a bigger net payoff
and be more cost effective, but it won't be less costly, and its inherently
riskier to do all at once, since component-wise transition gives you a faster
cycle time in terms of realizing value even if you end up doing a full
replacement in the end.)

> And if you have a complex application that utilizes many of Ruby's idioms to
> deal with the complexity, it's extremely unlikely that you can simply
> replace parts of it with C libraries without reorganizing in such a way to
> increase complexity.

I disagree. Anything you can do in Ruby you can do in API-equivalent C that
can still call out to the exact same Ruby code for the functions that aren't
being moved into C, so there is no reason at all for the kind of
reorganization you suggest, particularly if you are building with loosely-
coupled components in the first place.

If you are building a complex app and its all tightly coupled, you've got a
big maintainability nightmare no matter what language you're using, and that
has nothing to do with Ruby.

~~~
candybar
I completely disagree that Ruby's slowness is rarely the bottleneck. In this
benchmark, we have reasonably simple requests on decent hardware in the realm
of 1 second latency, where faster frameworks are 10+ times faster. We have
Sinatra being far slower than similar frameworks like Scalatra, Unfiltered.

Yes, you can write Ruby in C, but it would be almost as slow as writing Ruby
in Ruby. I don't really see the point of saying, you can do anything you can
do in Ruby in C, it would be much more verbose and about as slow. The point is
that true optimization may force you to do things that you can only do in C
and there's no guarantee that this optimized version can be easily utilized
from the rest of your Ruby code. This has nothing to do with tight-coupling -
it's simply taking advantage of the language's abstraction facilities.

And no, having to write hand-tuned Ruby, as opposed to idiomatic Ruby, to get
performance that can be had by writing, say, idiomatic Scala or Haskell is an
indictment of slow implementations and prevents you from taking full advantage
of the expressiveness provided by the language.

And that's before you get into things like your team may have to get bigger
because you need a C/Ruby-extension expert, half the team not being able to
understand a critical part of the code base (very few Ruby developers are
reasonably competent in C), etc.

Again, the whole point is that Ruby's performance problems pose a real pain
point. Yes, you can rewrite parts of it in C, yes you can mitigate by using
gems written in C, yes, you can spend more time optimizing, yes you can throw
more hardware. But all of those are costly and it's disingenuous to pretend
that a problem doesn't exist simply because a workaround does.

------
freework
I really don't like these benchmarks. Its like benchmarking Fizzbuzz or
something. Frameworks don't do anything. No one chooses a framework (at least
I don't) based on performance. You choose one framework over the other because
you like the API and/or language. I myself am a framework author (giotto, a
python framework that was not included in these benchmarks). If my framework
had been included, I'm sure it would end up dead last. When I built it, I
wasn't thinking about performance, I was focusing on building a framework that
would result in applications that are easy to understand/debug and fast to
write.

~~~
lucian1900
Not sure why this is downvoted so badly, the comment is largely correct.

This benchmark is even less useful than alioth's shootout, I'm not sure why
there is so much effort put into it :)

~~~
trailfox
For many systems performance matters. Claiming that there is no difference
between Ruby and C is just sticking your head in the sand.

~~~
freework
You can build an application in Ruby, deploy it onto 15 machines, and it will
outperform the same application written in C and deployed to only one machine.
Performance is more of a function of the underlying hardware than the language
used to build it.

~~~
trailfox
If a language is 30x less efficient than another language then you would
likely need 30x more servers. Many folk are simply not prepared to spend 30x
more than they need to on hardware. It's the difference between 20 servers and
600 servers.

Case in point: "How We Went from 30 Servers to 2":

[http://blog.iron.io/2013/03/how-we-went-from-30-servers-
to-2...](http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html)

~~~
kbenson
That's only true when you are using the full resources of one or more servers.
If you are only using 1/100th of the server's resources, then being 30x less
efficient still doesn't require any more servers.

That doesn't negate the point though, language performance matters at certain
scales.

------
rdtsc
Very nice presentation!

One thing I am wondering is "what about concurrency level"?

Just because a server can handle 10x the number of requests when doing a
single request a time for 1000 requests, doesn't necessarily mean it can also
handle those 1000 request at 10x performance when they all come in at once or
in a short time period.

I saw some tests have "256 concurrency" does that mean they are sending 256
request concurrently? I want to see them play more with those numbers. Why not
have 1024 or more. Then also play with the number of available CPUs and see
which frameworks can auto-scale based on that. Some that can process
sequential requests fast might fall face down when faced with slightly
increased concurrency, in that respect these benchmarks are a bit misleading.

On the other hand it is good to see latency. That is a important. Now latency
vs level of concurrency would also be interesting.

------
qingu
Thanks for doing these extensive benchmarking tests. It would be really
helpful to see a more complex example that includes user authentication. Aside
from the benchmarks it's also a really good starting point to compare the code
in different languages and get a first impression of a framework.

On a side-note, I'd really like to know why so few start-ups seem to be using
Spring. It could be just a wrong impression . But from what I have seen most
start-ups use RoR or Django. My guess is that Spring is less flexible and less
known outside big companies, where it is usually the default. It could also be
that Spring works better with the waterfall model whereas Django or RoR are
better suited for explorative programming and that fits the respective spheres
better.

~~~
sghill
> It could also be that Spring works better with the waterfall model

I've used spring mvc in an agile setting a couple times now, and it has worked
fine. It doesn't tend to make developers all that happy, in my experience. If
you're in an enterprise full of spring, starting up the next app with it can
be attractive -- there likely already exists a bunch of tooling and knowledge
around spring.

I wouldn't use spring directly if I were trying to build something quickly for
a startup. I'd be more apt to reach for grails (which wraps spring),
dropwizard, or any of the other rapid-development frameworks.

------
campnic
I think the overlooked part of this, once we step back from the natural desire
to pick 'the best', is that people who care about the platforms are providing
a vast set of starting examples for people looking to get started on each
network. Its easy to do a side by side comparison of similar tasks across
languages which is something that is very valuable and, in my experience,
relatively novel. Thanks for all your amazing work!

------
reactor
Off all the top performers, Go seems to be the only sane choice to write a web
app. Moreover it is at the sweet spot; expressive, flexible, simple, super
performant, good community etc. I think it is convincing enough for me to give
Go a serious look for our new app.

~~~
gtaylor
As someone who is also really enjoying Go, I think you need to add a huge,
gigantic disclaimer before making a statement like this: Go's ecosystem of web
development packages is in its infancy. You're not going to find any super-
well-documented, super mature/stable web frameworks (thought a few are showing
great promise). and some of the individual components (for example, Gorilla)
are looking very good, but still have some more cooking to do.

I love the language, but let's not get too carried away until the ecosystem
grows. The reality is, if you're going to use Go for web dev, you're going to
need to be prepared to do a whole lot of things on your own.

~~~
reactor
I've read about the limitation of Go in terms of third party libs, but I
believe it is only temporary. I had a glance at the std libs, looks really
good for a new language. Our new app doesn't require all the bells and
whistles of a full fledged framework.

~~~
voidlogic
>>I've read about the limitation of Go in terms of third party libs

Must not be looking on github... Yes its new, but in a majority of cases I
have not had any trouble finding third party libraries and the std ones are
wide and excellent.

------
hippich
Whenever I hear "Java" I also get association "slow". But looking into this
list - java web frameworks doing incredible job!

~~~
why-el
I always assumed the slowness is associated with build times and such. You are
slow developing a Java web app, but not running it.

~~~
mikkelewis
and GUIs.

~~~
marcosdumay
At the time Java gained its fame, the GUI applications were unresponsive _,
not slow.

_ They could lock at any time, for a perceptible few hundreds of miliseconds,
but had still a nice average speed.

------
LukeHoersten
The Haskell Snap Framework has a new http server in the works. It's completely
rewritten on a new library called io-streams with performance in mind. Initial
benchmark results look promising.
[http://snapframework.com/blog/2013/03/05/announcing-io-
strea...](http://snapframework.com/blog/2013/03/05/announcing-io-streams)

~~~
mercurial
On the other hand, I'm genuinely surprised the current stable version is not
doing any better.

~~~
mightybyte
This benchmark probably has more to do with the performance of the JSON
serialization and database libraries than with web framework performance.

~~~
mercurial
If this was the only reason, Snap's performance would be closer to Yesod.

------
seldo
Feature request: it would be nice to have permalinks (even if they were really
messy URLs) to filter-sets so that I can share the slice of the benchmarks I'm
looking at with people, without having to list the filters manually.

~~~
bhauer
Agreed! I had hoped to do that in this round but simply ran out of time. I
will make it a priority. I'd love to allow people to share specific
comparisons with one another.

------
rustc
I wonder why the 9 lowest ranked frameworks are all in PHP...

~~~
showerst
Not that PHP is particularly fast, but I notice that Raw PHP does about middle
of the pack, but all of the ORM versions seem to do terribly. Interesting
stuff.

I've usually had pretty good luck with Slim, I'll have to try a version with &
without redbean and see how big a difference it makes.

~~~
cmircea
Probably because all those big frameworks have to initialize everything for
every request. Parsing, connections, configuration, you name it.

~~~
notJim
Raw php also has to initialize connections, and it does really well, so I
don't think that's it. To me, that points to overhead from initializing the
objects as the bottleneck. I would have thought that with APC, this wouldn't
be a major issue, though, so I wonder if that's still not it.

~~~
cmircea
APC only caches the opcodes, so the interpreter doesn't have to parse your
code. But the framework still has to set up itself for each request
individually. Parse configuration, create objects, etc. It adds up quickly.

------
bdesimone
I'd like to thank you for the effort, and genuine openness of this project. I
hope you'll continue to expand these tests.

------
mitchi
<http://openresty.org/> Looks interesting but at a quick glance it looks like
you are programming in configuration files? I'm not sure I like the idea.

~~~
dataminer
You can try Lapis <http://leafo.net/lapis/> its a new framework built on
openresty.

~~~
est
There's also <http://luvit.io/>, a nodejs core (libuv) but with LuaJIT2

------
ipostonthisacc
Lua is impressive - the first as a non-compiled lang. Is that Lua or Luajit in
OpenResty?

~~~
xt
LuaJit (or optionally Lua)

------
blissofbeing
For those wanting the link straight to the benchmarks its here:

<http://www.techempower.com/benchmarks/#section=data-r4>

------
oberhamsi
I implemented the Ringo app for this benchmark and of course ran it against
Node and a couple of others to see how we would perform in this neighborhood
before I opened the pull request.

And since that day I've been wondering: why does NodeJs (=V8 JS engine in C)
talking to MongoDB have higher response times and latency than Ringo (=Rhino
JS engine on JVM) talking to MySQL. The only thing where Node beats us JVM
guys seems to be the JSON response test.

~~~
ricardobeat
Node favours concurrency over raw speed; calls deferred with process.nextTick
and callbacks end up costing time, in exchange for better concurrency. I think
a blocking driver could leave Ringo in the dust, but it would be useless.

~~~
oberhamsi
I will test that. Should the latency stay more constant with higher
concurrency? Or what am I searching for?

Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32,
64, 128, 256):

nodejs (mongodb raw)

12,541 22,073 26,761 26,461 28,316 28,856

ringojs (mysql raw)

13,190 23,556 27,758 30,697 31,382 31,997

both look like they got room to grow

~~~
rdtsc
I think 256 is too low. It should at least start at 256 and go to 1024
perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.

~~~
bhauer
Hi rdtsc,

Until the project includes a WebSocket-enabled test or a test with forced idle
time (e.g., waiting for an external service to provide a response),
concurrency higher than 256 yields very little of interest. The reason being
that we are fully saturating the server's CPU cores at 256 concurrency [1].

Increasing the client-side concurrency level simply means that the front-end
web server (or built-in web server's socket listener thread) needs to maintain
a small queue of requests to hand off to the application server's worker
threads. It doesn't make the server any faster at completing those requests.
I've written some more about this at my personal blog [2].

[1] Caveat: Some frameworks appear to have locking or resource contention
issues and do not saturate the CPU cores. We will attempt to capture CPU
utilization stats in future rounds since this might be of interest to readers
and framework maintainers. But increasing concurrency would not increase CPU
utilization in these scenarios either.

[2] <http://tiamat.tsotech.com/rps-vs-connections>

~~~
ricardobeat
The point is benchmarking concurrency limits, not that it would improve raw
performance.

~~~
bhauer
Ah, okay. That's interesting, I suppose. You basically are interested in
seeing at what concurrency level the server starts spitting back 500-series
responses (or simply doesn't provide a response). Basically, how many
concurrent requests are needed before the server's inbound request queue
overflows.

We could test that out at some point.

------
Goranek
Wow Go is getting faster and faster

------
Xeoncross
What the‽ How is it possible my MicroMVC framework + ORM is faster than Native
PHP + ORM?

~~~
bhauer
MicroMVC is doing very well! It is possible that your ORM is more performant
than PHP ActiveRecord, which is what we used in the PHP + ORM test.

------
dthakur
I asked for this before. Could you please add error bars to your plots?

~~~
kainsavage
There is a column labeled 'errors' for this, but basically only Lift and
Phreeze had errors.

~~~
shellac
OP means 'error bars' in the sense of giving and idea of the accuracy of the
measured value (e.g. standard deviation, standard error).

------
codeulike
Is it linux only? Would have been nice to be able to compare c#/MVC

~~~
iknight
I hear they are accepting pull requests...

<https://github.com/TechEmpower/FrameworkBenchmarks>

~~~
codeulike
Ah yes, someones added aspnet_mono now, which seems to be MVC based
[https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/aspnet-
mono)

------
X4
I just wanted to say: Thank you!

This is highly respected work (bookmarked) and deserves all the upvotes HN can
give.

~~~
bhauer
Thanks for the kind words, X4! With each round, we anticipate a spectrum of
responses, but it's nevertheless a continuous surprise and honor to hear
positive responses.

------
buster
It'd be nice to also have a "lines of code" comparison, because i can see how
some benchmarks are at the top but after looking at the code they also tend to
be quite "manual".

~~~
bhauer
We do plan to start tackling that other dimension, efficiency. Source lines of
code is one objective measure that, for all its faults, we plan to include.

In a previous round, I pasted the relevant code directly into the results
view. I will likely do that again soon since it's convenient for the reader.

For the time being, I invite you to browse the Github repository and examine
the test implementation source.

~~~
voidlogic
gzipped source size in generally considered a better measure of effort than
actual LOC. The reason for this is that the languages with a lot of boiler
plate usually have tooling, IDE's etc to make that pain go away, and gzipped
source, while still counting repeated boiler plate end up weighing it less
than unique lines of code.

------
artursapek
Wow, I should really give Go a try.

~~~
swah
Agreed, but the language is sufficiently (edit) MORE verbose than Python that
it is still a valid strategy to write in Python first, and then rewrite in Go
for performance..

~~~
papsosouid
If go is less verbose, then why write it in a more verbose language first? The
reality is, go and python are pretty much even for verbosity, and there is
absolutely no benefit to writing your app in python first and then go. You'd
be better off writing it in go and then rewriting it in go if you want the
benefits of learning from your first attempt.

~~~
mseepgood
I'm a fan of Go, but Go is definitely a bit more verbose than Python.

------
desireco42
While I really enjoy results, I still would prefer to see smaller number of
frameworks with specific tasks that are common in web development, logging in
(assume user and pass are provided), listing of something. These two would be
way more meaningful then anything else to me.

Again, thank you very much for hard work. I think for some there are some
revelations there, like that C framework.

~~~
bhauer
We aim to add more tests in time.

Take a look at the new Fortunes test. That test lists rows from a database,
sorts them, and then renders the list using a server-side template. It's only
implemented in 17 of the 57 frameworks right now, but we hope to have better
coverage on that in time as well.

~~~
desireco42
Yes something along those lines was what I was thinking. Thank you.

------
bilbo0s
Wow... some of these tests are still pretty severely hobbled.

Is there some reason that you use built in json serialization for some
frameworks and not others?

There is also a lot of heterogeneity in the implementation of the multiple
queries test. For instance, even if I only look at... say... java frameworks,
you seem to implement the exact same feature in very different ways between
platforms. For instance, for servlets, you will store all of the results in a
simple array... and then write them out when you are done. Like so:

    
    
        final World[] worlds = new World[count];
        final Random random = ThreadLocalRandom.current();
        
        try (Connection conn = source.getConnection())
        {
          try (PreparedStatement statement = conn.prepareStatement(DB_QUERY,
              ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY))
          {
            // Run the query the number of times requested.
            for (int i = 0; i < count; i++)
            {
              final int id = random.nextInt(DB_ROWS) + 1;
              statement.setInt(1, id);
              
              try (ResultSet results = statement.executeQuery())
              {
                if (results.next())
                {
                  worlds[i] = new World(id, results.getInt("randomNumber"));
                }
              }
            }
          }
        }
        catch (SQLException sqlex)
        {
          System.err.println("SQL Exception: " + sqlex);
        }
        
        // Write JSON encoded message to the response.
        try
        {
          Common.MAPPER.writeValue(res.getOutputStream(), worlds);
        }
        catch (IOException ioe)
        {
          // do nothing
        }
      }
    
    

But for other frameworks, like Vert.x, you use CopyOnWriteArray to store all
of the results... and then write them out when you are done. Like so:

    
    
        private final HttpServerRequest req;
        private final int queries;
        private final List<Object> worlds = new CopyOnWriteArrayList<>();
    
            .
            .
            .
    
        @Override
        public void handle(Message<JsonObject> reply)
        {
          final JsonObject body = reply.body;
    
          if ("ok".equals(body.getString("status")))
          {
           this.worlds.add(body.getObject("result"));
          }
    
          if (this.worlds.size() == this.queries)
          {
            // All queries have completed; send the response.
            // final JsonArray arr = new JsonArray(worlds);
            try
            {
              final String result = mapper.writeValueAsString(worlds);
              final int contentLength = result.getBytes(StandardCharsets.UTF_8).length;
              this.req.response.putHeader("Content-Type", "application/json; charset=UTF-8");
              this.req.response.putHeader("Content-Length", contentLength);
              this.req.response.write(result);
              this.req.response.end();
            }
            catch (IOException e)
            {
              req.response.statusCode = 500;
              req.response.end();
            }
          }
        }
    
    
    

In other words, you literally create a new array each time you add a result to
that CopyOnWriteArray. In fact, not only are you creating a new array, but you
are creating new copies of the data in the array as well. Seems a little
strange??? DEFINITELY inefficient. Is there a reason that is implemented
differently? It seems to me that, at the least, they should both use arrays...
but maybe there is something more you guys are testing???

The Onion C based code is written in an even MORE efficient manner for the
multiple queries test. It actually stores it's results in json format from the
outset! Like so:

    
    
        json_object *json=json_object_new_object();
        json_object *array=json_object_new_array();
        int i;
        for (i=0;i<queries;i++){
            json_object *obj=json_object_new_object();
    
            snprintf(query,sizeof(query), "SELECT * FROM World WHERE id = %d", 1 + (rand()%10000));
            mysql_query(db, query);
            MYSQL_RES *sqlres = mysql_store_result(db);
            MYSQL_ROW row = mysql_fetch_row(sqlres);
    
            json_object_object_add(obj, "randomNumber", json_object_new_int( atoi(row[1]) ));
            json_object_array_add(array, obj);
            mysql_free_result(sqlres);
        }
        json_object_object_add(json,"json",array);
        const char *str=json_object_to_json_string(json);
    

The equivalent java code would be something like:

    
    
        private final HttpServerRequest req;
        private final int queries;
        // INSTEAD OF:
        //private final List<Object> worlds = new CopyOnWriteArrayList<>();
        // HAVE:
        private final JsonArray worlds = new JsonArray();
    
            .
            .
            .
    
        @Override
        public void handle(Message<JsonObject> reply)
        {
          final JsonObject body = reply.body;
    
          if ("ok".equals(body.getString("status")))
          {
           // INSTEAD OF:
           //this.worlds.add(body.getObject("result"));
           // HAVE:
           this.worlds.addObject(body.getObject("result"));
          }
    
          if (this.worlds.size() == this.queries)
          {
            // All queries have completed; send the response.
            // final JsonArray arr = new JsonArray(worlds);
            try
            {
              // INSTEAD OF:
              //final String result = mapper.writeValueAsString(worlds);
              // HAVE:
              final String result = worlds.encode();
    
              final int contentLength = result.getBytes(StandardCharsets.UTF_8).length;
              this.req.response.putHeader("Content-Type", "application/json; charset=UTF-8");
              this.req.response.putHeader("Content-Length", contentLength);
              this.req.response.write(result);
              this.req.response.end();
            }
            catch (IOException e)
            {
              req.response.statusCode = 500;
              req.response.end();
            }
          }
        }
    

With a similar change for Servlets. According to the benchmark results, Onion
comes out on top. It's the fastest. But how much of that's because it seems to
be written correctly and other tests seem to be written without taking
advantage of the same efficiencies.

Is it the case here that some people have sent you test code optimized for
their own frameworks?

If that is so, you should add some tests that would not be so amenable to
optimization. I'm not picking on Onion here by the way. In fact, the argument
could be made that Onion is not actually 'optimized', so much as just written
correctly, and the other frameworks have tests written incorrectly. But I just
wanted to know if you guys actually intended to use these different
implementations for some reason that I am unaware of? Do they make the tests
more fair somehow???

~~~
bhauer
Hello bilbo0s.

Thanks for taking the time to dig in and provide some feedback. As much as
possible, we want each test to be representative of idiomatic production-grade
usage of the framework or platform. Furthermore, we have solicited
contributions from fans of frameworks and the frameworks' authors. A side
objective is that the code double as an example of how best to use the
framework or platform.

All of this means we fully expect that the implementation approaches will vary
significantly.

The multiple query test has a client-provided count of queries, so in most
Java cases, we create a fixed-size array to hold the results fetched from the
database. I wrote the Servlet and Gemini tests, so I can confirm that behavior
in those tests.

We are not Vert.x experts and we have not yet received a community
contribution for the Vert.x test. However, it is our understanding that
idiomatic Vert.x usage encourages the use of asynchronous queries. The
question then is: how do we collect the results into a single List in a
threadsafe manner? Is your JsonArray alternative threadsafe? Admittedly, using
a CopyOnWriteArrayList gave us pause, but we are not (yet) aware of a better
alternative.

The Onion test was contributed by a reader and admittedly its compliance with
the specification we've created is perhaps a bit dubious. We want a JSON
serializer to process an in-memory object into JSON. I'm not certain if the
Onion implementation matches that expectation, but the test implementation
nevertheless seemed sufficiently idiomatic for his platform.

We're certainly open to more opinions on that matter.

~~~
bilbo0s
"..The multiple query test has a client-provided count of queries, so in most
Java cases, we create a fixed-size array to hold the results fetched from the
database. I wrote the Servlet and Gemini tests, so I can confirm that behavior
in those tests..."

I agree, that approach would be best. I just was unsure why you didn't do it
in Vert.x.

"...it is our understanding that idiomatic Vert.x usage encourages the use of
asynchronous queries..."

Someone can correct me if I am wrong, but my understanding of Vert.x is that
any query you send to the event bus is already asynchronous. There is no need
for a developer to worry about threads at all when writing a vert.x handler.
That handler will only ever be called from a single thread. So using a simple
array is fine. Using the JsonArray is even better, because then it matches the
Onion test idiomatically speaking. Which, I agree, is what you should be going
for.

"...The Onion test was contributed by a reader and admittedly its compliance
with the specification we've created is perhaps a bit dubious. We want a JSON
serializer to process an in-memory object into JSON..."

Please don't misunderstand, the Onion test does what you want it to do. As
well, it does it in the correct idiomatic fashion. That's exactly how I would
write the Onion test. I was just wondering why the other tests went out of
their way to decode Json and the reencode Json for each result. Onion only
ever encodes to Json once, other tests are encoding and decoding multiple
times. I only pointed out Vert.x because it was the most egregious. I mean in
that case the answer from the persistor is already in Json. It is put in a non
Json data structure... and then that data structure is encoded to Json??? Just
seemed weird.

\----

EDIT: Just verified that there is no need for thread safe code in a Vert.x
handler. (Gotta say... that is pretty slick)

On a connected note... man ... these tests are a VERY good way to learn more
about these different frameworks!

\----

~~~
bhauer
You're right, this is fascinating stuff. Like I said, we had not yet received
a pull request for the Vert.x test, but presumably we will get one before
Round 5? :)

We had not previously understood that there was no need for thread-safe
behavior within a Vert.x handler. Removing that (apparently fictional)
requirement allows us to use just a simple array. Out of curiosity, can you
point me to where you found confirmation that handlers do not require thread
safety?

Thanks again for your feedback!

Edit: spot checking Vert.x with a simple array does not appear to affect
performance to a measurable degree.

~~~
bilbo0s
Sorry (for the tardy response)... minor emergency came up.

Anyway, the relevant part of the manual is:

<http://vertx.io/manual.html#concurrency>

Also discovered you are running the equivalent of a single threaded
environment anyway. (8 workers on an 8 core machine)

Any reason for that???

Maybe more confusion about Vert.x concurrency??? If so, here is the relevant
documentation on worker verticles:

<http://vertx.io/manual.html#worker-verticles>

Long story short, you dedicated the entire machine (8 cores) to the equivalent
of database connection management, (the persistor). Very little of the
machine, (whatever is context switched in effectively), is dedicated to
request handling. Try something a bit more fair... like 4 workers and 4 web
request handlers.

Also, I was going through the Node.js tests and I had a question... do you
guys do any clustering for the Node tests at all? Or are these results from
the tests run on a single Node?

Sorry for all the questions, just want to make all of the tests do the same
thing across all of the frameworks so that when you guys run it again, we can
use that data here in a more meaningful fashion. For us, it is useful.

~~~
bhauer
We used 8 Vert.x verticles on i7 because there are 8 HT cores and our
understanding is that the "best practice" for Vert.x is to create a number of
verticles equal to the server's core count [1]. Obviously we would be happy to
hear from Vert.x experts about an more idealized configuration. Admittedly, we
have not spent a great deal of time attempting to tune Vert.x and, like I said
earlier, we have not received a pull request [2].

In all tests, the database test is allocating a greater amount of effort to
database connection management (in totality: the handling of connections,
statements, queries, and result sets) versus request handling. This is not
unique to Vert.x. The reason some frameworks' database tests achieve nearly
50% as many requests per second on i7 versus the pure JSON test is simply that
at the ~210k rps range for the JSON tests, we are running into a Gigabit
Ethernet wall (which I have commented about elsewhere). If we had 10 GBE, the
JSON test results on i7 would be even higher. (Also see comments elsewhere
about our intent to normalize, to a degree, the response header requirements
since the variation observed is attributable to response-headers [3].)

Yes, the node.js tests are running with the cluster module [4].

Thanks for the comments. We have received great feedback in the previous
rounds and this round received even more attention so there have been some
more good questions. Unfortunately, there has also been some rehashing, which
indicates we're not doing a great job of explaining to people how each
environment is configured (linking to the repository only goes so far). That
said, we also continue to receive some fantastic pull requests. Thanks to
everyone who has helped out!

[1] [http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-
node-...](http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-
simple-http-benchmarks/#comment-69)

[2]
[https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/vertx)

[3]
[https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...](https://github.com/TechEmpower/FrameworkBenchmarks/issues/113)

[4] <https://news.ycombinator.com/item?id=5645598>

------
desireco42
I have one more question, I see a number of really fat frameworks on top, if
rails is faster, then it must be because it is heavily optimized and people
who made it are super smart, but for example codeigniter is above slim, kohana
is above fuel, like twice the speed for Fortunes test. This isn't what I would
expect at all, I would also expect Rails to be below php frameworks for speed
alone.

Did you use orm when it was available? Or just used raw query?

This is in line with my previous query about more usage scenarios (and thank
you for fortunes test) who would be like we use them usually.

My point is, that the same reason why we use framework instead of raw
language, we use orm instead of direct sql.

~~~
oberhamsi
the source is available. as a first guess: if an ORM is the typical way to do
it in an env then it was used.

~~~
desireco42
yeah, my bad, should look into the source :)

------
rallison
For those new to these benchmarks, all of the source code is up on github:

<https://github.com/TechEmpower/FrameworkBenchmarks>

We welcome all pull requests, suggestions and criticisms.

~~~
exabrial
Any chance you could add JEE6? The two major JSF 2.1 implementations MyFaces
and Mojarra are both missing. I don't expect awesome performance numbers, but
since JSF2.1 is the 'official' (sigh) web framework of JEE6, it would be
interesting how awful they are compared to some of these other languages.

I guess though if this is only testing JSON serialization, it may not make
sense. Perhaps adding JAX-RS implementations like CXF, Jersey, RESTEasy, and
RESTLet would be more appropriate.

~~~
henk53
>it would be interesting how awful they are compared to some of these other
languages.

The World Wide Wait benchmark, which tested quite a lot, showed quite
favorable numbers for both JSF implementations.

------
pjmlp
What about vibe.d? While D is listed, vibe.d seems to be missing.

~~~
bhauer
Good catch. I have "D" listed because I heard from the vibe.d community [1]
that a pull request was incoming. We have not yet received it. Can you help
the vibe.d guys get it submitted?

[1]
[http://forum.dlang.org/thread/urpqdftuofgwespkcdxg@forum.dla...](http://forum.dlang.org/thread/urpqdftuofgwespkcdxg@forum.dlang.org)

~~~
pjmlp
I am active on D forums and remember reading about it.

I guess everyone is busy because of the DConf 2013 <http://dconf.org/>

Otherwise I will remember them next week.

------
axlerunner
Good to see Java is holding its own.

~~~
bjhoops1
Really. I've recently been getting into node/express after years of Java, and
these results make me feel a whole lot less cool.

~~~
oberhamsi
try our RingoJs: it's JS on the JVM. Scripting Java with JavaScript.

~~~
bjhoops1
This, just in case noobs were not confused enough by Java/JavaScript? :)

~~~
oberhamsi
confusing but accurate :)

The ability to script Java is one of Ringo's killer features - for this
benchmark, for example, we dropped in two jars (JDBC myqlconnector &
connection pooling from apachecommons) and glued them together with 10 LOC of
JS.

------
minamea
I wonder if anyone can put a D lang web framework on there? Vibe.d seems to be
the only viable one: <http://vibed.org>

------
geekam
Some of the PHP frameworks do better than Python frameworks.

------
riprock
Compojure is looking really attractive ("modern" dynamic language with JVM
performance backing it up). How is the ecosystem around Clojure web
development?

~~~
mark_l_watson
The ecosystem around Clojure web development is in good shape. I have used
Compojure and Hiccup (and Noir) for a large part of my web development in the
last few years and it has been a happy experience.

I have had a little less joy experimenting with both Clojurescript and
Ember.js (with Clojure back end services): I eventually get things working,
but at a huge time cost over writing non-rich clients just using Hiccup.

------
niggler
How are multiple cores handled? For example, with node.js, the standard way to
scale is to run multiple instances (but according to
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nodejs/setup.py)
only one instance is run).

I wonder how the results would change if only one or two cores were enabled
(using taskset or isolcpus)

~~~
pfalls
We use the cluster module to handle multiple workers
([https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nodejs/hello.js)),
so far we've received feedback that this is a valid approach.

~~~
pkroll
For tests where you specify the number of cores, do you specify 4 or 8 on the
i7?

~~~
bhauer
8 since the i7 is four physical cores with hyper-threading.

------
cloverich
I know the following is small, but surprised Pyramid didn't make the cut. I've
used it for a couple of projects outside of work, and among the several other
web frameworks I've worked with (Struts, Play, Django), it seems relatively
mature and well documented.

[http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch...](http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch/#)

~~~
voidlogic
>>didn't make the cut

Did you submit a pull request for it and it was denied?

------
Spiritus
Would love to see <http://luvit.io/> in there, to see how it holds up against
Nodejs.

------
luisbebop
awesome. congratulations guys. nice job! you are defying the status quo about
some frameworks with numbers instead of ego talks.

------
kclay
Why I'm still using Play! these results are disappointing once again, atleast
beat node.js. Go is looking great though.

~~~
terhechte
I've dabbled a bit with Play, but found it too slow (with Scala). Have a look
at Scalatra or, if you don't need a full blown framework, Spray

~~~
room271
Or Unfiltered which performed very well in the tests too.

------
cies
Could anyone explain: why is gemini sooo much faster then the others in these
tests?

I believe that the way these test are setup slightly advantage gemini, and
more broadly java. Since they do not measure memory usage, or tasks that make
memory usage critical, which is something JVM sucks at.

~~~
bhauer
Hi Cies,

Gemini is our in-house framework and there are two points to consider:

(a) We are obviously very familiar with Gemini and therefore know how to use
it effectively. For example, we know that we prefer to deploy Gemini
applications using the Caucho Resin application server because it has proven
the quickest Java application server in our previous experience. Of course,
the other Java Servlet-based frameworks also benefit from deployment on Resin
in these tests.

(b) In our design of Gemini, we do keep an eye on performance. But as the data
shows, there are faster options.

Although we included Gemini in these tests, we did so because we wanted to
know how it stacked up against other frameworks that we routinely use on
projects. See more information in response to an earlier question here:
[https://groups.google.com/d/msg/framework-
benchmarks/p3PbUTg...](https://groups.google.com/d/msg/framework-
benchmarks/p3PbUTg-Ibk/fOjQMxESEJ0J)

Incidentally, the memory usage profile for the Gemini test is fairly compact,
in terms of used heap space within the JVM, as are most of the Java tests in
this project. With no need to do so, we're not trimming the memory allocation
for the JVM to a minimum in our tests. But if we did, as you point out, the
tests we've implemented so far don't require much memory.

------
ausjke
Hold on, Symphony2 is that bad?? I use Drupal, which will use Symphony2, this
sucks!

------
ihsw
It's very depressing to see Symfony2 at the bottom of these lists. Although
there are quite a few performance optimizations that can be done to improve
this, there are few excuses for such poor performance by default.

~~~
krg
If you have improvements to the test (that would be realistic for a production
deployment) then please submit a pull request. We definitely want to show each
framework doing the best it can do.

A few frameworks have a "stripped" version (just Django and Rails so far) to
try to show the best that can be achieved when typical functionality is
stripped out. Essentially optimizing for this test, which is interesting even
if it isn't the point of these benchmarks. If you think Symfony2 would benefit
from a separate "stripped" test then please consider a submitting a pull
request with that.

~~~
redtuesday
A stripped test is on the way it seems. One user submitted a raw test a few
days ago which doesn't use doctrine.

------
ninjazee124
Guess Lift is not all that high performance framework that they claim it to
be!

~~~
SkyMarshal
Depends on whether any Lifters have optimized it for these tests or not,
which, after skimming the github repo and commit history, does not appear to
be the case.

Same with any of the other frameworks - some have been optimized by their
fans, others are running in default configs.

Techempower should add a filter to show only frameworks that have been
optimized.

~~~
krg
At this point, most of the tests were contributed/improved by the community.
Not all have been reviewed by experts in that framework, and I agree that an
"expert reviewed" marker would be nice.

That said, we have tried to not run anything in the "default" configs, but
rather the "production deployment" configs if we could find documentation on
that. Unfortunately there is a huge variability across frameworks in how good
the "production deployment" documentation is.

~~~
SkyMarshal
Have you guys reached out directly to the mailing lists/google groups/github
issues/etc of all the frameworks you're testing, asking for experts to take a
look?

I know some of these communities don't really frequent HN and Reddit, but they
all frequent their mailing lists.

------
manulp
Any chance of seing the python frameworks behind uwsgi instead of gunicorn?

~~~
pfalls
Here is the issue for just that:

[https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...](https://github.com/TechEmpower/FrameworkBenchmarks/issues/114)

If you're interested, we'd love to have some help getting this accomplished.

~~~
dorfsmay
I look at that, I don't understand what the issue is?

Can you explain. I might be able to help, I've got some experience with uwsgi
+ bottle.

------
derengel
Can there be a sinatra benchmark with raw database connectivity?

~~~
redtuesday
Yeah, you could submit the test to them for example . They accept pull
requests. ;o)

------
fetbaffe
Would be nice if a PHP Fat-free framework was included in the benchmark.

<https://github.com/bcosca/fatfree>

------
aualin
Is it Lua or LuaJIT?

~~~
camus
LuaJIT

------
camwest
What version of the programming languages are being used?There is a major
difference between Ruby 1.9.x and Ruby 2.x

~~~
krg
Ruby 2.0.0-p0

<http://www.techempower.com/benchmarks/#section=environment>

~~~
hilem
though they're also using Rails 3.2.11, which not only isn't the latest 3.2.*
release but doesn't take advantage of Ruby 2.0... it would be more interesting
to see the latest Rails 3.2.* on Ruby 1.9.* along with the latest Rails 4
release candidate on Ruby 2.0.0-p0

~~~
krg
I created a Github issue to address the upgrade to 3.2.13, and pfalls is
working on getting this done.

[https://github.com/TechEmpower/FrameworkBenchmarks/issues/24...](https://github.com/TechEmpower/FrameworkBenchmarks/issues/247)

I encourage you to benchmark Rails 4 yourself and see if there is a measurable
difference from the latest 3.2.x, to get a preview of the impact it will have.
When Rails 4 is released we'll definitely want to upgrade to that.

Edit: pfalls is too fast for me, and just completed the upgrade to 3.2.13 and
closed the issue. The next round will use Rails 3.2.13.

------
tlear
Since this is multi core test, how many instances of Node are running
actually?

~~~
rzimmerman
I was wondering the same thing. If they're running on a quad core (or EC2
Large) the results could look pretty poor with a single node instance.

------
thomasfl
pg: Do you consider a rewrite of Hacker News to something faster?

~~~
Narretz
Is HN slow? I'd rather have some UI improvements.

------
escaped_hn
When is gemini going to be released?

------
dallagi
what about django / flask + pypy?

------
dallagi
what about django

~~~
pekk
The benchmarks include Django, Flask, Bottle, and Tornado.

