
Frameworks Round 6 - amarsahinovic
http://www.techempower.com/blog/2013/07/02/frameworks-round-6/
======
columbo
I'm really surprised at the performance of C#... honest no-flame question: how
is possible it that Python is beating C#? I've been out of the .NET world for
several years, almost a decade, but I always assumed they would be at Java
level performance.

edit add: This is really fantastic work

~~~
iaskwhy
I guess you need to change the Hardware to "Win" to see it shining.

~~~
TheAnimus
Indeed, mono performance isn't great. That is the problem with C#, you need
ultimately to run it on IIS for web platform deployment, sadly under anything
else it is rather slow.

Also these tests are a bit unfair, as I think they give benefit in some areas
to frameworks that cache by default, rather than by request.

Ultimately, what I want to know is total cost, that means hardware, licenses /
support agreements, development time, developer ease of hiring.

That isn't to say these aren't interesting, but they are rather apples to
oranges. As I posted in the other flamebate thread about MS stack > *, I have
set a note in my calandar to help improve the example for MVC.Net
([https://news.ycombinator.com/item?id=5976540](https://news.ycombinator.com/item?id=5976540)).
Hopefully I will be able to come then when not in an exclusive agreement, they
suck I know, but it pays my mortgage!

~~~
bhauer
I missed your later reply in that other thread. If the node.js implementation
is caching results from the database, it should be modified to _not_ do so.
The original implementation did not, but it has seen several updates from
node.js SMEs. Can you point to what makes you suspect that it is caching? The
requirements do not allow for caching [1]. Future test types (perhaps in Round
7) _will_ allow for caching, but none of the present test types allow for it.

[1]
[http://tewebdev.techempower.com/benchmarks/#section=code](http://tewebdev.techempower.com/benchmarks/#section=code)

------
dom96
Finally! I was disappointed when this round was postponed.

I couldn't wait to see how well my framework (Jester;
[https://github.com/dom96/jester](https://github.com/dom96/jester)) performed,
not only because it was my framework but also because it is written in Nimrod.
I am a bit disappointed by the results but at least there is a lot of room to
improve now, I didn't have much time to properly implement concurrency for
Jester so I opted for just spawning a couple of processes which is definitely
not ideal.

I hope that Round 7 will go better for Jester. In the meantime I hope that you
will check out the Nimrod programming language ([http://nimrod-
code.org](http://nimrod-code.org)) despite the results that my framework
achieved.

To the team that carried out these benchmarks, thank you!

~~~
bhauer
Finally indeed! In future rounds, I hope that we can spend less time trying to
resolve problems--if there are problems, we'll just post the results and move
on to the next round. Better to keep iterating, and allow the experts who work
with each framework resolve issues as they come up.

Thanks very much for contributing Jester to the test! I really appreciate your
spirit and hope to see it improve in Round 7 as well. I'm looking forward to
seeing the next pull request. :)

~~~
dom96
Indeed, although people may be disappointed if their framework fails and they
have to wait another month for another round of results, but then I suppose
the next round will come a lot quicker if you have less issues to fix.

No problem at all. Thank you for merging my process spawning silliness. You
will definitely see a lot more improvements from me :)

EDIT: A little issue I just noticed: the jester "Front-end server" is shown as
"None" when in fact it's "nginx".

~~~
MetaCosm
I think they would be less disappointed if it meant a consistent schedule.
These benchmarks are really a lot of fun, not sure if they change hearts and
minds, but they are beautiful and fun.

I think being time-boxed rather than implementation-boxed make sense. If you
miss a month, you got 30 days to fix it.

Also, they could only "update" the test rules quarterly or every 6 months,
give the authors a bit of time to respond.

------
bhauer
Thanks for posting this, amarsahinovic.

This is the latest round of our ongoing project to benchmark the performance
of fundamental services provided by modern web application frameworks. With
the continued contributions of the developer community, the number of
frameworks and platforms included is now 74! Round 6 also adds the often-
requested plaintext test with HTTP pipelining (thanks in large part to William
Glozer of Wrk fame).

The results web site also now includes a "framework overhead" tab that allows
you to compare a framework versus its underlying platform. This was originally
conceived by a Hacker News user named goodwink [1].

Like the round before, Round 6 gave us some technical challenges, including a
period where we were unable to collect any latency data. Today, some problems
persist, such as the Dart test (yes, there is server-side Dart) failing to
complete for most of the tests. However, rather than continue to defer Round
6, we've decided it's better to move forward and use the round-over-round
progression to iterate, fix, and improve. No problem that exists today is set
in stone.

To that end, as always, I would love to hear any feedback, advice, questions,
criticism, and especially contributions. And we hope you find this information
helpful!

[1]
[https://news.ycombinator.com/item?id=5454970](https://news.ycombinator.com/item?id=5454970)

~~~
jcampbell1
Luminus appears to be a clojure framework, not a php framework.

Thanks again for posting these tests.

~~~
bhauer
Oops! My mistake in classifying. Many apologies to fans of Luminus. I've just
corrected this.

------
coolsunglasses
I've followed yogthos' work on Luminus since the beginning.

It doesn't merit mention independently of Compojure. It's just some sane
defaults for pedagogical reasons for getting started writing web apps on top
of Compojure. It adds nor removes no runtime or anything else. As it currently
exists, it's a Leiningen project template generator. That's about it.

Luminus is isomorphic, for the purposes of a benchmark, with Compojure.

I'm sure yogthos will be flattered when I tell him his web app templating
thingy was included though.

~~~
bhauer
Sunglasses! Good to see a familiar name.

Incidentally, yogthos should be aware that Luminus was included:
[https://github.com/TechEmpower/FrameworkBenchmarks/pull/293](https://github.com/TechEmpower/FrameworkBenchmarks/pull/293)

------
mark_l_watson
Disturbing results since I so frequently have used Sinatra for hosting REST
APIs for customer projects. Rethink needed.

For my own projects I use Clojure + Compojure and more recently added Node.js
(often with Meteor) and was pleased to see that they benchmark well. One thing
I especially like about Compojure with the Hiccup HTML generating library is
that I see syntax errors immediately in the editor. I suppose that Scalatra
with embedded HTML would provide the same benefit.

~~~
bemurphy
Maybe benchmark it yourself? I think something is amiss, it should easily
'hello, world' at 2x higher rate than Rails. Poked around the setup a little a
while back but nothing jumped out.

Hunches: it's either going through webrick or not running with RACK_ENV
production

~~~
bhauer
Unicorn, nginx, production environment.

[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/sinatra/setup_ruby.py#L18)

------
Historiopode
Wonderful, thanks for the continuous updates. I hope you will keep gaining
momentum around this initiative; having a solid community-supported set of
benchmarks which routinely improves would be a true asset.

By the way, I would like to offer a UX suggestion: it would be nice if you
could select one or more frameworks and have them highlighted in the result
listings, so as to be able to notice their relative position at a glance.

~~~
bhauer
Thanks for the feedback. The highlight is a good idea and I've created an
issue for the suggestion [1]. I'll try to get that added in time for the next
round.

[1]
[https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...](https://github.com/TechEmpower/FrameworkBenchmarks/issues/359)

------
bliker
Honest question: Why are most of the frameworks that I know of in bottom half?
When does speed becomes relevant?

~~~
RyanZAG
Three reasons as to the hierarchy:

1) Most popular frameworks you would encounter run on higher level dynamic
languages such as PHP, Ruby or Python. These languages are a decent bit slower
than performance optimized languages. The languages towards the top are all C,
Java, Go, etc.

2) The frameworks towards the top are generally lower level - they do less
automatic 'magic' than the more popular frameworks that handle things like
validation automatically.

3) In general, it's more likely you will have heard of older frameworks than
newer ones. The frameworks towards the top are generally all less than 2 years
old (excluding Java servlets which have just always been decent). This means
they make use of newer designs such as non-locking and asynchronous connection
handling.

When does speed become relevant? It's actually always relevant as soon as your
product gets traction. For example, 1 server at the top of that chart may be
able to handle the same load as 10 servers towards the bottom. If you're using
a framework at the bottom and paying heavily for 10 servers, you could
increase your startups runway substantially by swapping. However, swapping may
cost more in engineer-hours than the servers.

Basically, the relevance of speed is directly related to your margin. If your
margin is high (investment banking maybe?), then speed is almost irrelevant,
you can afford a whole building of servers. If you're competing heavily on
mass market web apps, speed is fairly critical as you can out price
competition.

~~~
bliker
Thank you for great response.

But for me all this seems quite biased towards small Frameworks/Platforms. I
will hardly ever serve one JSON response form full stack framework. If I just
made some C server that would do those tasks very well it would be first in
this list but very far of anything of value.

Albert Einstein once wrote, “Everybody is a genius. But if you judge a fish by
its ability to climb a tree, it will live its whole life believing that it is
stupid.”

~~~
rallison
There is no evidence that Einstein ever wrote or said that [1].

[1] [http://quoteinvestigator.com/2013/04/06/fish-
climb/](http://quoteinvestigator.com/2013/04/06/fish-climb/)

------
deelowe
Thanks for continuing to do these. As far as I know, no one else is doing
something like this in such a comprehensive way.

------
neya
An honest confession from someone inspired by these comparisons. I've been
following these benchmarks very closely right from round one and ever since
I've been waiting to see my favorite framework, Ruby on Rails perform decently
to some extent. I waited till the last round to see some improvements and
while I DID see some marginal improvements, it wasn't as expected.

This then provoked me to do some basic math. Imagine a real world startup
scenario wherein you need to your application to deliver API responses (JSON)
or even ordinary HTTP responses. If you use a framework from the bottom of the
list (like Rails), then you may be able to serve, say, 'X' customers on your
site using your application at a time. Now, if you use something from the top
of the list, you may be able to serve roughly about 20 times (20X) more number
of customers on a single server. This not only means less cost, it also means
you can worry about scaling and stuff much much later than you would have to,
unlike on a low performant framework.

For this reason, I decided to conduct an experiment for myself. Pick up a
framework from the top of the benchmarks and experiment with the learning
curve and the language. Well, after carefully evaluating, I decided JAVA
wasn't for me (too much of a beast). But Scala seemed to hit the sweet spot
with excellent language (1 line of Scala = 10 lines of Java on an average)
features along with the trust of the JVM.[1]

So, I chose Scala and started experimenting with it. I started with the play
framework and was not fond of certain decisions that were made within the
framework. Also, the play framework is nothing like Ruby on Rails (but it
claims to be so) and was more of self-serve and also quite heavy for my taste.
Hence, I went even farther up the list and chose Scalatra. Generally,
framework selection is a matter of personal preference and each one should
choose whatever aligns with their philosophy. For me, it was scalatra.

Frankly, I haven't been able to ship out features at the speed at which I used
to, with Rails, but there was something very important that I learnt too -
Don't get comfortable with full stack frameworks without understanding what
goes underneath. Rails suddenly seemed like an enemy to me because it was
doing everything so magically that I literally had NO IDEA until I had to
implement them myself with Scalatra. For example, do you know what AES-GCM is?
Do you know HOW the secure cookie store in Rails works? What algorithm it
uses?? Do you know you have to change the secret pretty often??? Do you know
how authentication in Rails (Devise) works??? Can you build it from
scratch???? I knew none of these, until I had to implement them myself for my
application on top of Scalatra. It was seriously a pain because for the first
one week, I did absolutely nothing, in terms of progress with features. But
later, I started loving this way of development rather than relying on 'magic'
frameworks. Now 'magic' seems scary for me because I cannot actually see
what's happening underneath.

So, to cut a long story short, Should you choose the fastest framework? Or
should you follow the 'ship it first' policy with an average framework??

My advice - Start with a full stack framework, something like Rails or Django,
but also in parallel, try to understand how every bit of your framework works
and try to implement it into a micro framework based on something from the top
of the benchmarks (like Go, Scala, etc.). Most importantly LEARN. Something
new everyday! And slowly shift your development towards these high performant
frameworks as and when you see fit.

This is just based on my experience :)

[1] 1 JVM = 10 Thins, Source: [http://www.slideshare.net/pcalcado/from-a-
monolithic-ruby-on...](http://www.slideshare.net/pcalcado/from-a-monolithic-
ruby-on-rails-app-to-the-jvm)

~~~
gog
I had similar ambitions, and I wanted to play with the top performers, so I
started looking at the Java (gemini) ecosystem.

And after an hour of surfing I remembered why I didn't follow through the last
time I had this idea. The Java ecosystem is complex, I mean really complex.

Here are some of the terms I came by that look like I should grasp before
trying to make something: servlet (various versions), servlet container
(resin, tomcat, jetty, ...) , OSGi, jar, war, wab, jboss, glassfish, java
standard/enterprise edition, where are classes being imported from,
configuration files for different containers, JNDI, classpath...

Are there any resources for Java like [http://mirnazim.org/writings/python-
ecosystem-introduction/](http://mirnazim.org/writings/python-ecosystem-
introduction/) for Python or
[http://www.phptherightway.com/](http://www.phptherightway.com/) for PHP that
would help someone to start navigating all those java-isms?

"Getting started" article for developing web applications with gemini would
also be helpful.

~~~
krg
Hi gog,

I'm sorry you spent time trying to get started with Gemini, because it's our
internal framework that isn't released yet. That would have been frustrating
for you!

From our "Motivations" section:
[http://www.techempower.com/benchmarks/#section=motivation](http://www.techempower.com/benchmarks/#section=motivation)
"Why include this Gemini framework I've never heard of?" We have included our
in-house Java web framework, Gemini, in our tests. We've done so because it's
of interest to us. You can consider it a stand-in for any relatively
lightweight minimal-locking Java framework. While we're proud of how it
performs among the well-established field, this exercise is not about Gemini.
We routinely use other frameworks on client projects and we want this data to
inform our recommendations for new projects.

~~~
gog
Thank you,

now I understand why I couldn't find anything useful on this page
[http://www.eclipse.org/gemini/web/](http://www.eclipse.org/gemini/web/) :)

Nevertheless I am still looking for good resources on getting started with
Java (ecosystem wise, not the language itself).

~~~
columbo
Have you considered starting with Grails? It's a much easier jump into the
"java-ish" world. The main benefit is you can choose to run a grails
application and rely on java jars for when you need performance boosts.

~~~
wasd
We've been using Groovy internally for a few years for prototyping and
experimentation. I thought about slowly introducing Grails as an alternative
to Spring for some projects. My experience was that it vaguely reminded me of
Rails but needed a lot more polish. I had a huge issue trying to chase down
archaic stacktraces and found that there is little developer momentum pushing
it forward. I usually found myself googling and reading random blogs to figure
out simple things I couldn't infer from the docs. Where as Rails magically
makes things work, I found that a lot of times Grails made things magically
not work. I wouldn't personally suggest it for anything customer facing.

~~~
vorg
You could use JRuby and Rails if you need the JVM.

Groovy's OK for wraparounds testing and/or running Java classes, what it was
originally intended for, bringing closures and terser list/map syntax (altho
JRuby et al have those too). It's all rather slooooow though. Grails uses
Groovy's MOP which was added later, but I don't use them much. I toyed with
Groovy's interceptors and categories when they were added, even wrote notes on
them, but they broke in the upgrade from 1.5 to 1.6. There's a lot of such
cruft in Groovy which just lapses from version to version through lack of use
and support. They brought in AST transforms to get programmers to extend
Groovy's functionality, but when they do, the Groovy managers wil swipe the
code and pass it off as their own in the next version of Groovy (e.g. Groovy++
written by Alex Tkachman was cloned as the static compilation in Groovy 2.0
and the static traits in the upcoming Groovy 2.2).

tldr... Use grOOvy for quickies handling Java classes, and for Grails where
it's required, but look at a serious programming language for anything larger.

~~~
wasd
I would much prefer using JRuby/Rails but we have a bus factor of 1 with Ruby
:(.

------
VeejayRampay
For such a young language, I'm really surprised by the performance of Go
there. Quite awesome.

~~~
YcombRegBroken
Round 4 of these tests is what triggered my dive into Go this past month, and
it has been a revelation. If you add in the concurrency capabilities, the
language simplicity, and nice balance between overly terse syntax (Python one
liners...) and overly verbose typed languages (Java), the whole package is
even more impressive.

The only real thing I think it is missing is a "high productivity" application
framework like Rails to elevate it from "damn, I'm impressed" to "it would be
ill advised to use almost anything else".

</reluctant fanboy rave>

~~~
Peaker
Have you tried modern ML-style languages? e.g: Haskell, OCaml, F#?

They also combine the nice conciseness of Python, with great concurrency
(better than Go) and a lot more safety (less late night crashes).

~~~
TylerE
With respect, and I say this having experimented with Haskell, OCaml, any many
other functional languages as well, just, no. Those languages come at a MUCH
higher mental cost than go. Go is wonderful because writing it doesn't feel
like I'm attempting a CompSci doctoral thesis. I can also say I have yet to
see the Go runtime crash, and I am using in production.

I also challenge your "better concurrency" claim.

~~~
Peaker
What do you mean by "mental cost" here? Cost of initial learning? Then I
agree... But learning a language is an O(1) cost that takes a few weeks to a
few months. We program with the language for many years.

I never said the Go runtime will crash, but your programs will, because of
unsafe nullability, mutability in the concurrently shared state, and various
other problems in Go.

Better concurrency in Haskell: In addition to having light-weight threads like
Go and channels like Go, Haskell also has software transactional memory which
Go lacks. Haskell also has `par` annotations on pure expressions that allow
parallelization with guaranteed lack of effect on program semantics. Haskell
also has data parallelism.

~~~
YcombRegBroken
Haskell (and most of the other languages mentioned) is exceptionally clever.
This, above all else, is it's downfall. For me, especially when tasked with
building a high productivity development team, clever code is a ticking time
bomb. It's easy to write, but hard to maintain and modify. It requires more
mental ram to analyze any given piece of code, and is much more difficult for
multiple programmers to contribute to. It's tiny, to be sure, but, again, I
think that is negative, not a positive. Maximum clarity is not minimal code.

Go, on the other hand, is not clever. It's boring as hell, honestly. This is a
Very Good Thing® when it comes to building out a dev team, and I feel, the
single biggest reason Google put the resources into creating it.

As for the crashing, of course I see some dangerous areas. Educating
developers on avoiding a small regions of pitfalls is much easier than
managing a team of clever coders though.

~~~
Peaker
I disagree that Haskell is "clever". I think Haskell is "smart".

Using "Maybe a" when you have a nullable value is smart, not "clever". It aids
maintenance and readability, not hampers it.

Using pattern matching is the same.

Haskell builds on mathematicaly simplicity, which makes things hard to grasp
at first. This may be mistaken for cleverness.

Unless I'm misunderstanding you -- can you give an example of something clever
in Haskell?

~~~
YcombRegBroken
I'm not a haskell expert (obviously), but the time it takes me to parse things
like this is time I would rather spend reading 5-10x the number of lines and
getting the meaning right away.

    
    
      let loeb x = fmap ($ loeb x) x in 
      loeb [ (!!5), const 3, liftM2 (+) (!!0) (!!1), (*2) . (!!2), length, const 17]
    
    

(btw, I have no idea what the hell this does. Something to do with
spreadsheets, apparently. I found it on
[http://www.haskell.org/haskellwiki/Blow_your_mind](http://www.haskell.org/haskellwiki/Blow_your_mind),
which has enough cleverness to make me want to cry)

I've seen cleaner and more readable code in production haskell, but this sort
of thing happens enough that I'm very cautious.

~~~
efnx
I think the first paragraph of the page linked explains why that code is so
impenetrable. You will not have to read or write Haskell like that, ever. Good
find though! On a similar note, the Haskell community is amazing. You can
learn absolutely everything between the Haskell wiki, freenode irc and
hackage. How great is it that EVERY lib/package/framework is documented on
hackage in exactly the same format? Very great. Coming back to JavaScript is a
bummer :(

------
antihero
How are certain languages _so_ much faster even when pulling from the
database? Surely the database would be the bottleneck here? Surely if it's
something cool the driver is doing, that could be implemented in other
drivers?

Also, why are some frameworks on there a bunch of times (e.g. ASP MVC)?

~~~
continuations
> Surely the database would be the bottleneck here?

The idea that "it doesn't matter which language you use because DB would be
your bottleneck" might have been true 10 years ago but it's definitely not
true anymore.

Latest versions of MySQL and Postgresql are extremely fast. They are written
in C/C++ and heavily optimized. MySQL serving data from RAM can easily do
500K+ rps on a commodity server. Even if you need to go to disks there are
$200 SSD's that give you almost 100K IOPS. Put a few of them in a RAID and
you're getting hundreds of thousands of IOPS for very little money.

In all likelihood application code written in Ruby/Python/PHP with frameworks
piled on top of abstractions is going to be orders of magnitude slower than
the DB server written in C that has undergone 15+ years of heavy
optimizations.

~~~
antihero
Ok, but say we use vanilla Python WSGI with a psycopg2 (C Python PostgreSQL
extension) directly. Will this come close to the performance of the C++
framework thing?

~~~
continuations
I'd say a vanilla Python application without ORM would perform similar to a
flask app without ORM. According to
[http://www.techempower.com/benchmarks/#section=data-r6&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r6&hw=i7&test=db)

flask (on Gunicorn): 6,138

flask (on PyPy): 8,167

Django + ORM : 4,026

cpoll-cppsp : 114,711

So while a light weight Python app without ORM might be 50% faster than a
full-blown Django app, it's still almost 20 times slower than a C++ app.

That shouldn't be surprising, by all accounts Python is at least an order of
magnitude slower than C++.

You could speed up your Python program by running PyPy. According to the
benchmark that gives it a 30% speed boost, which is in line with my
experience. Another way is to deploy on uWSGI or Meinheld instead of Gunicorn.

No matter what you do, Python will not be close to the speed of C++. But what
really matters is: is it fast enough for your need?

------
rip747
i would really like to see how CFML engines (Railo, Adobe ColdFusion) and
frameworks (FW/1, CFWheels, ColdBox) stack up in this comparison.

FD: I contribute to CFWheels

~~~
bhauer
We'd love to receive these test cases as pull requests if you have the time to
contribute them. :)

[https://github.com/TechEmpower/FrameworkBenchmarks](https://github.com/TechEmpower/FrameworkBenchmarks)

~~~
rip747
checking out the documentation now. is there a google group or some place I
can post questions to if I need help?

~~~
bhauer
Indeed!

[https://groups.google.com/forum/?fromgroups=#!forum/framewor...](https://groups.google.com/forum/?fromgroups=#!forum/framework-
benchmarks)

------
elsurudo
I would love to see how the Goliath async server (Ruby version of Node.js is
one way of looking at it - [http://goliath.io](http://goliath.io)) performs,
as I've recently started looking into it for a part of a project.

~~~
ksec
I was just about to post this. And will the next round be tested on Rails 4.0?

~~~
mhixson
The best way to ensure Goliath is tested is to submit a pull request.

We'd rather test Rails 4 than obsolete versions of Rails:
[https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...](https://github.com/TechEmpower/FrameworkBenchmarks/issues/352)

------
clhodapp
The play-slick results seem to be missing. Was there some sort of problem?

~~~
bhauer
I had not named it uniquely. I've changed its name to "play-slick." Thanks for
pointing that out!

~~~
drummingds
Thanks for updating that. I noticed that the Slick version seems to outperform
the regular Anorm version in most tests. Do you have any insight into why that
might be? I figured it would've been the other way around.

------
evab
Can anyone explain why the "gemini" framework with an ORM outperforms raw
servlets/queries in almost all the cases?

What kind of optimizations are done ?

~~~
bhauer
Evab, Gemini is our in-house framework [1] that we've included in these tests
for our own learning experience. We're not obsessed with performance, but we
do try to avoid problems that are easily avoided. :) For instance, where
possible, we have used Java's concurrent data structures and reduced the use
of locks.

A difference in these tests is that we have our own connection pool. Speed was
not a main objective when we wrote it--it's so old that if my memory is
correct, it predates the availability of the connection pool that now ships
with the MySQL driver, which is what we've used in the Servlet tests.
Nevertheless, my conjecture is that perhaps it's just a tad quicker at
selecting an idle connection. Incidentally, for a while in an earlier round,
the Go guys had an alternate test implementation that used a concurrent queue
for connections and it performed fantastically. We've been toying with
changing Gemini to do the same at some point.

For the benchmarks project, we classified ORMs as "raw" (no ORM), micro, or
full. In Gemini, we have a micro ORM. The following are examples of its use in
the test code [2]:

    
    
        // Get all rows from Fortunes table.
        final List<Fortune> fortunes = store.list(Fortune.class);
    
        // Get a random World row.
        worlds[i] = store.get(World.class, random.nextInt(DB_ROWS) + 1);
    
        // Run a batch of updates from a List of entity objects.
        store.putAll(Arrays.asList(worlds));
    

"Micro ORM" is loosely defined, but we've applied it to ORM options that offer
some abstraction over plain old SQL, but aren't as comprehensive as the
standard-bearers. For example, while we have a data structure that represents
relationships, we do not have a higher-level query language that traverses
relationships automatically nor any similar mechanism for easy relationship
traversal. It turns out that across all of these frameworks, there are dozens
of what we are calling Micro ORMs.

Getting back to your question, the Gemini ORM does leverage a few tricks for
performance such as ReflectASM for object creation and prepared statements for
queries. But there isn't a whole lot of fancy work going on.

It's not glamorous, but the most important high-performance elements we make
use of in Gemini are the JVM platform and Java's concurrent data structures.
As demonstrated by the likes of Undertow (the web server component of JBoss
WildFly), mind-boggling performance is available on the JVM.

[1] [https://groups.google.com/d/msg/framework-
benchmarks/p3PbUTg...](https://groups.google.com/d/msg/framework-
benchmarks/p3PbUTg-Ibk/fOjQMxESEJ0J)

[2]
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/gemini/Source/hello/home/handler/HelloHandler.java)

------
npalli
Does anyone have more info on "cpoll-cppsp"? Hearing about it for the first
time, it seems to be smoking all the other frameworks.

~~~
ambiate
I was actually just checking it out for myself.

[http://sourceforge.net/projects/cpollcppsp/](http://sourceforge.net/projects/cpollcppsp/)

[https://github.com/xaxaxa/workspace/](https://github.com/xaxaxa/workspace/)

Seems to be a C++ framework with dynamics in the form of C++ code using an ASP
style for embedding into the resource.

------
ditados
i don't see any mention of uwsgi or gevent in these benchmarks. Deploying
Python apps without properly configuring the container/webserver will skew the
results considerably (probably applicable to a couple of other entries as
well).

------
continuations
cpoll-cppsp made huge performance gains from round 5 to 6. For example
multiple queries went from 1,872 rps in round 5 to 7,252 in round 6. What
caused that?

Also what happened to cpoll-cppsp in the plaintext test? Its performance there
took a deep dive.

~~~
bhauer
I know that the contributor of those tests made several changes. We weren't
able to identify why it fails dramatically on the plaintext test, but I
suspect it's a combination of the two unique characteristics of that test: the
use of HTTP pipelining and higher client-side concurrency. My guess is that
the pipelining in particular is causing problems.

~~~
MetaCosm
I really hope you guys start doing more in-depth high-concurrency testing.
This is a huge risk / problem with _many_ frameworks and kits. It seems to be
a blackhole in your tests for some reason, which is a shame because the are so
well done in other regards.

Having a framework which falls down horribly at high concurrency is something
people need to know about -- in how they structure and design applications and
how they structure deploys. It can also help them pick frameworks that fit
them better (some frameworks will do great at high concurrency, some won't).

Can't wait till you start adding high concurrency (which should NOT require
any code changes from the implementation providers, you add a zeroes to the
concurrency.. 100, 1000, 10000 tests).

------
camus
Nice, Some solutions are quite fast , but as soon as the database needs to
perform heavy stuffs , well the perfs decrease dramatically. The really
bottleneck seems to be the database in most cases. However ,some frameworks
are basically quite slow.

------
cmccabe
What about Cobol on COGS?

