Theres a consistant, considerable gap between their "raw" benchmarks (things like netty, node, plain php, etc.) and frameworks hosted on those same platforms. I think this is something we should keep in mind when we're tuning performance-sensitive portions of APIs and the like. We may actually need to revisit our framework choice and implement selected portions outside of it (just like ruby developers sometimes write performance-critical pieces of gems in C etc.) or optimize the framework further.
I'd like to crunch these numbers further to get a "framework optimization index" which would be the percentage slowdown or ratio of performance between the host platform and the performance of the framework on top of it. I might do this later if I get a chance.
Benchmark_Routes 100000 13945 ns/op
Benchmark_Pat 500000 6068 ns/op
Benchmark_GorillaHandler 200000 11042 ns/op
Benchmark_Webgo 100000 26350 ns/op
ok github.com/bradrydzewski/routes/bench 12.605s
Benchmark_Routes 100000 21446 ns/op
Benchmark_Pat 100000 14130 ns/op
Benchmark_GorillaHandler 100000 17735 ns/op
Benchmark_Webgo 50000 33726 ns/op
ok github.com/bradrydzewski/routes/bench 9.805s
Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.
I would personally choose my framework not just based on performance, but also based on productivity. One that can help me write code that is easier to test and easier to maintain in the long run.
> Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.
If you're performing a DB query on every request, you're doing something wrong. In the real world your app will check Memcached, and if there's a cached response, it will return it. Thus making the framework performance quite important.
Benchmark_Routes 10000 234063 ns/op
Benchmark_Pat 10000 233162 ns/op
Benchmark_GorillaHandler 5000 265943 ns/op
Benchmark_Webgo 5000 348349 ns/op
ok github.com/bradrydzewski/routes/bench 10.062s
More SPEEEEEEEEEEEEED coming down the pipe in 1.1 for Go's net/http. :)
Some of the other platforms are much less amenable to that since the standard primitives the platform exposes are very primitive indeed (servlets api, rack api, etc.). Perhaps there's some value in looking at how your favorite framework stacks up against its raw platform and trying to contribute some optimizations to close the gap a bit.
Given that the json marshalling and server components would be exactly the same between go and webgo, I'm curious as to whether changing the url recognised to be just /json in the goweb tests would make any difference, any reason it was different?
Shouldn't all these examples at least be trying to do the same sort of work? For such a trivial test differences like that could make a huge difference to the outcome.
It's great to see replicable tests like this which show their working, but they do need to be testing the same thing. I also think they should have something more substantial to test as well as json marshalling on all the platforms, like serving an HTML page made with a template and with the message, as that'd give a far better indication of speed for tasks web servers typically perform.
Still, it's a great idea and really interesting to see this sort of comparison, even if it could be improved.
I can't think of much else that this little web.go framework does (assuming the fcgi bits etc are unused now and it has moved over to net/http). I don't think many people use web.go, gorilla and its mux router seems to be more popular as a bare bones option on Go, so it'd possibly be interesting to use that instead. It'd be great to see a follow up post with a few changes to the tests to take in the criticisms or answer questions.
While you may come in for a lot of criticism and nitpicking here for flaws (real or imagined) in the methodology, I do think this is a valuable exercise if you try to make it as consistent and repeatable as possible - if nothing else it'd be a good reference for other framework authors to test against.
Also, They used Go 1.0.3... I hope they update to 1.1 next month. Most everyone using Go for intensive production uses is using Go tip (which is the branch due to become 1.1 RC next month)
To select a suitable tip build we use http://build.golang.org/ and https://groups.google.com/forum/?fromgroups#!forum/golang-de... . My recommendation would be to find a one or two week old build that passed all checks, do a quick skim of the mailing list to make sure there weren't any other issues and use that revision. Also, you will see some the the builders are broken-
Of course if your application has automated unit tests and load tests, run those too before a full deployment.
If stable means suitable for production, Go tips vastly improved performance, especially in regards to garbage collection, make it more suitable than 1.0.3 for large/high-scale applications in production.
Also, the 1.0.3 thing is probably dragging on the numbers a bit. 1.1 would boost it a little. Not enough to get it into the top tier... but a little.
Also, for Vert.x, they seem to be only running one verticle. Which would never happen in real life.
Play could be optimized a bit... but not much. What they have is, to my mind, a fair ranking for it.
Small issues with a few of the others but nothing major. I think Go and Vert.x are the ones that would get the biggest jumps if experts looked at them. And let's be frank... does Vert.x really need a jump?
So what they have here is pretty accurate... I mean... just based on looking through the code. But Go might fare better if it used tip. And Vert.x would DEFINITELY fair better with proper worker-nonworker verticles running.
the Vert.x example, as configured, blocks massively as well waiting for mongodb queries.
Also tested Cake just because it had been a while since I have used it... and I couldn't believe it was that much worse than PHP. Your numbers there seem valid though given my test results. That's pretty sad.
Finally, tried to get in a test of your Go stuff. I'm making what I think are some fair changes ... but it did not speed up as much as I thought. In fact, initially it was slower than your test with 1.1.
So after further review... well done sir.
>> I'm not sure many people would use [xxx] in real life. I don't know... maybe some people... certainly not pros.
We tested node.js with both Mongo and MySQL. Mongo strikes us the more canonical data-store for node, but wanted to include the MySQL test out of curiosity.
Framework Framework Index
cake 18.9% (312 vs 59)
compojure 12.1% (108588 vs 13135)
django 16.8% (6879 vs 1156)
express 16.9% (42867 vs 7258)
gemini 12.5% (202727 vs 25264)
go 13.3% (100948 vs 13472)
grails 7.1% (28995 vs 2045)
netty 18% (203970 vs 36717)
nodejs 15.6% (67491 vs 10541)
php 11.6% (43397 vs 5054)
play 20.6% (25164 vs 5181)
rack-jruby 15.6% (27874 vs 4336)
rack-ruby 22.7% (9513 vs 2164)
rails-jruby 22.7% (3841 vs 871)
rails-ruby 20.7% (3324 vs 687)
servlet 13.4% (213322 vs 28745)
sinatra-jruby 21.2% (3261 vs 692)
sinatra-ruby 22.2% (6619 vs 1469)
spring 7.1% (54679 vs 3874)
tapestry 5.2% (75002 vs 3901)
vertx 22.3% (125711 vs 28012)
webgo 13.5% (51091 vs 6881)
wicket 12.7% (66441 vs 8431)
wsgi 14.8% (21139 vs 3138)
I found it interesting that something like tapestry took a 20x slowdown when going from dedicated to ec2, while others only took ~5x slowdown.
Edit: To hopefully make it clearer what the percentages mean - if a framework is listed at 20%, this means that the framework served 1 request on ec2 for every 5 requests on dedicated hardware. 10% = 1 for every 10, and so on. So, higher percentage means a lower hit when going to ec2.
Disclosure: I am a colleague of the author of the article.
So, in responses per second, the throughput on ec2 was 5.2% that of dedicated hardware, or approximately 20 times less throughput. The use of the word slowdown was possibly a bad choice, as none of my response had to do with the actual latency or roundtrip time of any request.
Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected.
Express and Sinatra really stand out as widely-used, very high percentage of platform performance retained frameworks here. I've never used Vert.x, but I will certainly look into it after seeing this. I'm very impressed that Express is so high on this list when it is relatively young compared to some of the others and the Node platform is also relatively young.
Play seems particularly disappointing here since it seems any good performance there is almost entirely attributable to the fast JVM it's running on. Compojure is also a bit disappointing here (I use it quite a bit).
Here's the source: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
So the question stands: If Play & Netty are using the same JSON serialization code, why is Play seven times slower?
I too wanted Play to show higher numbers. There's certainly a possibility we have something configured incorrectly, so we'd love to get feedback from a Play expert.
About the cost per additional line of code for PHP, it mainly comes from not having an opcode cache and having to load and interpret files on every visit. mod_php was and will always be trash. I commented earlier about it too.
In case of Ruby, and talking about Rails, even when using Passenger, the rails app is cached via a spawn server. That's not the case with PHP.
Similarly, Python has opcode cache built-in (.pyc files). Also, I am not sure about gunicorn but others do keep app files in-memory.
Servlet does the same about keeps apps in memory. You get the idea.
Frameworks definitely have an impact but it's very hard for one person to know the right configuration for every language. You had done some good work there, but it will take a lot of contributions before the benchmark suite becomes fair for every language/framework.
We're hoping to post a follow up in about a week with revised numbers based on the pull requests and any other suggested tweaks we have time to factor in. We're eager to see the outcome of the various changes.
See e.g. https://github.com/playframework/Play20/blob/master/document... and https://gist.github.com/guillaumebort/2973705 for more info.
Do you mind breaking it down for me a bit please?
Cost in dollars or cost in hardware utilization or some other cost?
Knowing how much I'm giving up in performance in order to get the features a given framework gives me is an important consideration. Also understanding when it's worthwhile to work outside the framework on the bare platform given the speedup I'll get versus the cost I'll incur by doing so is a very important optimization decision-making tool.
See my "framework optimization index" in comments below for a rundown on all these ratios which I was able to back out from this set of benchmarks.