Hacker Newsnew | comments | show | ask | jobs | submitlogin
bradleyland 471 days ago | link | parent

Preface: This post is going to come across as a Rails apologist piece, but please read the entire thing before you reach a conclusion. Please also consider that you could apply these same arguments to just about any of the high-level language based frameworks on the list. I use Ruby on Rails in my comparisons, but I'm a huge fan of Node.js, Python/Django, and Go.

I fully respect the JVM family of languages as well. I just think that Mark Twain said it best when he said: "There are three kinds of lies: lies, damned lies, and statistics." It's not that the numbers aren't true, it's that they may not matter as much, and in the way, that we initially perceive them.

Performance is certainly something you should consider when selecting a language/framework, but it is not the only thing.

========================

You should undertake a detailed examination of these statistics before making any decisions.

Issue #1) The 30-50x performance difference only exists in a very limited scenario that you're unlikely to encounter in the real world.

Look carefully at the tests performed. The first test is an extraordinarily simple operation: take this string, serialize it, and send it to the client. This is the test in which we see massive differences:

Gemini vs Rails

25,264/687 (gemini/rails-ruby) = 36.774

25,264/871 (gemini/rails-jruby) = 29.000

Node.js vs Rails

10,541/687 (nodejs/rails-ruby) = 15.343

10,541/871 (nodejs/rails-jruby) = 12.102

That's a 37x performance win for Gemini, and 15x for Node.js.

Side note: You might be wondering why I didn't compare to the top performer, Netty. Netty is more like Rack. You build frameworks on top of Netty, not with Netty. As a Ruby dev, you could think of this in the same context of comparing Ruby on Rails with Rack; not a good comparison. Hence We won't compare Rails to Netty.

The error would be in extrapolating that a move to Gemeni or Node.js would give you a 37x or 15x performance increase in your application. To understand why this is an error, we jump down to the "Database access test (multiple queries)" benchmark.

Issue #2) Performance differences for one task doesn't always correlate proportionally with performance differences for all tasks.

In the multi-query database access test, we start to see the top JSON performers slow down significantly when compared to the slow down for Rails:

Gemini vs Rails

663/89 (gemini/rails-ruby) = 7.449

663/108 (gemini/rails-jruby) = 6.138

Node.js vs Rails

116/108 (nodejs-mysql-raw/rails-ruby) = 1.077

60/108 (nodejs-mysql/rails-ruby) = 0.555

In this scenario -- which is arguably much closer to the real world -- Ruby on Rails closes the gap and even beats some of the hip new kids.

But why? The in-depth answer to this question would require a lot of space, but the really, really short version is kind of a "what's the sound of one hand clapping" response: Ruby isn't actually all that slow.

To understand what the hell that means, check out this presentation from Alex Gaynor (of rdio/Topaz fame):

https://speakerdeck.com/alex/why-python-ruby-and-javascript-...

Ruby is just about as fast as C, provided you're comparing it to C that does exactly the same operations on the hardware as the Ruby code. Don't get me wrong, that's a HUGE provision. But it warrants close examination.

The real benefit of lower level languages like C is that they give you the flexibility to drill down in to your actual bare-metal operations and optimize the way the program executes on the hardware. As Alex points out, we don't currently have that level of flexibility in languages like Ruby (without dropping down to inline C), so we suffer a performance penalty.

This penalty is huge for simple tasks because they involve only a handful of operations that execute extremely quickly. As you add complexity, however, the benefits of micro-optimizations get lost in the vastness of the overall execution time.

Look at it like this. When Gemini hits 36,717 req/s in the JSON test, each request only lasts about 1.6 ms. This is only possible because of the simplicity of the operations being done on the hardware. Ruby loses big here because there is a lower boundary to the way you can optimize without dropping down to C.

gemini: 1.6 ms per request

rails-ruby: 87.3 ms per request

When we look at the multi-query database access test, we can see how the optimization at the low level gets lost in the sea of time taken to process the request.

gemini: 90.5 ms per request

rails-ruby: 674.2 ms per request

Granted, that is still over a 7x performance win for Gemini, but this is where the Ruby arguments about programmer efficiency come in to play. I don't know Gemini, so it may very well beat Rails in that comparison too. Ruby is getting more performant with every release though, so it's easier to justify on the basis of preference alone when we're this close.



chrisdone 470 days ago | link

Don't conflate Ruby with Rails. Ruby _is_ slow:

http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...

http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...

and so is Python:

http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...

The only time Ruby or Python are fast is when the program is not running Ruby or Python but running some C code underneath. If your programs only consist of that, you can very well say "this ‘Ruby/Python’ code is fast". But as soon as you have something that isn't in your standard library, welcome to the actual language, and welcome to performance problems.

Elaborating on the implications of this: whenever you actually _use_ the language to do some abstraction, you pay heavily for it: http://bos.github.com/reaktor-dev-day-2012/reaktor-talk-slid...

-----

bradleyland 470 days ago | link

You should really check out Alex Gaynor's slide deck. Nothing I said disagrees with what you've said here, provided you take the entire thing in context.

-----

jalopy 471 days ago | link

That's a very detailed and thoughtful response. You make some valid points. Maybe I'll try to craft some more complicated benchmarks that replicate normal CRUD operations found in most webapps.

-----

bradleyland 471 days ago | link

That's kind of missing the point. In statistics, there exists something called confounding variables. Confounding variables are factors that affect your outcome, but are hidden, or are outside your control. As your example becomes more complex the opportunity for confounding to impact your result goes up significantly.

I believe the multi-query database access test is actually a good example of a "complex enough" test, but not too susceptible to confounding. In this test, we see that Rails isn't so far behind.

Basing your choice of framework on speed alone is a pretty bad idea. When you select a framework, you need to optimize for success, and the factors that determine the success of a project are often less related to the speed of a framework and more related to the availability of talent and good tools.

That's not to say you should ignore speed entirely, but that you have to weight your factors. There is a tendency to believe that you will need rofflescale when you really won't. Keep that in mind when you're weighting your factors.

-----




Guidelines | FAQ | Lists | Bookmarklet | DMCA | News News | Bugs and Feature Requests | Y Combinator | Apply | Library | Contact

Search: