

Is Ruby Fast Yet? - j_s
http://www.isrubyfastyet.com/

======
Freaky
> The benchmark premptively runs the fans and pauses between runs in order to
> cool off. These measures give JRuby at least a 25% speed gain. However,
> JRuby still might be running slower than it would on a box with appropriate
> cooling.

Right. Your computer can't be trusted to not throttle its performance due to
crap cooling, and you're trying to _benchmark_ using it?

~~~
miahi
The choice of hardware is quite strange. Should be fun to check the results on
normal hardware though.

------
bri3d
Without knowing what runtimes and parameters are used it's really hard to tell
what's going on in this benchmark; JRuby can see 100% and sometimes more
improvement on many workloads by tweaking JVM options.

It looks like JRuby is just run under the system JRE (which is what?) with no
additional options; that's going to be slow for most workloads.

It'd be nice to see a lot more documentation around the runtime configurations
and what the Rails app itself is doing.

~~~
hornetblack
Down the bottom he states that it is running on Osx. The fan is premptively
spun up, which gives Jruby a speed boost. It's probably going to run even
faster with proper cooling.

------
RyanZAG
Not sure how comparing different versions of rails shows if its 'fast' yet.
I'd think setting up some request target and seeing how far RoR is from that
target with each version would be better. Or comparing RoR to other platforms
is also a good bet, eg
[http://www.techempower.com/benchmarks/](http://www.techempower.com/benchmarks/)

I just don't really see how this answers the question other than 'ehh I think
it looks a bit faster than it was awhile ago?'

EDIT: Also interestingly, JRuby seems to be slightly better on the small
instance EC2 test at techempower than regular Ruby (small instance should be
similar to a laptop?). Yet in these graphs it's much slower. Anybody know the
reason?

~~~
jarrett
Amazon says their "m1.small" instance has 1.7 GiB of memory. That's a lot less
than most current laptops.

Maybe JRuby uses more memory in total, but can handle more concurrent requests
per MB of memory used? That could explain the discrepancy if it's true, though
it's just speculation on my part.

Another theory is that the benchmarks are too application-specific. Rails is a
complex framework with which you can build many different kinds of
applications, each with its own performance characteristics. Conceivably,
application A could perform better on JRuby, while application B could perform
better on MRI. Depending on which application you use for your benchmark,
either interpreter could appear faster.

This calls into question the utility of benchmarking an entire framework. It
might be better to do micro-benchmarks, with each one testing a specific
scenario. Scenarios could include, for example: A request that loads 1,000
database records into memory; a request for a static image; a request that
queries a many-to-many relationship; a request that saves an uploaded file to
disk.

~~~
lukeschlather
The thing is, those are all I/O bound operations and in the case of the
databases most of the code you're benchmarking is native extensions, so I
would expect the difference to be negligible.

Testing whole Rails apps really does seem like the best way to get good data.
The interesting speedups are likely going to be in places like string matching
/ string copying and other places that don't involve network access.

------
Jgrubb
JRuby question -- Why is it that much slower? I'd have thought that part of
the point was to take advantage of the JVM's built in optimization.

~~~
mml
My question as well. Every perf benchmark I've seen of jruby over the years,
it's much faster than MRI (given time to boot & JIT anyway).

I think this guy's numbers are very suspect.

~~~
jshen
I've benchmarked MRI vs jruby a number of times over the years on real apps
and MRI usually cones out faster on average with worse numbers at 95th and
98th percentiles due to GC performance.

I never took the time to figure out why it is faster, but I assume that it is
due to C libraries being faster for json serialization and my apps doing
primarily json serialization after fetching from the database.

Edit: I should test the performance of the json libs, which would be easy to
do

------
matthuggins
Wouldn't it make more sense to test various features of Ruby for comparison
rather than comparing an entire web framework? Given that, the URL should be
israilsfastyet.com, or more appropriately
howdoesrailsbehaveonvariousrubyversions.com

~~~
mikesilvis
Agreed. This is much more of a test of the rails framework then ruby

~~~
jrochkind1
Well, it's a comparison of different ruby versions, using Rails. That's pretty
useful to people who use Rails, which is a lot of people.

Of course, it's not a real Rails app presumably, but just a sample/dummy one.
So you may or may not see comparable speed-up with a real app. But it's
probably suggestive of the upper limit of improvement you can get by switching
ruby versions.

"Will I get a performance improvement from my Rails app by upgrading ruby" is
a real question a lot of people will have. Micro-benchmarks of ruby wouldn't
answer it as well as the OP does.

------
timmorgan
I'm surprised at how negative the comments are here. Graphing how different
Ruby versions/implementations perform running Rails is very helpful! Glad to
see that MRI continues to get faster with every version. Merry Christmas!

~~~
jarito
The negativity comes from the fact that this benchmark is useless and, at
worst, misleading. True benchmarks are hard to get right. Running a bunch of
stuff on a laptop isn't sufficient evidence of anything.

Kudos to making the benchmark opensource at least, but even from a quick peak,
I wouldn't trust this at all. As others have mentioned, JRuby shouldn't be
this slow. Odds are the benchmark is throwing away the JIT gains or the
configuration is poor. Add that to the fact that there is no discussion of
what a 'rails request' is or what version of Java is being used and there
seems to be little to learn from this research.

~~~
regularfry
While I'm sure it's not the aim of this site, there is some small value in
knowing what sort of performance you get with a naive, untweaked deployment.

~~~
brianhempel
Website author here. On the contrary, you have nailed it. Noob-friendlyness is
a valuable part of a language ecosystem.

------
munificent
I've been hacking on a little scripting language of my own lately[1]. To get a
handle on its perf, I have a few small benchmarks[2] I run that compare it
against Lua (5.1.5), Python (2.7.5), and Ruby (2.0.0p247). (All of those and
my language are bytecode-compiled, dynamically typed languages, hence that
comparison.)

I happen to have Ruby 2.0 installed, and I was surprised to see it handily
beat the other languages. I had expected Ruby to be the slowest of the pack,
but they've apparently made big strides in 2.0.

On the off chance that anyone is curious, here's the results of running them
on my machine:

    
    
        binary_trees - wren   3531  0.28s
        binary_trees - lua    1366  0.73s  258.40%
        binary_trees - python 1420  0.70s  248.64%
        binary_trees - ruby   3170  0.32s  111.39%
    
        fib - wren            2688  0.37s
        fib - lua             2998  0.33s   89.65%
        fib - python          1390  0.72s  193.38%
        fib - ruby            3645  0.27s   73.74%
    
        for - wren            9092  0.11s
        for - lua             9387  0.11s   96.86%
        for - python          3171  0.32s  286.74%
        for - ruby            8319  0.12s  109.30%
    
        method_call - wren    4703  0.21s
        method_call - lua     1780  0.56s  264.20%
        method_call - python   806  1.24s  583.33%
        method_call - ruby    3408  0.29s  138.02%
    

Those numbers are score, time, and percentage. Score is basically 1/time, just
to yield a number where bigger is better. Time is best of ten runs. Percentage
is how my language's score compares to that one. So, for example, the 264.20%
lua method_call score means my language is about 2.6x faster than Lua on a
benchmark designed to stress dynamic method call performance.

[1] [https://github.com/munificent/wren](https://github.com/munificent/wren)
[2]
[https://github.com/munificent/wren/tree/master/benchmark](https://github.com/munificent/wren/tree/master/benchmark)

~~~
StefanKarpinski
Oh, man, this reminds me of the pain of writing even the most trivial
microbenchmarks in seven or eight languages. How about throwing in some of the
fast scripting languages for healthy competition? E.g. JavaScript, LuaJIT and
Julia. Feel free to borrow (MIT licensed, but I'm willing to relicense
whatever I can):

[https://github.com/JuliaLang/julia/tree/master/test/perf/mic...](https://github.com/JuliaLang/julia/tree/master/test/perf/micro)

~~~
munificent
I'm certainly impressed by those implementations, but I don't think I'm
_competing_ with them. :)

I had JS (using node) in there for a while, but it was regularly 10x compared
to the other languages. Using a JIT makes it effectively a different class of
implementation. It's like entering a car in a bike race.

Thanks for the benchmarks! I might steal some of those. They may be more
arithmetic focused than I need (which makes sense for Julia, but less sense
for an embedding scripting language where most math is probably done in the
host app).

~~~
StefanKarpinski
It was a turn of phrase, but really I just meant that in my experience it's
healthy to have something to aspire to – that's why we always compare to fully
optimized C and Fortran. It's easy to fall into an "interpreter bubble" and
forget that there's a whole other realm of performance out there. And of
course, interpreters _can_ be quite fast – CPython is no slouch and Wren is
giving it a run for its money, so kudos. Femtolisp [1] (which we use for
Julia's parser) is a great example of a small and simple, yet fast and
featureful interpreter. I really want Jeff to write a little book about it,
walking the reader through the implementation.

[1]
[https://github.com/JeffBezanson/femtolisp](https://github.com/JeffBezanson/femtolisp)

~~~
munificent
> I just meant that in my experience it's healthy to have something to aspire
> to

I had actually thought that's what I was doing when I compared it to Lua! I
was quite surprised at how well Wren did in comparison. The trick will be to
see if I can keep it that fast as the implementation gets more robust. It
takes some shortcuts right now that don't scale well, but happen to be faster.

> I really want Jeff to write a little book about it, walking the reader
> through the implementation.

I would love that. I've skimmed femtolisp a few times and it's clean but
suffers from the almost total lack of documentation that other programming
language implementations also seem to have.

~~~
StefanKarpinski
I don't think that the standard Lua implementation is especially fast – it's
the LuaJIT implementation that's so impressively fast.

Programming language implementors do seem to be even less fond of writing
comments than most programmers, don't they? I think one of the issues is that
internals can change a lot fairly quickly and dealing with comments just makes
that worse.

------
austinpray
>Hardware: Mid-2007 MacBook Pro

Wouldn't it make more sense to do this on generic server hardware with a LTS
Linux OS? Once that vintage and customized MacBook bites the dust the website
pretty much dies as well.

~~~
Tloewald
There's nothing magical about a Linux server. If you're arguing an LTS edition
will run on newer hardware, well sure but the benchmarks will be just as
incomparable.

~~~
awinder
You're right that theres nothing magical, but there is a clear advantage to
faster disks, a more modern & faster CPU, newer & faster ram (this is running
DDR2 in the benchmark), etc. I'd be curious why this particular machine was
used for benchmarking, and if the same benchmarking method yielded different
results on EC2 or something like that.

~~~
rschmitty
You should check out
[http://www.techempower.com/benchmarks/](http://www.techempower.com/benchmarks/)
where they test on hardware and in the cloud

------
noelwelsh
If you're going to call it "Is Ruby Fast Yet", you should really be comparing
Ruby implementations to implementations of other programming languages. Asked
this way, I believe the answer to the question is a definite "no".

As a small example, I benchmarked a real Scala web app running on worse
hardware (a 1GB Linux VM) at over 650 requests/s. There was no degradation in
load at this rate of requests, but it was 10x the load we saw in production so
I stopped there. The benchmark of Rails here goes to ~120 requests/s.

~~~
warmwaffles
Well there is a JRuby and Rubinius comparison. Those are different
implementations of the the interpreter. Ruby will never be faster than C, but
this site is aiming to show the improvements over time.

------
sams99
FYI we are also working on a similar project

see: [http://samsaffron.com/archive/2013/12/11/call-to-action-
long...](http://samsaffron.com/archive/2013/12/11/call-to-action-long-running-
ruby-benchmark) and [http://community.miniprofiler.com/t/is-ruby-fast-
yet/197](http://community.miniprofiler.com/t/is-ruby-fast-yet/197)

------
phaedryx
I was expecting Ruby benchmarks, not Rails benchmarks.

------
w0rd-driven
There is one sentence missing from each graph that would've made this a lot
easier to grok:

"x is better" where x is more requests, faster startup time, or less memory
usage.

I prefer graphs that lead me to what I should expect and that sentence would
go a long way to eliminate the "mental math" I had to do to understand each
one.

On another note, the 2 year graphs are particularly interesting. They're more
consistent than not but those inconsistencies seem a little awkward.

~~~
brianhempel
The variance is sometimes due to RVM troubles. The benchmark is partly a test
of the entire Ruby ecosystem. If RVM is bugging, then people are installing
sub-optimal implementations. For example:
[https://github.com/wayneeseguin/rvm/issues/1418](https://github.com/wayneeseguin/rvm/issues/1418)

------
yuvipanda
5s startup time for a web framework seems rather high. I wonder how well it
compares to similar frameworks in other languages (Django? ASP.NET MVC?)

~~~
x1024
Eh... no it doesn't.

~~~
briantakita
It is when your development process involves rapidly iterating the code by
running the process many times (i.e. TDD). Anecdotally, when the feedback loop
is tighter, I'm much happier. Developing (especially refactoring) in Rails
feels like watching paint dry, since it's soooooo slooooow. Every time the
process needs to restart, I'm left
hanging........waiting......for..............the..............
..........process.......................to....................
...................................................s..........
................t.............................a...............
..............r...............................................
..tttt...........

Also the benchmark is the best case. In real world projects, I've seen Rails
take over 60 seconds to start (it took a few days of development effort to
bring it down to ~20 seconds). Every time you rerun tests, run a rake task,
start or restart your server, etc. It's like the heartbeat of the development
process grinds to a halt when you add more features to your Rails server.

And no, tools like Spork are not the answer, since you still have to balance
the amount that is resident in memory to fork with coverage of what you can
change. It also opens up a fractal of complexity.

~~~
tawm
Oh god. Please edit your comment so it doesn't break the whole layout.

~~~
briantakita
No can do. It got downvoted and HN does not allow me to edit negative karma
posts. So please upvote my previous post so I can fix the layout of these
comments :-)

I'm assuming the negative votes are due to the formatting and not the message.
I hope you, dear reader, don't miss the irony of this situation...

If it's downvotes because of the message, well I feel bad for the downvoters
and wish them much luck with using their framework.

~~~
Udo
You missed the edit window. If you don't come back and edit your post within
one hour, it's locked. Negative karma has nothing to do with it. That's the
risk of fire-and-forget posting.

~~~
briantakita
Or it's the risk of failing to account for human behavior...

It's not exactly a moral imperative to obsess over your HN comments, IMO.

Most likely, it's a simple bug, that seems to be fixed now :-) No ethics
involved. Nothing to see here.

Btw, in the OP, I was trying to express the effect that the increased startup
time affects flow and iteration. It's a similar effect to a comment breaking
the layout of the page.

I just learned something new today :-)

------
spoiler
We can at least see MRI has gotten faster with each new version, but we knew
that already.

Also, I'm a bit surprised that this doesn't include more implementations of
Ruby. Where's Rubinius? IronRuby? REE? I hope the list gets updated.

Furthermore, the benchmark itself could be better; I won't say I am
disappointed, but I am not too thrilled either.

------
lowglow
[serious]

I'm building [http://techendo.co/](http://techendo.co/) and it's running rails
on heroku. I was wondering if there was some easy ways to get some better
performance out of it immediately now that we're picking up traffic. If anyone
could give me some pointers or link me to a page I'd appreciate it.

~~~
awj
That's an incredibly broad question, so it's pretty hard to answer. Here's
some general ones:

* More instances will give you more requests/second, but only so long as secondary system (database, search server, whatever) can keep up with the load

* N+1 queries will kill your performance. They can add a lot of time to processing requests.

* Everything in the most used parts of your site should hit an index on the database. If it's not I can almost guarantee it will be slow.

* Look for work that you can move "offline". As long as they go out within a minute nobody will notice that you aren't sending emails at the exact time they are being generated. Move that work to a background queue so your web server can just add a queue entry and get on to the next request.

* Cache anything you reasonably can. It's a lot cheaper (i.e. faster) to read from a cache than to do a database lookup. It's cheaper to read from a cache than to render an ERB template. Caching can introduce its own difficulties, but the performance gains can be worth it.

Without knowing some specifics of your implementation I really can't give you
better advice. Depending on the specific libraries you use it might not even
be possible to move to JRuby due to support for C extensions. You might get
some performance benefits out of it, but I'd expect it to only buy you some
time to execute on some of the above improvements.

~~~
pooriaazimi
Tools like [https://github.com/MiniProfiler/rack-mini-
profiler](https://github.com/MiniProfiler/rack-mini-profiler) really help with
the N+1, and general insight into how long database calls usually take.

Relevant RailsCast:
[http://railscasts.com/episodes/368-miniprofiler?view=asciica...](http://railscasts.com/episodes/368-miniprofiler?view=asciicast)

------
polysics
As a non-Rails Rubyist, I will have to run benchmarks myself as usual, because
people seem to be still equating Ruby to Rails.

------
gesman
Compare to what?

And the answer is no, and unlikely to be any time soon:

[http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=query)

------
astrodust
There's great data here that's obscured by choosing terrible colours for the
graph and showing things that are pretty much irrelevant. A summary chart
giving relevant percentage differences would be a lot better than this line
chart.

------
davidw
I don't like those charts at all. I want to see the progressive Ruby versions
on the X axis, I think. The dates are pretty meaningless. If it's the same
code, it should run in the same number of cycles; anything else is just noise.

That said, +1 for having a go at it!

Also, yes, Ruby is pretty slow. But it's fast for getting something up and
running quickly. If you get wildly successful, you can always swap it out for
something else if you need to.

------
saalweachter
If I understand this correctly, this is running periodically on someone's
Macbook; people are complaining that's a terrible platform for a benchmark.

It should be possible (if you care enough) to re-run this from the source
control history on platforms of your choosing, across whatever date ranges you
wish. Anyone really, really care enough?

------
mpweiher
tl;dr: no.

~~~
michaelhoffman
I was expecting the web page to just consist of "No."

~~~
mpweiher
Yep, that was my expectation as well :-)

------
lazugod
Are web fonts fast yet?

------
shocks
Yes, if we're talking about development time! :P

------
kbar13
benchmarking a web framework on an old macbook pro?

------
wadus1
this is incomplete and then bullshit, it's comparing a single-core with a
multicore running a single-threaded process.

------
bernardom
I was expecting a one-time-use site such as
[http://www.whatwouldmyharvardgrade.be/](http://www.whatwouldmyharvardgrade.be/)

------
fantnn
Was expecting one of those generic yes/no sites, a la
[http://istherejava0day.com/](http://istherejava0day.com/)

