
Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0 - resca79
https://gist.github.com/acapilleri/6898533
======
cpuguy83
This is in no way a good test. Yes, Rubinius takes longer to load than MRI.
It'a also JITed, so it needs time to warm up to start screaming.

Also, this is in dev mode... did you disable Rack::Lock?

MRI can only handle 1 request at a time, Rubinius can handle whatever your
concurrency level is (assuming you've removed Rack::Lock from your
environment)

~~~
resca79
Rails 4 is thread safe and I used puma as server Also the app is in production
mode and the db pool is 16

~~~
fbernier
Rails 4.0.0 application starting in development on
[http://0.0.0.0:3000](http://0.0.0.0:3000)

This most likely means Rack::Lock is used in your test.

~~~
resca79
I used the following command rails s puma -e production

------
jacques_chester
Reliable benchmarking is hard. There are actual _journals_ about measuring
computer system performance.

Some questions:

* How is startup time controlled for?

* Why only one run?

* Why so few requests?

* Why no warmup period?

* Why this particular configuration?

* What is the OS, hardware configuration?

* Why is it running in a development environment instead of a test or prod environment?

~~~
ebonfortress
I would really like to see some articles about proper benchmarking procedures
at 1000+ requests per second. Are there any?

~~~
jacques_chester
The ACM's SIGMETRICS[1] are the main group publishing this kind of research.
Some of it gets very specialised -- just picking a title from the most recent
_Performance Evaluation Review_ [2], I get "YOUQMON: a system for on-line
monitoring of YouTube QoE in operational 3G networks".

However, being the ACM, it's basically inaccessible unless you sell a kidney.

An organisation with a slightly more industrial focus is the Computer
Measurement Group[3]. Like the ACM ... bring money.

[1] [http://www.sigmetrics.org/](http://www.sigmetrics.org/)

[2]
[http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFT...](http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFTOKEN=88550832)

[3] [http://www.cmg.org/](http://www.cmg.org/)

~~~
look_lookatme
Sorry to go off-topic, but does anyone know if it's possible to get your hands
on these pubs/journals for free at the Science, Industry and Business Library
[1] in NYC?

[1] [http://www.nypl.org/locations/sibl](http://www.nypl.org/locations/sibl)

------
warmwaffles
I would also love to see an application in production measured with NewRelic
or something else and compare the results. A good way to test this would be to
spin up three servers.

* 1 load balancer that plays round robin on the requests.

* 1 server running MRI

* 1 server running Rubinius

Each with NewRelic installed and compare the A/B test that is taking place for
a period of about a week of heavy usage. Obviously the gems that the rubinius
application would have would be different than what the MRI gems would be. But
the core of the application would remain the same.

This would give me and others better insight as to how the two interpreters
are running in a production.

------
bhauer
Although you may be running a single-threaded test intentionally, I generally
recommend using a multi-threaded load simulation tool versus Apache's ab tool
[1]. My preference is wrk [2], especially since it provides for time-limited
tests rather than request-limited. Another alternative is WeigHTTP [3], which
is basically a multi-threaded replacement for ab.

[1]
[https://news.ycombinator.com/item?id=6211883](https://news.ycombinator.com/item?id=6211883)

[2] [https://github.com/wg/wrk](https://github.com/wg/wrk)

[3]
[https://github.com/lighttpd/weighttp](https://github.com/lighttpd/weighttp)

------
JimmaDaRustla
So, MRI is way faster than Rubinius in this case? Interesting...

~~~
mrinterweb
Not necessarily. No warmup time was allowed for rubinius's cache. Also we
don't know how concurrency was being handled. Rubinius should be able to
utilize multiple cores. More information about the hardware, rails app, way
that the puma server is being started, warmup time, duration of request time
spent on the database, etc should be made available before any remote
conclusion like which one is faster can be made.

~~~
JimmaDaRustla
That was my concern - Rubunius' selling point is concurrency, so I'm sure
there is an approach to implementation, which is why I've never used it.

------
ksec
Gist title is Ruby 2.0 MRI vs Rubinius 2.0 While HN Title is Ruby 2.1

So which is correct?

~~~
resca79
sorry.. I have updated the test with ruby-head Ruby 2.1 MRI 2.1 Rubinius 2.0

------
harlock81
very interesting, thanks!

