

Ruby Benchmark - rbxbx
http://rubybenchmark.com/

======
Groxx
A handy list, I'll have to read through it more completely some time. There
are some surprising speed differences. For instance, array.drop is hella-
faster than array.clear, and array.shift(n) is faster yet (what?).

Nitpick on their array tests, however:

    
    
      measure "Array#delete_if" do
        a = ARRAY.dup
        a.delete_if {|o| o}
      end
    

Ah, _no_. If o == false, that won't delete it. Also, you're calling object ==
true; how about a.delete_if {true}?

    
    
      "Array#delete_if{true}" is up to 30% faster over 10000 repetitions
      ------------------------------------------------------------------
    
        Array#delete_if{true}     1.42472314834595 secs    Fastest
        Array#delete_if{|o| o}    2.03995013237    secs    30% Slower
    

Also:

> _gsubbing with string literals vs regex. For some reason regexp is faster. I
> suspect it converts string literals to a regexp anyway._

Really? You're surprised regex literals are faster than regex strings?

The gem however is interesting and nice and simple, I may end up using it for
my future testing. Haven't looked through the code, however: anyone know if
it's making any mistakes?

~~~
wzdd
I'm not a Ruby programmer, so I'm interested to hear your view on this: as a
non-Rubyist reading Ruby code, the "assign variable to a new array" method is
much more readable to me than than the .shift(n) method and is only 2% slower.
Is shift(length-of-array) an idiomatic way to clear an array in Ruby? From my
perspective the site looks like a great way to encourage confusing code for
little gain.

~~~
Groxx
No, it's just a benchmark testing many ways to clear an array; there's nothing
idiomatic about shift(n) that I'm aware of. Benchmarking code should always be
taken with a grain of salt, it can get pretty needlessly esoteric at times in
the pursuit of speed.

I _would_ be interested in if anyone knows how the garbage collector handles
a.drop vs a = []. Maybe a.drop collects earlier, and the gc is causing the
slowdown?

~~~
cheald
GC should be disabled for proper benchmarks anyhow, honestly. Or at least
measured. REE has handy switches to measure that sort of thing.

~~~
rue
Disabling the GC means your benchmark is worthless.

1\. Valid GC during execution: you are removing an actual factor;

2\. "Invalid" i.e. unrelated GC during execution: your benchmark is too narrow
in scope.

~~~
cheald
It depends on what you're benchmarking. If you're benching wall time, then
yes, you need to have GC on, and you need to do a ridiculous number of
iterations to average out GC runtimes. You also need to invoke the GC before
you begin your benchmark; doing a collection pass early in the benchmark for
all the objects created on startup/setup skews the benchmark.

Separate benchmarks should be conducted for execution speed (CPU time) and
garbage generation. Saying "X is faster than Y" tends to suggest a CPU time
benchmark. "X is faster than Y with W delta garbage and Z delta less time
spent in GC" suggests a proper wall time benchmark.

------
jackowayed
This is awesome. It would be really cool if they added comments so that people
in the know could explain things (and possibly dispute the outcomes)

~~~
sandrot
Thanks. Comments would be cool, I'll give it a shot.

------
zackadams
Nice work Sandro.

Andy Stone has a benchmark file he's been maintaining that has some similarity
to your project. Check it out if your looking for inspiration on additional
tests:
[http://github.com/stonean/benchmark/blob/master/benchmarks.r...](http://github.com/stonean/benchmark/blob/master/benchmarks.rb)

~~~
sandrot
Great resource. Looks like I've got a lot of porting to do =). Thanks.

------
viraptor
It doesn't seem to throw out / properly mark insignificant results:
<http://rubybenchmark.com/reports/10> (is there any reason these 2 variants
could get different bytecode results?)

That's the way echo -vs- print wars start...

~~~
sandrot
Looks like there can be a more significant difference after 100,000
repetitions: <http://rubybenchmark.com/reports/13>. In general, the program is
pretty dumb and my math isn't that great. Accepting patches:
<http://github.com/sandro/bench_press>

------
pieter
> "while" is up to 92% faster over 100000 repetitions
    
    
        Array#each_with_index    1.3768820762634277  secs    92% Slower
        while                    0.10703921318054199 secs    Fastest
    

Don't you mean it's around 13x fast (or 1200% faster)?

~~~
Chirono
This is always rather ambiguous terminology. For some people, it makes no
sense for thing a to be more than 100% faster than thing b. Other people see
it clearly meaning that thing a take half the time of thing b.

What we really want to measure is time quantities rather than speed
quantities. I find comparisons of time much easier to grasp than comparisons
of speed.

------
Pistos2
Although it usually only matters when compared code scores similarly, it is
important to be able to know when benchmark comparisons are _statistically
significant_ [1]. I wrote a library[2] a while back to do this. This becomes
even more of a concern when the benchmarked code is running on a machine which
is not well-isolated and dedicated to running the benchmark.

[1]: <http://en.wikipedia.org/wiki/Statistical_significance>

[2]: <http://github.com/Pistos/better-benchmark>

------
jacquesm
Funny, just finished installing rails 3 and the latest ruby. What a headache
that was, really. In spite of following a tutorial I ran in to a whole series
of showstoppers and only just now finally got it to work.

I've never played with ruby/rails before so I figured I'd better go and see
what all the noise is all about but I had definitely not counted on so many
installation issues.

I should probably clean this machine and do it all over again to be able to
write up the exact steps to get it to work (from not having ruby before) but
I'm too scared of having to go through another day of fiddling.

~~~
rbxbx
I'm not sure the relevance to the OG post, but I'll grant you that Rails 3 is
less than stable, thusly it's tagged RC2, ie: not a release proper.

~~~
jacquesm
You're right, it's not relevant, I just thought it was coincidence to see a
rails post at the top of HN while I'm taking a break from my troubles.

It looks like there is light at the end of the tunnel though.

~~~
rbxbx
If you're entirely new to the ecosystem, I would not be running Ruby 1.9.x or
Rails 3. It may seem counter-productive to learn 'legacy' software, but these
version will be around for a while I'm sure, you'll not hit nearly the snags
you currently are, and you won't have /that/ much to learn when the later
version are ready for you (and you them).

also, ruby != rails ;)

~~~
derefr
If you're quite familiar with Ruby/Rails, but you left the ecosystem for a
while (before 1.9 existed), would it be better to re-acquaint yourself with
1.8/Rails 2 first, or just dive into the new stuff?

~~~
Pistos2
[Setting Rails aside, ]the Ruby community needs help to move forward into the
world of 1.9. 1.9 has many improvements that most people would benefit from,
but many folks are either afraid to get their feet wet with 1.9, or have to
support legacy applications. If anyone would be writing anything new in Ruby,
I encourage them to write it for 1.9 and to leave 1.8 behind. For the past
year or so, only about 5% of my stuff has been written to support 1.8. The
rest has been written for 1.9, usually with syntax or feature usage that will
not work under 1.8.

When developing new Ruby stuff, one thing to watch for is interoperability
with old gems that are not 1.9-ready. But the good news is, most gems work
just fine under 1.9 -- at least in the non-Rails world. I think I recall only
one or two gems I tried to use that weren't updated to work with 1.9 yet, out
of many that I use or discovered.

------
AndrewO
That's a great idea. I'd really like to see a "raw" format option on the
server that would give the code and metadata in a structured format.

With that one could write a tool that suggests improvements to code as it
tests it using this as a sort of crowd-sourced benchmark suite. It may not be
100% accurate, but at least it would give verified data instead of the
educated-guess strategy most of us employ now.

------
ulvund
It might be fun to add some Date tests as well. The methods are very slow to
say the least. There is a 'date/performance' gem I think.

~~~
sandrot
Cool, I'll keep that in mind. The site is intended to have user-generated
content so if you find the exact thing you'd like to benchmark, use the
bench_press gem to test locally, then publish it to share your findings.

------
rue
I am, uh, highly sceptical of the value of this. That is, even ignoring the
variance between runs of the same code, differences between implementations
and within an implementation with slightly different objects.

Writing code based on "what is fastest" is an overall awful strategy.

------
justinchen
How about an RSS feed for the recent benchmarks?

~~~
sandrot
Top of my list!

------
xsltuser2010
The site is awfully slow at the moment. Is it written in Ruby ? That would
make the benchmark even more convincing ;)

~~~
rbxbx
Looks to have been fixed(?), perhaps the 2 day old project wasn't ready for
the HN frontpage.

