There is a neural net example benchmark in the topaz git repo. Don't know how representative that example is, but at least
startup time shouldn't be dominating the results...
$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.3.0]
$ ruby bench_neural_net.rb
ruby bench_neural_net.rb 17,74s user 0,02s system 99% cpu 17,771 total
$ bin/topaz bench_neural_net.rb
bin/topaz bench_neural_net.rb 3,43s user 0,03s system 99% cpu 3,466 total
| + x |
| + x |
| + + x x x |
| + ++ +++ + x xxx xx x |
|++ ++++++++++++++++++ x x x xxx xxx xxxxx |
|++ ++++++++++++++++++ +++ + ++ xxxxxxxxxx xxxxxxxxxxxxxxxx xx x xx|
| |______MA______| |________A________| |
N Min Max Median Avg Stddev
x 57 45.62364 46.437353 45.93506 45.951554 0.19060973
+ 57 44.785579 45.534727 45.042576 45.056702 0.16634531
Difference at 95.0% confidence
-0.894852 +/- 0.0656777
-1.94738% +/- 0.142928%
(Student's t, pooled s = 0.178889)
Whenever I see frontpages for these kinds of projects like "a faster X" or "X written in Blub", the first thing I want to see on the frontpage is how this new project compares to X in terms of quality and performance. Even specious benchmarks would help more than zero benchmarks.
I wish more frontpages for these kinds of projects would do that.
To clarify, I was trying to use "specious" as a synonym of "flawed." I thought this was the common usage, but apparently not.
As to your point, obviously no one should be making any decisions off of flawed benchmarks, but flawed benchmarks (not so far as outright lies, just flawed) at least give me an objective justification to investigate further.
Even some flawed benchmarks could help turn the initial tide of responses like "this is X written in Blub, it's bound to be better!" or "this is a faster X! Now everything will be twice as fast!" They're silly examples, but it seems like every time a new technology comes out, these are the kinds of knee-jerk, overly-optimistic reactions people tend to have.
1 obsolete : showy
2: having deceptive attraction or allure
3: having a false look of truth or genuineness : sophistic <specious reasoning>
So instead of taking the common meaning of specious, a deceptively attractive benchmark we have to go to a less common use of specious in order to construct an specious argument about the proper use of specious? Nevermind that it is distracting from the main point of the discussion about benchmarks in the context of Topaz.
It depends where you look for the common meaning. To native English speakers in the UK, Australia, New Zealand (ie outside North America, coincidentally including the country where the English language developed) specious has one very clear meaning
I would also direct you to the usage examples in your link. All of which use specious in a context that implies deception of outright falsity. I have never ever seen specious used synonymously with obsolescence.
I disagree with your argument! If inventing the language meant you got to choose all the definitions we wouldn't have English in the first place. Shift happens. I don't think this particular usage is common, though.
I was using "specious" as a synonym of "flawed." Perhaps that was the incorrect usage.
But in the case of advertising a new library/project, which is arguably one of the main functions of the frontpage, flawed benchmarks (though not so far as outright lies) at least give me an objective reason to investigate further.
With no benchmarks, generally I'll open the page, mutter "that's nice" and move on with my business. I'd imagine I'm far from the only person who does that. Young projects don't help themselves when they don't effectively advertise themselves.
the idea was to implement enough of "hard stuff". So it should not change. but hello world is not a good idea (this one also might actually change due to library loading, but please don't benchmark it like that)
Perl 5 had an in-memory footprint of about 768k. VisualWorks Smalltalk had a standard one of 12MB, yet loaded and started it faster than Perl did with 768k. If just your standard library takes very long at all to load, something needs attention.
$ type /bin/echo
/bin/echo is /bin/echo
$ time /bin/echo "hello world"
$ type echo
echo is a shell builtin
$ time echo "hello world"
If the goal is to benchmark Ruby execution time, it's unfair. The slow startup time is a valid concern when considering short-lived sessions, but it's kind of meaningless when trying to benchmark a Ruby implementation.
Agreed. But then you're not benchmarking Ruby execution speed, which really makes comparing the two not very worthwhile. By the same notion, then when you compare on something like JRuby, you really should be comparing JDK 6, 7, and 8 builds, along with various startup flags, both JVM and JRuby. E.g., short runs may benefit greatly from turning off JRuby JIT, JAR verification, and using tiered compilation modes. May as well add Nailgun and drip in there as well, since both can speed up startup time on subsequent runs. Which one of these is going to be the JRuby you use in your comparisons? Or are you really going to show 20 different configurations?
You can run into the same problem with MRI and its GC settings. If too low for your test, you're going to hit GC hard. It's best to normalize that out so you have an even comparison. Confounding variables and all that.
There's a lot you can do outside the Ruby container to influence startup time. When comparing two implementations, the defaults are certainly something to consider, but not when trying to see which actually executes Ruby faster. They are two different metrics of performance and should be compared in isolation.
There are no 'fair' benchmarks, all benchmarks should be biased to the problem you're actually solving running your actual workload. If you can't replay your workload at multiples of real volume then you should probably work on doing that before benchmarking as it helps you out with the real problem of verifying your infrastructure.
In general a benchmark is probably the worst metric you could ever use for deciding on an implementation. Unless the profit margin of your business is razor thin and dependant eeking out every last drop of performance, and even then most of those gains will be from extremely small sections of code that are probably best written in assembler by a programming God, and you should investigate FPGAs, ASICs, and other high performance solutions.
If your benchmark (infrastructure) involves a database (or anything that uses disks) that's probably going to be the problem long before the speed of your language / language implementation.
In all comparisons, you should remove confounding variables. Yes, you should benchmark something you actually care about, otherwise what's the point? That doesn't mean all other variables are immediately null and void. That's why I said said if your goal is to measure ruby execution time, you should remove startup time.
As for the practice of benchmarking in general, you're partially right. Micro-benchmarks are usually useless because they don't map to real work load. But profiling and speeding up small portions that are used heavily can have drastic improvements that in isolation seem small -- the death by a thousand cuts problem. Not all improvements come from isolated instances with very slow performance profiles.
This fallacy about DB access and not needing to optimize really needs to go away though. Even if 50% of your app is spent hitting DB, you have opportunity to speed up the other 50% and it's likely far easier. Ruby in particular is ripe for improvements on the CPU side. I managed to reduce my entire test suite time by 30% by speeding up psych. I managed to cut the number of servers I need in EC2 in half by switching from MRI, Pasenger, and resque to JRuby, TorqueBox, and Sidekiq. And I've managed to speed up my page rendering time anywhere from 8 - 40x by switching from haml to slim. None of these changes required modifications to my DB, none required me to write assembly, none required me to switch to custom-built hardware, and each helped reduce the expenses for my bootstrapped startup, while improving the overall experience for my customers.
I'm not trying to be snarky here, but that would advocate for no improvements anywhere else. Why did Ruby 1.9 bother with a new VM? Why try to improve GC? Why bother with invokedynamic? Why speed up JSON parsing? Why bother with speeding up YAML? Yet there's obviously value in improving all these areas and they speed up almost every Ruby app.
It's overly simplistic to say the only option is to cache everything. Or that your DB is going to be your ultimate bottleneck, so the other N - 1 items are worth investigating.
And even in the link you supplied, the illustrative example is getting a 20 hour process down to 1 hour without speeding up the single task that takes 1 hour. It suggests there's an upper limit, not that because there is an upper limit you can't possibly do better than the status quo.
" that would advocate for no improvements anywhere else"
Amdahl's law advocates starting from the part that takes the most time. In a database application, it can be interpreted as either A) improving the connector or B) reducing the application's demand for database resources.
"Why did Ruby 1.9 bother with a new VM? Why try to improve GC? Why bother with invokedynamic? Why speed up JSON parsing? Why bother with speeding up YAML? Yet there's obviously value in improving all these areas and they speed up almost every Ruby app."
JSON parsing improves those applications that use JSON parsing, and in many applications JSON parsing is the main operation. There are many other applications for which garbage collection is the limiting factor. You are taking my comment, which was addressing the parent comment's remark that "This fallacy about DB access and not needing to optimize really needs to go away though.", way out of context. It's not a fallacy -- you need to know what is dominating execution time and how to improve that aspect.
Take it to the logical extreme -- you could just write in x86 assembly directly. The program would be faster than ruby, but the development time would not make assembly a worthwhile target.
"And even in the link you supplied"
What link did I supply? I recommend the Hennessy and Patterson "Computer Architecture" book :)
Sorry about the link comment. I'm so used to Wikipedia links being passed around I must have instinctively looked there.
In any event, we probably agree on more than we disagree. I never disagreed with working on the DB if that's truly the bulk of your cost. But, you do actually need to measure that. It seems quite common nowadays to say "if you use a DB, that's where your cost is". And I routinely see this as an argument to justify practices that are almost certainly going to cause performance issues.
Put another way, I routinely see the argument put forth that the DB access is going to be the slowest part, so there's little need to reduce the other hotspots because you're just going to hit that wall anyway. And then the next logical argument is all you need is caching. The number of Rubyists I've encountered that know how to profile an app or have ever done so is alarmingly small. Which is fine, but you can't really argue about performance otherwise.
But if it's not executing ruby, then it can't be ruby execution time... That's the point I'm making. By all means, if you want to measure start-up time, that's valid as well, just not for "how fast does this execute Ruby."
An EC2 server doesn't net me anything in revenue. It costs us money and falls into the category of expenses. Reducing expenses makes us more profitable. Reducing our expenses by 50% made us roughly 50% more profitable.
Better than that, this savings isn't one-time. It's recurring, as EC2 is recurring. But we've also reduced the expense growth curve (the savings wasn't linear), so we can continue to add customers for cheaper.
The A/B testing thing is a complete non sequitur. a) there's no reason you can't do both. b) most A/B testing yields modest improvements.
I find it interesting that reducing your EC2 usage by 50% decreased your expenses by 50%, it means one of two things, you aren't paying your employees and don't have any overhead, or the cost of EC2 dwarfs the cost of your employees and overhead.
If it's the latter, I'd seriously consider colo as you can probably reduce costs by another 80%.
Obviously the discussion was scoped around non-personnel expenses. I'm not going to dump an entire P&L here. And this is now wildly off-tangent.
I was illustrating that there is real world gain to be had by doing something as simple as switching to a new Ruby or spending some time with a profiler. These weren't drastic code rewrites. They didn't require layers of caching or sharding of my database. I fail to see what's even contentious about this.
As long as it can reasonably expected to be mostly bug free and support everything you need it to with little changes to the app. It wouldn't need to require to much time playing around with it before the EC2 savings are eaten up by the wage costs of spending time on it. (Depending on how many servers you are running of course)
This isn't a theoretical argument. I actually did this and it didn't take all that long and some of it was even fun. An added benefit is my specs run faster, too. So developer time is saved on every spec run now. You also hit that intangible of improved developer happiness.
Additionally, the X time exceeds Y cost argument really only works when people are optimally efficient. Clearly those of us posting HN comments have holes in our schedules that might be able to be filled with something else.
In production server apps the JVM always running anyway, and under some degree of Hotspot optimization, so for a JRuby benchmark to be informative and worth anything to you, you'll want to account for that.
Topaz doesn't run rails yet (as far as I know, I didn't even dare to try!), so I doubt you'll find any benchmarks ;) There is one benchmark in the bench/ directory of the repository you can try though!
$ time ruby -e "puts 'hello world'"
The program 'ruby' can be found in the following packages:
Ask your administrator to install one of them
I'm all for new Ruby implementations, but be very careful about judging performance of an incomplete implementation.
It's "trivial" to make a fast language that looks quite a bit like Ruby. It is a lot harder that make a language that remains fast in the face of handling all the quirks of the full Ruby semantics, though, such as selectively handling the risk of someone going bananas with monkey-patching core classes that could happen at any "eval()" point.
What you just wrote is exactly what I've been telling people about every new "Python" implementation for the last 3 years. Believe me, I understand this argument completely, that's why I made sure we had all the hard bits (monkey patching core classes, eval, etc.) before I released this.
1) Note that the post doesn't sell its speed; all it says is that they are interested in a high-performance Ruby
2) Python is a very similar language to Ruby, and Pypy already runs python very rapidly. This doesn't guarantee that the Ruby interpreter will be anywhere near as fast, of course, but it does give evidence that it's possible.
3) As kingkilr notes below, they've taken into account your argument and they believe they've implemented enough of the language to be confident that they can run it rapidly. No reason you need to believe him, but it's worth listening to.
I will be the first to ask: why did Alex / the pypy devs start this project? I thought the three community funded ideas were great (STM, Python 3, and Numpy). What prompted this new project? Is it meant to attract new developers to pypy as platform for implementing languages? Is it a side project? Is it a long term project meant to become the premium implementation of ruby as pypy is becoming for python? Etc.
Not criticizing at all by the way, just curious about the motivation / background / context for the project, which is missing from the docs.
Hey Alex, let me tell you I completely understand points a) b) c) and more. It seems we started our competing Ruby on PyPy implementations at about the same time ! My goal was to build a robust and documented Ruby parser, and hope PyPy make it magically fast. To that end I embedded PLY and modified it to not require kwargs. Then a thunder of work fell from the skies, and I was just recently unearthing PyPy 1.9 to put the thing back on track.
Congratulations for the release :-)
$ ls -l playground/pypy
drwxr-xr-x 12 lloeki staff 408 18 Feb 2011 ply-3.4
drwxr-xr-x 15 lloeki staff 510 3 Apr 2012 pyby
drwxr-xr-x 23 lloeki staff 782 27 Mar 2012 pypy-pypy-2346207d9946
drwxr-xr-x 18 lloeki staff 612 27 Mar 2012 pypy-tutorial
-rw-r--r-- 1 lloeki staff 14927806 27 Mar 2012 release-1.8.tar.bz2
drwxr-xr-x 10 lloeki staff 340 3 Apr 2012 unholy
It would be nice to see Rpython/pypy turned into a project not unlike WebKit where many different projects benefit from the same open-source core. If pypy is flexible enough to implement all of ruby and all of Python, has a reasonable amount of maturity, and is already rather fast it seems like a good candidate for such a thing.
2. get and extract the PyPy source from bitbucket release packages, which includes the RPython translator, written itself in Python and, although working on CPython, best run under PyPy as installed on step 1.
3. write a (R)Python module including a def target returning the entry point function, and call the translator upon your module, like so:
I wish getting the translator was more 'packaged' and did not involve getting the whole PyPy source but I hear this is in the works, and it is reasonably easy already. The hardest part is actually figuring the translator is not part of PyPy binary release, and that it's available straight from bitbucket.
Forgive me for asking what may be a very stupid question: Can someone explain to me the need for a Ruby interpreter built on top of Python? He mentions performance in the announcement, but is this really going to be faster than, say, MRI? Thanks in advance.
PyPy is totally not like LLVM in many regards. It targets a very different demographics. Some differences:
* RPython comes with a good garbage collector
* the language where you specify what's going on is RPython in which you write an interpreter. Then you get a JIT using a few hints. This is difference than "interpreter in C + compiler to LLVM" scenario by quite a bit.
* RPython comes with a set of data structures that are higher level (lists, dicts, etc.) and is a GCed language. JIT is also well aware of those.
I'm actually not a native speaker, so indeed, I might not know. I checked in the dictionary though. This is an analogy, but not a very good analogy, because of the reasons that I pointed out. Are you disagreeing with any particular reasoning there? Or should I just say "they're similary, but"?
An analogy does not require that the items involved actually be similar, only that the relationships in each are the same. For example, "Gasoline is to cars, as sunlight is to trees" is a perfectly good analogy, even though gasoline is nothing like sunlight, and cars are nothing like trees.
In much the same way, comparing "Apple writing a C interpreter on top of llvm" to "Alex Gaynor writing a Ruby interpreter on top of RPython" makes a very good analogy, even though Apple is not much like Alex Gaynor, Ruby is not much like C, and PyPy is not much like LLVM.
A good explanation, and further, many times a good analogy relies on the things being very different in every way except for the one way it intends to emphasize. An analogy is often a way of saying: "Let's focus on this (often very abstract) aspect of interesting similarities. Yes, you can easily point out many other dissimilarities. That ease means I'm not talking about those aspects, so if we understand each other those don't even have to be mentioned."
You're right that the analogy doesn't quite capture everything here. On the other hand the analogy gets across most of the point.
I would have said "That's mostly right, but the interesting thing about sharing code on a VM is access to VM features and code shared on the platform, not just that it's possible to decouple a front end from a VM."
Thanks for that simplified explanation. As someone not familiar with Python and Pypy I was having a tough time getting past the "Ruby on Python is efficient?" block; the analogy to LLVM is helpful indeed.
So, lets talk about JRuby. JRuby is a Ruby implementation on the JVM, and its really awesome power is actually that you can wrap and interface with Java libraries, and still write all of your app code in Ruby.
If Topaz was able to give me the ability to access NLTK, but still write in Ruby? I'd be overjoyed.
> If Topaz was able to give me the ability to access NLTK
I doubt that's going to happen. Topaz is not a Ruby running on the CPython interpreter, the Topaz VM is developed in RPython. As far as I know, RPython doesn't provide that capability either (between VMs coded in RPython), which fijal seems to confirm.
So that would be a very cool project to merge python <-> ruby (or any other interpreter written in RPython). It requires quite a bit of work to pull it off the ground though. I would be happy to provide guidance on that, but I definitely won't do it myself (just yet).
One of the major launch criterion was having "enough of Ruby that the performance won't change". What does that mean in practice? We have exceptions, we have bindings, and we have all manner of other obscure corner cases. Charles Nutter (JRuby dev) writes about this: http://blog.headius.com/2012/10/so-you-want-to-optimize-ruby...
Does Topaz benefit at all from the work that's been done already on Rubinius? I seem to recall that Rubinius implements as much of Ruby in Ruby as possible. It seems like this would help Topaz as well in implementing the standard library. Or are the implementations different enough that this doesn't work?
> Some observers objected to Go's C-like block structure with braces, preferring the use of spaces for indentation, such as used in Python or Haskell [braces are optional in haskell]. However, we have had extensive experience tracking down build and test failures caused by cross-language builds where a Python snippet embedded in another language, for instance through a SWIG invocation, is subtly and invisibly broken by a change in the indentation of the surrounding code. Our position is therefore that, although spaces for indentation is nice for small programs, it doesn't scale well, and the bigger and more heterogeneous the code base, the more trouble it can cause. It is better to forgo convenience for safety and dependability, so Go has brace-bounded blocks.
Implementing other languages on top of the PyPy toolchain is an interesting concept, and I'd like to see it happen more often. But calling this "Ruby in Python" is sure to invite flamewars from all sides. The actual Topaz site seems to be careful not to do this, but if this thread is any indication, a fair number of people are already mistakenly calling it that.
At least once a year there's a maelstrom of posts about a new Ruby implementation with stellar numbers. These numbers are usually based on very early experimental code, and they are rarely accompanied by information on compatibility. And of course we love to see crazy performance numbers, so many of us eat this stuff up.
Could this lead to a runtime in which I can use really cool ruby modules from python and really cool python modules from ruby? I mean, seriously, these two languages both have some pretty awesome stuff, and regularly I say about one, "man I sure wish I could use $module from the other".
Excellent news. I wish this project the best of luck. More competition in the Ruby implementation field is always a good thing for end users, it fosters innovation and provides choice and a good kick in the ass to everyone.
I mean, it's cool that people want to expose Python's version of Rubinius as a building block to yet another Ruby interpeter, but, come on. Why is this a thing?! Why not waste time improving the JVM, contributing to Rubinius, patch the MRI, or a million other open source projects already tackling Ruby being "slow". By the way, has anyone else realized that, while Ruby is slow, computers are getting faster for us too!
No. It's Ruby in RPython, where PyPy is Python in RPython. The pypy project is currently in the process of splitting "RPython" (the VM-development framework) from PyPy (the Python VM) to make that clearer.
RPython is a general purpose, language-agnostic (ish?) core for implementing JIT-ed garbage-collected languages, it's kind-of similar to LLVM being a framework for implementing compilers.
(now because rpython is a proper subset of python you can run your rpython VM on a python interpreter without translation, but that doesn't give you any specific python bridge, in the same way coding your VM in C doesn't give you a c FFI for free)
RPython is generally language-agnostic, although dynamic langauges are much better suited for this sort of transformations. So it makes sense for any kind of dynamic language (like prolog even), but makes less sense for say Java or C.
The same, as far as I've seen. RPython is both the VM-framework and the Python subset which exists within (and for) that framework (RPython is not considered and developed as a general purpose language, but as a language for building VMs within the RPython system/framework/toolset).
The important distinction really is between RPython (a language and framework for building virtual machines) and PyPy (a virtual machine implemented in/on RPython)
Not really. RPython is a statically-typed language which is used to implement PyPy's virtual machine framework. PyPy is itself a Python VM built on top of that framework, but the framework is designed to be reusable. So Topaz is a Ruby VM using that same virtual machine framework.
It's Ruby in RPython, the implementation language of the PyPy VM toolchain. It has nothing to do with integrating with Python. It's an alternative Ruby VM using a proven JIT generator technology that happens to be built with a Python-like language. Hope that helps.
I think you're missing the point. It's not written in Python. It's written i RPython, which is vastly different. Technology behind this and PyPy is the same, however there is no Python usage here.
Additionally, erlang has avoided scaling issues by being functional. This might be good or bad, depending on your viewpoint, but just implementing a ruby interpreter in erlang won't cut, because you're not answering any of the hard questions about the state. You might answer "meh, ruby is just a broken language", but well, this has nothing to do with that article.
As RPython is a subset of Python, that statement is technically correct but misleading. Stating that Topaz is "Ruby written in Python" leads to the idea that it's written in / runs on CPython, which is really not the case.
The Erlang runtime imposes a different set of constraints than Python that would impact the design of the language being built on top of it. Given that, someone already has come as close as you can to porting Ruby to the Erlang VM, it is called Elixir. http://elixir-lang.org/
No it's not. Technically speaking yes, but without Python there would be no RPython. In other words you're betting on three language/vm communities: Ruby language, Python (RPython!), LLVM (iirc).
I don't believe introducing a lot more dependencies and complexities will cure any problem. Sure, it's a nice project and the work shows the great skills of the developer. But it's not practical to use this besides some non-crictical fun projects.
Yes, but betting on Python is the whole point of PyPy. The idea is to build a toolchain using a relatively high-level programming language like RPython to build new languages.
As far as I understood is that this allows for low-level concepts to be changed more easily, while low-level implementations always suffer from basic decisions (e.g. you will never get reference-counting out of CPython, while you might be free in to change that in an RPython-implemented language).
So, the entire point of PyPy is being able to implement other high-level languages in RPython, so an attempt to implement Ruby is definitely very interesting! Also, the resulting binary is not depending on Python whatsoever.