

Mozilla's JM crosses Apple's Nitro threshold - dpurp
http://arewefastyet.com/#

======
jorangreef
A good browser benchmark would also measure:

1\. Browser boot time.

2\. Browser shutdown time (on clicking Close).

3\. Tab boot time.

4\. Tab shutdown time.

5\. LocalStorage boot time (on first call to LocalStorage API, some browsers
load LocalStorage into memory at this point, blocking the UI).

6\. IO performance of LocalStorage, WebSQLDatabase, IndexedDB for massive
get/set operations (bandwidth) and many get/set operations (latency).

7\. ApplicationCache update time (and update time after an interrupted update
cycle).

8\. Smoothness of moving a DOM element across the screen by adjusting absolute
position (some browsers are choppy with this).

9\. Javascript parse time.

10\. Minimum setTimeout frequency (is it 1ms or 5ms?).

11\. Hash change event propagation time.

12\. Stack size limit.

13\. Speed of Array.shift on massive arrays.

14\. Vertical height of browser chrome at top of the window in pixels (less is
better).

15\. Vertical height of browser chrome at bottom of the window in pixels (less
is better).

16\. Number of colors used in browser chrome, including favicons in location
and bookmark bars etc. (less is better).

17\. Fullscreen support and switch to fullscreen time.

18\. Developer Console boot and shutdown time.

19\. Number of words, tabs, nested interfaces used across all Preferences
pages (less is better).

20\. Number of user actions (clicks, keypresses, mouse movement distances)
required to clear LocalStorage, History, WebSQLDatabase, IndexedDB, Cache,
ApplicationCache (less is better).

21\. Default storage limits for LocalStorage etc. (higher is better).

22\. Data-URI size limits (higher is better).

~~~
fliph
Agreed that most of these would be good components of an overall browser
benchmark, but the linked benchmarks are JavaScript benchmarks only, so things
like browser start-up time and chrome dimensions wouldn't apply.

~~~
jorangreef
When users think "fast", they think "fast" as in "fast", not "JM fast" or "TM
fast". It would be better for "arewefastyet.com" to address "fast" from the
point of view of the user rather than from the point of view of the Javascript
engine developer.

~~~
fliph
As I understand it, arewefastyet.com was created as more of an
internal/community dashboard specifically for the JIT team at Mozilla rather
than to reflect on the browser as a whole to the userbase.

------
samps
One possible takeaway from these graphs is the urgent need for better
JavaScript benchmarks. Both SunSpider and the V8 benchmarks are sets of
microbenchmarks and the difference between them is enormous: JM+TM is
significantly faster than plain JM on V8 but is a little slower on SunSpider.
The fact that the conclusions can vary so greatly among these two (both deeply
flawed) benchmark suites suggests that the true "real-world" performance of
the engines is all but entirely unmeasured. Which loads Gmail fastest? That
would be a step in the right direction.

~~~
natmaster
Here's an alternative:
<http://krakenbenchmark.mozilla.com/kraken-1.0/driver.html>

I have to agree, that things like machine translated scheme and OS scheduling
are not representative of real world applications. (Part of the V8 suite:
<http://v8.googlecode.com/svn/data/benchmarks/v6/run.html> \- specifically
Richards, and EarleyBoyer)

~~~
buster
Keep in mind that V8 is not only running in the browser, but has completely
different usecases as well (see node.js).

That it's a javascript benchmark only means that.. it's used by google to
finetune their code.. not created to compare javascript in browsers among
several different browsers.

~~~
natmaster
And yet that's what people seem to like using it for.

I still don't see why machine translated scheme is a good benchmark for fine
tuning code. Javascript is a fine language to work in - especially if you are
wanting to use it for things other than a browser (then you must really like
it!).

~~~
buster
I haven't seen the code but based on the description it's a benchmark that was
originally in scheme and ported to javascript.. sounds fine to me. And "kernel
simulation benchmark" may have a usecase too, just because you can't think of
it, doesn't mean it isn't there ;) I can think of tests for scheduling,
threading, asynchronous code, I/O when i think of "kernel simulation".

Again, people like to use V8benchmarks, when it's not made for browser
comparisons. The fact that it's often used only shows that there is a lack of
a real cross browser benchmar suite.

~~~
natmaster
"...Florian Loitsch's Scheme2Js compiler"

No, Florian Loitsch isn't a human translator for hire - he wrote a compiler.

~~~
buster
Yes, but the benchmark is not "how fast does V8 translate scheme to JS", it's
"how fast does V8 run benchmarks in javascript that where translated from
scheme some time ago". Atleast that's how i understand it.

~~~
_delirium
That's true, but machine-generated code can be pretty nutty, depending on what
exactly the Scheme2JS compiler does. It's legitimate JS code, but it might not
be representative of what JS-written-in-JS looks like.

~~~
buster
True, but i see that this is a valid testcase. For example GWT generates JS
out of Java, so it might be useful to also benchmark machinegenerated JS.

~~~
ootachi
I think it'd be useful to benchmark _GWT_ -generated JS (and also
Objective-J), but not Scheme2Js, which is rarely if ever used in practice.
Compilers for different languages will emit code with completely different
performance characteristics and feature usage patterns. (For instance,
Objective-J particularly stresses the call site caching mechanisms in modern
JavaScript engines with its "objc_msgSend" equivalent, since the indirect call
in that function branches to so many different call sites.)

------
nathanhammond
This appears to be what got them there:
<http://andreasgal.wordpress.com/2010/10/13/compartments/>

Also worth noting, there is no longer a gigantic "NO" down at the bottom of
the page. :)

~~~
mgedmin
I miss that "NO".

I expected to see "YES!", or maybe "MAYBE" in its place, but there's nothing
there.

~~~
stuaxo
Yeah I was hoping for MAYBE, and then if it ever gets quicker than chrome a
YES!

------
ericflo
What actually surprises me about this is just how much faster V8 was (and
really, still is) compared to all of the others. It also looks like they're
starting to hit a point of diminishing returns. Here's hoping they're not!

------
knuckle_cake
Only x86, though. x64 isn't there yet. <http://arewefastyet.com/?machine=4>

------
Lagged2Death
I am always puzzled by the attention paid to JavaScript execution speed.
Obviously, it's important (and should be) to the people who are intimately
involved with the maintenance and development of the JavaScript engines, but
apart from that, why do so many people in the web business care about it so
much?

What sites and/or web apps are so JavaScript-heavy that they are bottlenecked
by slow JavaScript execution?

In most cases, aren't the speed of a user's network connection and the
responsiveness of the web server itself far more important to the user
experience of a web app than the browser's JavaScript speed? Is GMail on
Chrome really a radically different experience from GMail on IE8?

~~~
queensnake
People are looking ahead to html 5, when real apps can be implemented entirely
in a browser.

------
amix
I have heard from the V8 developers that Mozilla do some optimizations just to
score better on benchmarks - i.e. they score really good on things that are
specifically benchmarked. I have not verified their claim, but generally for
me as an user Chrome and Safari feel a lot faster than Firefox.

~~~
jeresig
Two quick points (I work at Mozilla, but not on the browser - I'm also the
creator of the Dromaeo performance benchmark): _All_ browsers implement things
to get faster at these benchmarks. They implement caching in bizarre places
just to make an, otherwise, useless loop faster. They optimize methods that
are rarely used in modern JavaScript applications. This frustrates me
immensely. Most modern browser performance benchmarks focus exclusively on
JavaScript and completely ignore the DOM, rendering, painting, or any of the
other operations that are critical to the performance of a web site.

As to the faster claim - I've been using Firefox 3.6, Chrome, Safari 5, and
Firefox 4 interchangeably and I've noticed a _dramatic_ improvement in quality
from Firefox 3.6 to Firefox 4 - they've really done a great job with the
upcoming release.

~~~
amackera
Agreed. I'm just using development builds (and lately the official beta), but
Firefox 4 is leaps and bounds better than Firefox 3.6 (which was already
pretty good!)

~~~
trafficlight
Have you had problems where Firefox 4 just fails miserably to render a page,
but if you refresh it renders fine?

------
jsz0
I bet slow JM has cost FireFox a pretty significant number of users over the
last couple of years. Even the much smaller gap between nitro & v8 is
noticeable at times.

~~~
carussell
You have no idea what you're talking about. If you recognize it as
speculation, please don't do it when actual verification is such a simple
process.

Pre-emptive note to readers: Sure "You have no idea what you're talking about"
_sounds_ harsh, but is it wrong? Please consider that before acting upon any
knee-jerk reactions to my comment.

