

Testing JavaScript performance on Windows XP is a crapshoot, at best - boundlessdreamz
http://ejohn.org/blog/accuracy-of-javascript-time/

======
jd
I don't get this.

Quote: _To better understand the results I was getting I built a little tool
that runs a number of tests: Running an empty function, looping 10,000 times,
querying and looping over a couple thousand divs, and finally looping over and
modifying those divs. I ran all of these tests back-t0-back and constructed a
histogram of the results._

What's the point of doing this? It's a microbenchmark, and it's pretty unclear
what you're measuring. Why put so much effort in writing a blog post with
graphs and the whole shebang just for this?

Even if you _did_ get constant results the results would be uninteresting. The
results are either uninteresting and unreliable, or uninteresting and
reliable. Shortcircuit logic keeps me from worrying about reliability.

And that the Windows GetTickCount/GetTime functions have a bad precision is
common knowledge. No test is needed to establish that.

I'm pretty sure Resig is a smart guy, so I'm probably missing the point here.
But I really don't understand what he's trying to accomplish here, or why the
results are worth blogging about.

~~~
jeresig
The point isn't to measure something specific at a rate - it's to measure
_anything_ to see if the timer results continue to bin the results. It doesn't
matter if it's a microbenchmark, or not, at that point.

The issue here is that Chrome and Firefox 3 are perfectly capable of producing
accurate time results on Windows - so there obviously is a better solution out
there. In the ticket in which the Firefox team [1] fixed the bug they detail
the solutions that they used to fix it.

I would disagree that this is a knowledge issue here, as well. It certainly
wasn't obvious to the WebKit, Firefox, and Chrome teams who were all happily
testing on Windows XP using bum tests - and those are some pretty smart people
(IMO).

In reply to your comment further down the thread: "Measure browser
performance.... why? Why do you care how fast your browser is?"

Because it's better to have fast JavaScript than to have slow JavaScript? Any
performance tests are better than no performance tests - but I will definitely
agree that quality is a huge factor. For Dromaeo [2] I included a number of
DOM tests that provide a more-real-world result. I'm working to try and
provide more (especially related to CSS and rendering).

[1] <https://bugzilla.mozilla.org/show_bug.cgi?id=363258#c19> [2]
<http://dromaeo.com/>

~~~
jd
No. No. No. It doesn't matter. Not one bit. Exception: for FF maintainers,
opera programmers. But ordinary javascript programmers never have to worry
about a ms here or there.

Let me quote again from your blog post:

 _This was quite odd since the tests should've taken, at least, a couple
milliseconds to run and getting consistent times of "0" is rather infeasible,
especially for non-trivial code._

Conclusion 1: The code _is_ trivial! Why? it takes < 64ms to execute.

Conclusion 2: The actual time spent by the code doesn't matter, because you
can't notice the difference between the reported 0ms and the actual 3ms or so
that it takes to execute.

Next paragraph:

 _Second, I was running some performance tests, on Internet Explorer, in the
SlickSpeed selector test suite and noticed the result times drastically
fluctuating. When trying to figure out if changes that you've made are
beneficial, or not, it's incredibly difficult to have the times constantly
shifting by 15 - 60ms every page reload_

Why does it matter if loads 15ms to 60ms faster/slower? It's not noticeable.
And once you see that the difference is roughly 15ms a quick google search
will reveal it's a simple timer accuracy issue.

So why the graphs? Why test different browsers on different platforms when the
outcome can't be anything else than the 60ms fluctuations you've already seen?
But you knew the second you saw a single time reported as 0ms that that the
time accuracy is not on the 1ms precision level, which means that they use
GetTickCount or something which has a 15ms precision.

> It certainly wasn't obvious to the WebKit, Firefox, and Chrome teams who
> were all happily testing on Windows XP using bum tests

If they used the tests they would've noticed the 0ms results. Therefore, they
either thought 0ms means it's <15ms, which is fast enough, or they tested it
primarily on other platforms.

I don't believe for a second that the WebKit, Firefox and Chrome teams have
done ms comparisons with tests that don't report data on the ms level. 0ms
results stand out, after all. They probably are pretty smart people, and they
realized it doesn't matter.

> Any performance tests are better than no performance tests

Nope. Silly performance tests are worse than no performance tests.

------
StrawberryFrog
Well, I'm not a Javascript coder, but you use accurate tick counts for
performance profiling, not the coarse-grained time of day clock.

I thought everyone knew that?

~~~
pmjordan
I don't know of a way to access those from browser JavaScript though.

------
edw519
Javascript benchmarking: a precise estimate of a wild ass guess.

------
pmjordan
I have to wonder if part of the problem is that he's running Windows in a VM.
VMs are notorious for their odd timing behaviour. He's running the comparison
tests in a native installation of OSX, not in a VM.

~~~
jeroen
From the article:

 _I was worried that these results were from the virtual machine (I also
loaded up Parallels but saw similar results to running VMware) so I just
loaded Windows XP proper: Nope, the results are the same as using the VM._

