

Does Microsoft cheat in IE Test Drive? - wolframhempel
http://wolframhempel.com/2012/11/20/mandelbrot-set-explorer/

======
Trezoid
Running the test on the latest firefox nightly results in a time of ~3
seconds, so if microsoft IS cheating by doing some browser sniffing they might
want to widen the net somewhat, or end up looking bad (in 18 weeks...)

The other option, and still a little dodgy, is that they found operations that
ran poorly in V8, optimized the hell out of them in chakra specifically to say
"Look, we're faster then chrome!" while just letting the fact that
spidermonkey is faster slide because mozilla is way less of a threat then
google.

~~~
kevingadd
Everyone optimizes for their own benchmarks. You can say it's dodgy, but
that's really just an argument in favor of real-world tests instead of
microbenchmarks like this one. You don't have to 'rig' a microbenchmark to
come out the winner; you just pick the right microbenchmark, because it's not
a real world scenario.

Given the lack of evidence to support actual cheating you should probably just
apply occam's razor here and assume it's actually faster. Or you could use the
dark, hidden secrets of 'view source' to read the JS yourself. It has
comments!

~~~
Trezoid
making a benchmark specifically to be slower on your competitors browser?
Little dodgy.

From everything I've read IE10 apparently _is_ fast, and the numbers are
believable from my experience doing similar things (web workers drawing
fractal-like images on a canvas) but given that it's the only "pretty"
benchmark in the lot, it just seems a little strange.

~~~
kevingadd
I think you misread my post. My point is that everyone optimizes their
implementation so that their benchmarks run fast. Furthermore, people pick
benchmarks that run fast on their implementation. Neither of these are a
'little dodgy': They are how benchmarks get used.

You might note that they make for bad science, but benchmarks tend not to get
used in anything resembling a scientific comparison. Benchmarks get used for
comparative testing (did we make thing X faster than it was before?) and to
try and prove that your product is better than the competition (in which case
much larger lies are going to get trotted out either way).

------
opinali
Googler here. There is no cheating in the benchmark. But we noticed this, and
the V8 devs immediately diagnosed it as result of an optimizer failure (not
exactly a bug, but a limitation of current stuff). There's already an in-
progress patch that fixes this problem, so just wait for this to land:
<https://chromiumcodereview.appspot.com/10837165>

------
verroq
It doesn't matter how fast yours run. It's the comparison that matters.

------
kevingadd
This article feels like it ends 10% of the way to actually answering the
question it asks. "Hey, this thing is slow. I'll write something that might be
equivalent, and see if it's not slow, and then I'll stop." If you're going to
accuse someone of 'cheating' or otherwise rigging a benchmark, maybe you
should at least do a proper benchmark yourself, or figure out how they
cheated?

EDIT: Some basic digging reveals that this is using web workers, and V8's
profiler does't record samples for web workers. It appears to be much slower
in the current Release build of Chrome than it is in Canary, so whatever is
causing it to be slow in V8 is actively being worked on by Chrome's
developers.

Unfortunately Firefox's profiler only profiles the main thread as well.
Furthermore the mandelbrot demo doesn't work right in Aurora or Nightly, so
uh, yeah.

Given the use of workers, one guess would be that IE10 offers a much faster
implementation of postMessage, making it cheaper to send the rendered pixel
data back from the workers when it's done. Another obvious candidate would be
a much faster implementation of putImageData for rendering the pixels to the
canvas. A third candidate would be the use of something that IE10's JIT
optimizes that V8 doesn't and Firefox struggles with slightly. Try/catch would
be an obvious one; floats would be another.

The code for the demo is there, and it's got comments, so if you really think
they're cheating, read it. It looks fine to me and it looks like something
that V8 should be able to turn into highly performant assembly, so if it's
not, the fault probably lies with V8, not Microsoft.

~~~
Trezoid
Firefox nightly 20.0a1 (released today) runs the test...very well, actually.

------
Toshio
Chrome has a very good built-in performance profiler for JavaScript.

In the interest of scientific truth, someone (not me) should profile
microsoft's fabricated example and show the world all the dirty browser-
sniffing tricks.

~~~
kevingadd
You don't have to browser sniff to make something run slowly in V8. You can
just write ordinary JavaScript and discover that it runs awful because you hit
one of the hundreds of JavaScript edge cases that each JIT treats differently.

