
HTML5 Gaming Performance Test - AshleysBrain
https://www.scirra.com/blog/144/the-great-html5-gaming-performance-test-2014-edition
======
drewcrawford
Point of order on the methodology here.

This benchmark gets harder the longer it runs. Which is not in itself terribly
unusual. It is pretty smooth for the first bit, where the benchmark runs, but
right about where things get hard on my iPad Air is when the test stops.

So this raises a question about how the test endpoint was decided. Because of
course if you run longer, the performance would be worse, if you run shorter,
the performance will be better, etc.

Now according to the previous article [1], this endpoint is exactly 60
seconds, and an average is taken over that entire period except some time at
the beginning for JIT. Now 60 is a suspiciously round number to arrive at by
trying to hit X many sprites on the screen or numerically deriving some
benchmark objective. Probably, somebody just eyeballed what a Space Blaster
game looked like and one minute of runtime was about right.

That probably is a reasonable predictor for Space Blaster, but it is quite the
leap to use these results to predict "most other HTML5 games" as asserted in
this article. Some games will have half the sprites and others will have
double, some games will be more spikey, and others will be more well-behaved,
etc.

If I was going to use this benchmark to predict my game's performance, I would
do what the benchmark designer probably did. I would ignore the results table,
run the benchmark until the sprite count/size looked about right, and take my
own average of the FPS indicator around that time.

[1] [https://www.scirra.com/blog/107/boosting-mobile-
html5-game-p...](https://www.scirra.com/blog/107/boosting-mobile-html5-game-
performance-with-webgl)

~~~
kevingadd
I'd also be concerned that the benchmark doesn't present a realistic
performance profile, either in CPU or GPU terms.

It appears to have a low entity count up until the end, and not have any
considerable amount of collision detection or other cpu-intensive entity
behaviors. This means it's not really stressing a JS runtime and could
probably perform great under an interpreter (as demonstrated by Ejecta getting
a good framerate)

It also doesn't appear to produce a taxing GPU profile even on a canvas
implementation. It draws very few objects, and most of them share the same
texture and rendering state, ensuring that even a simple rendering
implementation can batch them up into a single draw call with very few
vertices. The framebuffer resolution is low and the overdraw is low as well.

A real-world game, even one that isn't 'too complicated' for mobile, would
have a dramatically different performance profile.

I think this is a solid benchmark, but it's irresponsible to make claims like
'if sbperftest runs well, most other HTML5 games should run decently as well'
and 'The next most common case is usually someone designing or porting such a
complex game that even a native engine would struggle on the limited, battery-
powered mobile hardware.'

You simply can't make those claims without a broader benchmark suite (which I
should note most of the professional mobile/desktop benchmarking suites come
closer to offering, with a handful of different benchmarks based on different
workloads).

~~~
AshleysBrain
Author here. Perhaps the "if sbperftest runs well, most other HTML5 games
should run decently as well" statement should have been inverted, so it reads
"if sbperftest doesn't run well, few other HTML5 games will". We develop a
major HTML5 game engine, and this statement is backed by quite a bit of
testing and profiling of real-world games.

The test is deliberately based on a real-world game (it was adapted from an
interactive game) so it's not a synthetic benchmark. No it is not enormously
onerous for the CPU or GPU, but if you look at the results many mobile devices
still can't quite manage a rock-solid 60 FPS, so I think it serves pretty well
to see if mobile devices are fast enough for relatively simple games. There is
actually a lot of collision testing in the laser-to-enemy intersection test -
all of the objects in the game have custom collision polygons, it's not a
simple bounding box test. The explosions use additive blending (or "lighter"
composite mode) which actually seems to stress many canvas2d implementations
but generally is a lot smoother with WebGL. So it's not _that_ simple a test.

I'm not sure why the 60 second period is important; the most intensive part is
the waves of green sprites which should come well within the measured period,
and it gets quieter after that when the test ends, so it's not skipping the
hard bit.

------
kevingadd
Great idea. One thing I see is that you're graphing FPS, which means that your
data becomes useless once enough of your target devices are able to sustain
60fps.

What I did for my HTML5 benchmarking solution for JSIL games is to instead
record the timing in MS for the update and draw stages of every frame. This
ensures that even if the benchmark is throttled to 60hz (Chrome does this no
matter how you schedule ticks) you can compare timings across browsers on the
same configuration and tell whether things have gotten faster.

You can run the benchmark for yourself here:
[http://www.playescapegoat.com/?replayURI=/stages2and3.replay...](http://www.playescapegoat.com/?replayURI=/stages2and3.replay&disableSound&fastReplay)

Click 'Show Log' to see the timing data. It's been a very useful tool for
evaluating improvements in JITcode quality, GC performance, canvas
performance, and various JIT regressions for complex JavaScript. Just running
it periodically has revealed dozens of regressions in Firefox and Chrome.

Hopefully with more usable data, your benchmark will contribute to spotting
regressions too :D

P.S. feel free to harvest stuff from the JSIL benchmarking code. Everything is
open-source.

~~~
rockdoe
_record the timing in MS for the update and draw stages of every frame_

This tells you nothing about when the frame actually makes it to the screen.
It's ok for optimizing your own code, not so useful for comparing across
devices, platforms & browsers.

~~~
kevingadd
Frame scheduling delay doesn't matter for performance benchmarking. Modern
browser compositors literally grab the frame you hand them and display it at
next vsync. It's irrelevant.

Note that a browser can also run rAF ticks at 60hz even if it's dropping half
your frames to hit 30hz vsync. You won't be able to tell.

It is the case that some browser compositors fuck up the presentation of
frames (hi Chrome), and that sucks, but you can't programmatically detect it.

------
CmonDev
Comparison to native/CLR/JVM would be nice.

------
awesomekling
The main bottleneck in WebKit was Object.prototype.hasOwnProperty(). It wasn't
optimized for the case where the argument is a concatenation of strings, e.g
hasOwnProperty("foo" \+ someString).

I landed a fix here:
[https://trac.webkit.org/changeset/168549](https://trac.webkit.org/changeset/168549)

~~~
AshleysBrain
Awesome, good stuff. IIRC that only happens a lot if Set with forEach isn't
supported, and it uses string properties on an object as a fallback.

------
agapos
Honestly, I was hoping to see an FxOS phone on the charts. I mean, the whole
thing is about the web apps, so what else could be a better way to measure
it's gaming capabilities than the Construct 2? And vice versa, the FxOS seems
to be an ideal (or at least, logical) target for Scirra.

------
camus2
I get a score of 35 on my MBP,so hard to believe it runs faster on mobile
devices,i'll make some tests later at home.

~~~
AshleysBrain
Sounds like software rendering, did you try other browsers or disabling the
blacklist? It's actually relatively common that high-end mobiles like the
Nexus 5 can outperform low-end or software-rendering laptops/desktops.

~~~
robert_tweed
Also, webgl is disabled by default in Safari. You need to enable the developer
menu to turn it on.

------
uptown
How do you have a mobile comparison like this without including the 5s, the
flagship device from Apple?

------
dzhiurgis
I hope this is not too much off topic, but is Chrome on Android running
natively or in Dalvik?

------
33a
FPS isn't really an issue for web games, it's input latency. Current browsers
add something like a 60ms delay to _every_ input event, and there isn't any
fast path for translating inputs to events on screen. This makes it very
difficult right now to create fun and fast paced interactive games in HTML5.

~~~
kevingadd
Input latency is an issue, sure, but saying that framerate doesn't matter is
just ignorant. The existence of attempts at improving throughput (like asm.js
and NaCL and Dart) demonstrates that performance is still a serious concern
for developers.

Also, keep in mind that high throughput - necessary for delivering a
consistent 60FPS - will naturally reduce input latency by reducing the length
of frames and the amount of time spent not pumping input messages.

P.S. It's my understanding that gamepad input in both Firefox and Chrome is
not buffered in any fashion, so that's going to be low-latency.

