

Web Latency Benchmark: A new kind of browser benchmark - ggurgone
http://google.github.io/latency-benchmark/

======
nrp
Awesome to see a non-Virtual Reality use of our Latency Tester.

~~~
modeless
It's an awesome piece of hardware, thank you for making it! I'm currently
brainstorming ways to use it more extensively in the tests.

This benchmark does have a connection to virtual reality, as part of my
inspiration to make it came from my experience using vr.js[1] and WebGL for
virtual reality in browsers[2]. The Web isn't yet ready for VR due to latency
issues, but I hope to change that!

On that note, are you guys interested in browser support for the Rift? I
believe the latency issues are surmountable with effort and some cooperation
from browser vendors, and I'd love to see something official in that direction
(in fact, I'd love to _work on_ something official).

[1]
[https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=7...](https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=753)

[2] This is what I made:
[http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s](http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s)

------
109876
1.1 out of 10 running the latest stable version of Chrome on a 2010 13"
MacBook Pro with a 2.40GHz Core 2 Duo, 8GB of RAM, and a Samsung 840 SSD.
Geez... Is there anything I can do to improve the score?

~~~
hnriot
odd, i have much the same mac, same cpu/ram with a regular 7200rpm disk, same
chrome and i get 2.5

------
azinman2
Has anyone assembled benchmarks across the different browsers yet? I like the
aims of this benchmark a lot -- I've recently switched away from Firefox to
Chrome simply because it feels less janky.

~~~
modeless
(benchmark author here) I don't have comprehensive numbers compiled yet, but I
can give you a general ranking of browsers. From best score to worst, the
current ranking is roughly:

    
    
      Safari
      Firefox (except Mac)
      IE 11
      Chrome and Mac Firefox

~~~
Semaphor
I wonder if I'm using my browser differently than others or if there is
something wrong with my machine. Whenever I try to switch from Chrome to FF it
seems way behind Chrome in terms of speed and snappiness.

edit:

Windows 8.1, SSD, i3, 16GB RAM (63% used)

Chrome 30.0.1599.101 m: 3.4/10

Chrome Incognito: 4/10

Firefox 25: 5.4/10

FF Private Browsing 3/10 O.o

All tests done at least twice with similar numbers.

But Chrome has been running for a few days while FF was just started. Guess
I'll leave FF running a bit and see if it changes anything.

Also interesting, FF takes around 2-3 times as long to run the tests than
Chrome.

The reason for the low Chrome score is this:

JavaScript jank Tests responsiveness during JavaScript execution. CSS: 1.0
frames jank, Scrolling: __170.1 __frames jank

~~~
modeless
There are many different ways to measure speed. For example this benchmark
says nothing about page load times or JavaScript speed; Chrome might
(hypothetically) be way faster than Firefox there and still score worse in
this test.

> FF takes around 2-3 times as long to run the tests than Chrome.

There are a couple of reasons for this. Firstly, lower latency is more
difficult to measure, so it takes more trials to get the same number of good
measurements. Secondly, Firefox has longer scroll momentum than Chrome, and
the benchmark waits for the scroll momentum to die down between each scroll
latency measurement.

~~~
Semaphor
The main problem I have with FF is it's scrolling performance ;)

Thanks for the explanation, makes a lot of sense.

Can you explain the extreme FF private browsing results?

~~~
modeless
I have no idea why a private browsing window would perform differently, but
you're right, I get the same results myself. That's weird! It's exactly the
sort of issue I hope to expose with this test. Latency issues like that are
often ignored because they're not visible, and I want to make them visible.

~~~
lstamour
Thanks very much for making this test. I use just about every browser out
there, and love Chrome on my new Nexus 5, but I've felt that with iOS 7 and
10.9, Safari made huge improvements in responsiveness and it was hard to pin
that down until now. I look forward to future optimizations. :)

------
benologist
I can't help but think this should be a test for websites, not computers. It's
individual sites full of 3rd party shit that are slow and you can see the
performance penalty by installing Ghostery.

~~~
modeless
It's more of a test for browsers than computers. You can compare Chrome vs.
Firefox vs. IE vs. Safari.

Measuring the input latency of individual websites would be interesting too.
I'd love to incorporate that into the benchmark if I could figure out a way to
do it.

------
Zikes
1.8/10 on a Dell Latitude laptop.

Did anyone else have difficulty getting it to work on a multi-monitor setup?
It gave me an error about trying to find the reference image and suggested I
switch monitors, but that didn't work. I finally had to disconnect my
secondary monitor altogether before it would work.

~~~
modeless
(benchmark author here) Interesting. I will have to do more testing on Windows
multi-monitor setups. I would not be surprised to learn that it's broken.

~~~
Zikes
If it helps at all, I'm on Windows 7 with my laptop screen as my primary
monitor and an LCD screen to the left as my secondary.

I was initially in Win7's "enhanced graphics" mode (can't remember actual
feature name) but when I tried to run the benchmark I was thrown into Basic
mode. I had a fair amount of other applications running, though, so I may have
just run out of memory.

~~~
modeless
Being thrown into Basic mode is unfortunately expected. I'll bet the reason it
didn't work is having the secondary monitor on the left. I think I only tested
it with the secondary monitor on the right.

------
MrBuddyCasino
Low latency is more important than overall throughput when it comes to UX, so
that is a measurement that makes a lot of sense. Has anyone experience with
it?

------
dmourati
I learned a new definition for jank today!

3.0/10 on MPBR early 2013 w 16GB and SSD.

Authors should consider a way to collect and present the data.

------
sp332
The windows version disables the "Aero Glass" transparency effects on my Win7.
A first glance looks like it might not be intentional, but it triggers some
kind of compatibility mode in the OS. Not sure why a headless daemon would
mess with transparency effects.

~~~
modeless
Benchmark author here. This is unfortunately intentional due to the way the
benchmark works. Measuring latency requires taking screenshots really fast,
and on Windows 7 it is very difficult to impossible to take fast screenshots
with Aero enabled. In Windows 8 they added a new DirectX screenshot API[1]
that makes this unnecessary (which is good because it is impossible to disable
Aero on Windows 8).

[1] [http://msdn.microsoft.com/en-
us/library/windows/desktop/hh40...](http://msdn.microsoft.com/en-
us/library/windows/desktop/hh404487\(v=vs.85\).aspx)

------
nej
Getting different scores when the test is ran at different browser sizes.

3.4 when the window is sized to it's smallest size.

2.5 when it's at maximum resolution (1600x1200).

Running on OSX 10.9.

~~~
modeless
Yes, window size can definitely affect latency and jank. The benchmark page
has a hidden full-window-sized layer that the browser must paint every frame,
and a large browser window requires more CPU and GPU work to paint.

------
k3n
Keeps failing for my Win7x64/30.0.1599.101 m.

> Failed to find test pattern on screen. Ensure that your browser's zoom level
> is set to "100%", and the top-left corner of the window is visible. If you
> have multiple displays, try moving the browser window to the main display.

* Zoom 100%: check.

* Main monitor (of 2): check.

* Top-left visible: check.

I give up.

~~~
modeless
Sorry to hear that. Due you have the Chrome flag for "Experimental Web
Platform features" enabled? I recently learned that there's an issue with the
test related to this flag.

Edit: It's more likely that it's just buggy support for multiple monitors. I
will look into fixing it; for now the only workaround would be disabling your
second monitor.

~~~
k3n
Sorry for the late reply; I did notice a Github issue for it so I commented
there. Thanks!

------
rMBP
Does not work in Safari on Mavericks using a rMBP @1680x1050 retina
resolution. It suggests browser zooming be set to 100% which it is.

~~~
modeless
Thanks for the report; I see the problem. I'll look into fixing it, but in the
meantime you can get it running by enabling WebGL in Safari (under the Develop
menu, and if you don't have a Develop menu you can enable it in
Preferences->Advanced).

Spoiler: Safari actually gets a very good score (relative to other browsers).

~~~
rMBP
Enabling WebGL works. My three tests scored 8.7, 6.7 and 6.3 respectively. I
guess Safari, like me, hates repetitive tasks. :)

~~~
modeless
:) There is a lot of run-to-run variance. Some of it is real in that browsers
do give different results, and some of it is just test noise. I'm still
working on improving the robustness. Recently I've been working on hardware
latency measurement which gives more reliable numbers.

------
lucideer
keep on getting [https://github.com/google/latency-
benchmark/blob/master/src/...](https://github.com/google/latency-
benchmark/blob/master/src/latency-benchmark.c#L505) on the first test... not
sure why.

~~~
modeless
Interesting, what platform and browser?

~~~
lucideer
Vista all attempted browsers (tried FF, Chrome, Opera)

~~~
modeless
Hmm, I don't have a Vista machine around to test on unfortunately.

------
andrethegiant
Has anyone else heard the term jank before? It's the first time I've seen it.

~~~
makepanic
There is a website [1] with more informations and talks about "jank".

[1] - [http://jankfree.org/](http://jankfree.org/)

------
cheald
3.3 on a beefy Windows 8 machine running Chrome 31.0.1650.39 beta-m.
Interestingly, running an Incognito tab with no extensions allowed to run in
incognito results in a 2.7.

Huh.

~~~
dmourati
Lower scores are better since we are measuring latency?

~~~
gjm11
Looking at the source, higher scores are better. There are a bunch of tests,
and each one produces a score from 0 (bad) to w (good) for some w. The total
score is the total of all these.

Given that all the other numbers on the page are ones for which smaller is
better, I think the main score itself should be accompanied by a note saying
explicitly that higher scores are better.

~~~
modeless
Good point. I'll add a note next to the scores.

------
carlob
I managed to get a 6.5 on Safari on Mavericks on a 3.5 year old MBP. All the
other browser I've tried were below 2, except for Opera Next (2.2).

------
madh
2.8 / 10.0 on a 1 week old 15" MBP (high-end) running latest stable Chrome.

------
ygra
Somehow the second test always timeouts in IE 11 on Windows 8.1 for me.

