It's an awesome piece of hardware, thank you for making it! I'm currently brainstorming ways to use it more extensively in the tests.
This benchmark does have a connection to virtual reality, as part of my inspiration to make it came from my experience using vr.js[1] and WebGL for virtual reality in browsers[2]. The Web isn't yet ready for VR due to latency issues, but I hope to change that!
On that note, are you guys interested in browser support for the Rift? I believe the latency issues are surmountable with effort and some cooperation from browser vendors, and I'd love to see something official in that direction (in fact, I'd love to work on something official).
1.1 out of 10 running the latest stable version of Chrome on a 2010 13" MacBook Pro with a 2.40GHz Core 2 Duo, 8GB of RAM, and a Samsung 840 SSD. Geez... Is there anything I can do to improve the score?
Has anyone assembled benchmarks across the different browsers yet? I like the aims of this benchmark a lot -- I've recently switched away from Firefox to Chrome simply because it feels less janky.
(benchmark author here) I don't have comprehensive numbers compiled yet, but I can give you a general ranking of browsers. From best score to worst, the current ranking is roughly:
Safari
Firefox (except Mac)
IE 11
Chrome and Mac Firefox
I wonder if I'm using my browser differently than others or if there is something wrong with my machine. Whenever I try to switch from Chrome to FF it seems way behind Chrome in terms of speed and snappiness.
edit:
Windows 8.1, SSD, i3, 16GB RAM (63% used)
Chrome 30.0.1599.101 m: 3.4/10
Chrome Incognito: 4/10
Firefox 25: 5.4/10
FF Private Browsing 3/10 O.o
All tests done at least twice with similar numbers.
But Chrome has been running for a few days while FF was just started. Guess I'll leave FF running a bit and see if it changes anything.
Also interesting, FF takes around 2-3 times as long to run the tests than Chrome.
There are many different ways to measure speed. For example this benchmark says nothing about page load times or JavaScript speed; Chrome might (hypothetically) be way faster than Firefox there and still score worse in this test.
> FF takes around 2-3 times as long to run the tests than Chrome.
There are a couple of reasons for this. Firstly, lower latency is more difficult to measure, so it takes more trials to get the same number of good measurements. Secondly, Firefox has longer scroll momentum than Chrome, and the benchmark waits for the scroll momentum to die down between each scroll latency measurement.
I have no idea why a private browsing window would perform differently, but you're right, I get the same results myself. That's weird! It's exactly the sort of issue I hope to expose with this test. Latency issues like that are often ignored because they're not visible, and I want to make them visible.
Thanks very much for making this test. I use just about every browser out there, and love Chrome on my new Nexus 5, but I've felt that with iOS 7 and 10.9, Safari made huge improvements in responsiveness and it was hard to pin that down until now. I look forward to future optimizations. :)
I can't help but think this should be a test for websites, not computers. It's individual sites full of 3rd party shit that are slow and you can see the performance penalty by installing Ghostery.
It's more of a test for browsers than computers. You can compare Chrome vs. Firefox vs. IE vs. Safari.
Measuring the input latency of individual websites would be interesting too. I'd love to incorporate that into the benchmark if I could figure out a way to do it.
There are already good tools for measuring network delays. Firefox's web inspector does a beautiful job of graphing how the various requests contribute to overall page load time. This tool is about measuring how responsive the UI is, even when you're still fetching more data.
Did anyone else have difficulty getting it to work on a multi-monitor setup? It gave me an error about trying to find the reference image and suggested I switch monitors, but that didn't work. I finally had to disconnect my secondary monitor altogether before it would work.
(benchmark author here) Interesting. I will have to do more testing on Windows multi-monitor setups. I would not be surprised to learn that it's broken.
If it helps at all, I'm on Windows 7 with my laptop screen as my primary monitor and an LCD screen to the left as my secondary.
I was initially in Win7's "enhanced graphics" mode (can't remember actual feature name) but when I tried to run the benchmark I was thrown into Basic mode. I had a fair amount of other applications running, though, so I may have just run out of memory.
Being thrown into Basic mode is unfortunately expected. I'll bet the reason it didn't work is having the secondary monitor on the left. I think I only tested it with the secondary monitor on the right.
Low latency is more important than overall throughput when it comes to UX, so that is a measurement that makes a lot of sense.
Has anyone experience with it?
The windows version disables the "Aero Glass" transparency effects on my Win7. A first glance looks like it might not be intentional, but it triggers some kind of compatibility mode in the OS. Not sure why a headless daemon would mess with transparency effects.
Benchmark author here. This is unfortunately intentional due to the way the benchmark works. Measuring latency requires taking screenshots really fast, and on Windows 7 it is very difficult to impossible to take fast screenshots with Aero enabled. In Windows 8 they added a new DirectX screenshot API[1] that makes this unnecessary (which is good because it is impossible to disable Aero on Windows 8).
Yes, window size can definitely affect latency and jank. The benchmark page has a hidden full-window-sized layer that the browser must paint every frame, and a large browser window requires more CPU and GPU work to paint.
Thanks for the report; I see the problem. I'll look into fixing it, but in the meantime you can get it running by enabling WebGL in Safari (under the Develop menu, and if you don't have a Develop menu you can enable it in Preferences->Advanced).
Spoiler: Safari actually gets a very good score (relative to other browsers).
I also get "Failed to find test pattern on screen." in Safari with WebGL enabled on a Mid 2011 Mac Mini running 10.9 with dual monitors. View is set to "Actual Size", and I've tried moving the Safari window between monitors.
Other scores - Chrome 30: 2.4/10. Firefox 24 and 25: 1.2/10.
:) There is a lot of run-to-run variance. Some of it is real in that browsers do give different results, and some of it is just test noise. I'm still working on improving the robustness. Recently I've been working on hardware latency measurement which gives more reliable numbers.
> Failed to find test pattern on screen. Ensure that your browser's zoom level is set to "100%", and the top-left corner of the window is visible. If you have multiple displays, try moving the browser window to the main display.
Sorry to hear that. Due you have the Chrome flag for "Experimental Web Platform features" enabled? I recently learned that there's an issue with the test related to this flag.
Edit: It's more likely that it's just buggy support for multiple monitors. I will look into fixing it; for now the only workaround would be disabling your second monitor.
We (at Empirical) use janky to describe animations that aren't butter as this author uses it. Not sure how standard but I suspect many people (native speakers) will understand when in context.
3.3 on a beefy Windows 8 machine running Chrome 31.0.1650.39 beta-m. Interestingly, running an Incognito tab with no extensions allowed to run in incognito results in a 2.7.
Looking at the source, higher scores are better. There are a bunch of tests, and each one produces a score from 0 (bad) to w (good) for some w. The total score is the total of all these.
Given that all the other numbers on the page are ones for which smaller is better, I think the main score itself should be accompanied by a note saying explicitly that higher scores are better.
