
Why we challenge Microsoft’s battery test - doener
http://www.opera.com/blogs/desktop/2016/06/over-the-edge/
======
kylealden
Edge platform team employee here. A couple thoughts on methodology, focusing
on our scripted test (under "stay productive longer" at
[https://blogs.windows.com/windowsexperience/2016/06/20/more-...](https://blogs.windows.com/windowsexperience/2016/06/20/more-
battery-with-edge/)), which is most comparable to Opera's test; nobody seems
to challenge our Netflix results:

* We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if you cut off ads, you're effectively skipping half the assignment. This is a valid way to improve the user experience, but not a valid way to measure browser efficiency, especially because it will disproportionately impact some sites (like the news sites that Opera's test focused on) and have no impact on other sites (like Netflix, where Edge demolishes the other browsers). It's basically skipping a lap of the race and then bragging that you finished faster.

* It's also worth noting that the ad blockers are often detected by many news sites and will actually prevent the main page content from loading at all. Not sure if Opera's test accounted for this.

* Our test was designed to mimic real-world behavior: Watching YouTube (foreground and background), shopping on Amazon, browsing the Facebook news feed, searching on Google, opening email in Gmail, and reading Wikipedia. To reduce variability, we used WebDriver to instrument the tests (supported by all four browsers tested) and made sure each task was timed rather than just a loop of consecutive tasks (which could disadvantage or advantage browsers depending on other factors like pageload performance or network conditions, in a way that doesn't reflect user experience, which is likely to linger on a page once it's loaded). We then used the Maxim 34407 power instrumentation built into the Surface Book (which is why we chose the Surface Book for this test) to measure actual instantaneous power consumption at the hardware, sampled once per second and then averaged across the duration of the test. We feel strongly that this is a highly scientific and defensible test setup which mimics typical user behavior and, significantly, measures the same markup and the same duration, on the same hardware, in every browser.

~~~
mynewtb
How did you control for the specific ads the browsers were served? Ads are
highly dynamic. Unless you (can) include the differences in your methodology
you better eliminate that influence.

~~~
patcheudor
Ding, ding, ding and to be fair this is the problem in both methodologies. The
only way to accurately perform this test is to spider a bunch of sites, save
the contents to a locally hosted HTTPD and ensure all third-party JS calls are
resolved locally and test both against the exact same sites as they were
spidered at a point in time. You can simply not account for changes which may
happen to the markup, ad network beacons, ads, metrics code, or even network
routing, all of which could influence the test in one way or the other if not
run from an identical static cache in a controlled environment. If I were
doing this test I'd then skip the whole scripting automation part and simply
add a meta-refresh to every page in the cache to sequentially take the browser
through the content, giving each page something like 10 seconds to load and
render. Simple, simple, and far more accurate.

~~~
csl
You don't need to go to such complicated lengths. Just perform enough tests
(as in, a statistically large enough amount) and a distribution will form.
That also captures the variability of real world network effects.

~~~
breakingcups
What about different ads served to different browsers? Someone running, say,
Opera will have a different ad profile than a Chrome user even when completely
blank cookie-wise.

~~~
drdeca
Maybe have them give the same user agent?

------
overgryphon
Microsoft didn't just do a battery test, they also release statistics based on
Windows 10 telemetry showing that Edge has better battery life than Chrome,
Firefox:

"In addition to testing battery life internally, Microsoft also looks at tens
of millions of PCs that send telemetry data to the firm, Weber said. And what
it sees there is simple: A small advantage over Firefox, but massive gains of
about 50 percent additional battery life when compared to Chrome."

[https://www.thurrott.com/windows/windows-10/68176/microsoft-...](https://www.thurrott.com/windows/windows-10/68176/microsoft-
touts-battery-life-advantages-edge-browser)

~~~
cronjobber
> they also release statistics based on Windows 10 telemetry showing that Edge
> has better battery life

It's such a pity I don't use _Microsoft Windows 10_ , because I'd be _so_
thrilled to know that through the magic of _telemetry_ , everything I do on a
_Microsoft Windows 10_ enabled computer could help Microsoft win some
_crucial_ PR battles.

~~~
KirinDave
As opposed to your _Apple iPhone_ where through the magic of _telemetry_ every
app is dumping your every finger motion into services like Mixpanel and Kahuna
for later analysis.

For clarity, they captured the extremely detailed telemetry _from their test
devices_. Most of the telemetry that Microsoft ships back to the mothership is
the exact kind of telemetry that apps from the Apple Store, iOS Store, Android
Store and Chrome Web Store submit: app specific telemetry. Many of these
include signals that help understand battery life impact, so every major
application platform (including the web) is doing this.

That stuff–stuff you agree to in the EULA as it stands– is absolutely
essential to improving app experience to the standards of the market. The
reason the mobile ecosystem has developed so fast (and that the web ecosystem
can develop so fast) is a whole lot of telemetry on user usage patterns,
habits, device failures, device selection, etc.

I'm not sure why it's so controversial. Even the Ubuntu store apps can
administer telemetry.

If you'd like to complain about the pervasive surveillance state in your
national scale, there are more fruitful avenues.

~~~
megablast
> As opposed to your Apple iPhone where through the magic of telemetry every
> app is dumping your every finger motion into services like Mixpanel and
> Kahuna for later analysis.

This is not true at all. And there is a difference between your smartphone and
your laptop, even if you can't see it.

~~~
MikeHolman
Yes, telemetry from my smartphone is even more invasive to my privacy, having
my location at all times, as well as being the single location with all my
communications.

------
amq
Opera with an ad blocker vs Edge without it doesn't seem like an ubiased test.

~~~
guroot
Even with the ad blocker being a feature in opera out of the box? They were
trying to show that opera can achieve better battery performance based on it's
features and optimizations. Does that not include the ad-blocker?

~~~
bialpio
In that case you might as well browse different websites because using
adblocker makes your browser render different content.

~~~
feld
Microsoft is welcome to include a native content blocker in their rendering
engine

~~~
zeta0134
While I applaud their work and appreciate that there is an extension framework
at _all,_ I seriously doubt this will happen unless they plan to also redesign
their home pages to remove the ads.

I just loaded up MSN.com and scrolled down the page. I count 3 ads total: one
gigantic banner at the very top of the page before all of the content, and two
smaller panners that pop into place as you scroll. These are served by
AdChoices. When I turn my ad blocker back on, these all disappear, and the
site loads much faster.

I think if Microsoft were to release a default ad-blocker, they would need to
re-code their own website to use native advertising rather than a third-party
network. I believe this to be an excellent practice, but I just don't see
Microsoft going this route. I'd love to be pleasantly surprised though.

~~~
a1a
I assume they would implement a white-list allowing "some non-intrusive adds"
similar to Adblock.

------
stephenr
So do the WebKit/safari teams' respective efforts to reduce power consumption
not count? It seems odd that they didn't compare against safari given Apple
makes a dedicated effort to reduce power consumption.

That it's a "mode" which defaults to off says a lot - the things it does
aren't acceptable all of the time apparently, so you choose between
features/performance, and battery life.

~~~
vilmosi
Well you can't really compare Safari with Edge as they are are exclusive to
different platforms.

~~~
blacksmith_tb
It wouldn't be too hard to dual-boot Win10 on Apple hardware and compare them,
but it might be a little awkward for Microsoft to do so officially.

~~~
adrianN
The OS is a pretty obvious confounding factor, don't you think?

~~~
JulienSchmidt
I think it would still be interesting. Win10 + Edge vs macOS + Safari.

In the end the user does not care which OS runs in the background, just which
platform offers the best experience - in this case the longest battery time.

~~~
Tloewald
There are plenty of suitable comparisons from which one can make an educated
guess. In general, Macs do better than comparable Windows machines in browsing
endurance tests, and Safari clobbers every other browser it has been tested in
for battery life (I have _not_ seen a comparison of Safari to this new version
of Opera, but judging by its performance relative to Chrome it could be
competitive with Safari).

Safari, of course, has lots of support for ad blocking short of Apple actually
doing the ad blocking itself. I suspect Safari + ad blockers would be the best
option across the board (it absolutely rocks on iOS devices).

------
kraftman
Obviously this is anecdotal but I've been messing around with my laptop a lot
recently trying to get to a full day of work on battery, and the best setup
I've found is windows 10 Insider with Edge + adblock.

Opera battery saver is great but they should enable it automatically whenever
you're on battery, as it has no performance impact that I've noticed.

~~~
JTon
Edge allows extensions now or are you using an OS-level adblock?

~~~
dangrossman
Edge allows extensions now.

[https://developer.microsoft.com/en-us/microsoft-
edge/extensi...](https://developer.microsoft.com/en-us/microsoft-
edge/extensions/)

~~~
vxNsr
I like how RES is one of the first options. I remember when the Windows 10
team did an AMA and a lot of the redditors said they'd try edge as soon as RES
was available. Apparently that's now.

------
jordanthoms
It's great to see battery efficiency becoming something browser vendors fight
over (even if it does come with a side of the usual benchmark gaming).
Hopefully it shames Google into improving Chrome's battery performance.

------
lavalampkid
Definitely not a fair test... Edge was rendering a lot more without an ad-
blocker.

~~~
Quai
You might argue that not rendering ads is a valid method to save battery.

~~~
bialpio
Correct, it is a valid method to save a battery. It makes the comparisons
invalid though, since you are not testing with the same input.

~~~
Retric
You are testing more or less the same input just producing different output.
This was a common strategy where graphics cards for example would degrade
image quality for speed when a driver noticed you where running a benchmark.

Which is one of the reasons benchmarks are so difficult.

PS: A more insidious problem is when you specifically optimize for something
because it's a benchmark.

~~~
bialpio
Not neccessarily same input - all depends how adblock works. I assumed that
with adblock, my browser will not even send a request to fetch the ads, so I
save on network traffic, and the browser will not have to render whatever
would have been fetched - this to me is a different input (to the browser's
rendering engine) than you would have without adblock.

With all this said, "proper-er" benchmark would be to show each of the
browser's power consumption with or without adblock, with or without battery
saver in Opera's case, all this done on a set of websites which are ad-heavy
and ad-light. :)

~~~
Retric
Server sends an HTML and browser fetches what it needs. Not fetching content
is simply an advanced form of caching.

More importantly even if you grab the content it's rendering that takes power
downloading files is very low energy. Even just disabling auto play sound /
video is going to save power.

PS: Assumptions about how browsers interact with websites is basically a
failure to understand what's going on IMO. Telnet to port 80, send fetch
request on a web-server and you will get HTML unless they require HTTPS.

~~~
bialpio
> Server sends an HTML and browser fetches what it needs. Not fetching content
> is simply an advanced form of caching.

I didn't understand this part. Caching is basically "not fetching content that
you already have", and I don't see how it is relevant here. I assume that when
using adblocks the browser will not even issue a request to fetch the data (be
it HTML, JavaScript, flash videos, images, etc.) that is ad-related, so it is
not the same as caching. And even if it did choose to fetch the data but then
skip rendering, it boils down to the same thing: browser engine has different
(presumably smaller) input in case when you use adblock.

By telnetting I will get a response, but this is only a first step. HTML will
have references to huge amount of other content that also needs to be fetched
(unless everything is embedded in the HTML - possible, but unlikely for any
bigger website). Open developer tools in any browser and go to network tab and
see for yourself how the amount of fetched data (and number of requests /
number of fetched files) differs when you use adblock and not. This is the
difference in the input for the browser. Opera mentions mlive.com - with
uBlock, FF fetches 2.5MB of data (caching disabled), without uBlock I get 5MB
(caching disabled). We can even take the example to the extreme - let's go to
the website that has only ads. Of course Opera will be better there - it will
have to render nothing, everything will be blocked by adblock.

~~~
Retric
You can also cache intermediate results from calculations, the idea is don't
fetch things you don't need. And as you know the result from fetching an add
is going to be blank you don't need to fetch it. Much like a screen reader not
downloading a JPEG.

As to downloading you can try it you self your talking fractions of a watt
savings from not downloading a few Meg's.

In terms of other downloads from a page, most of that is things like tiny
images which you can mostly skip. The Amazon homepage sends lots of junk but
all you actually care about is the search bar and links to other areas.

~~~
bialpio
But how is it all relevant to the discussion? Bottom line is that using
adblock in one browser and not using it in other browser makes the comparisons
meaningless...

~~~
Retric
People are objecting to add blocking as a perfectly normal browser feature.
There is a mindset that as soon as people start wearing augmented reality
glasses they are going to want context sensitive advertising in their day to
day lives. IMO, most people would much rather block real world adds and
overlay them with blank space or even nature scenes.

Browser advertising fits into the same argument as adds on cable television.
Subscribe to Hulu and no the adds don't go away.

~~~
bialpio
Let me reiterate as you did not address anything from my comment: But how is
it all relevant to the discussion? Bottom line is that using adblock in one
browser and not using it in other browser makes the comparisons meaningless...

~~~
Retric
Adblock is a browser feature just as much as GPU acceleration.

Suggesting you need to add third party GPU acceleration to make things fair is
ridiculous. But, you seem to think Adblock is somehow different.

~~~
bialpio
No, I can make things fair by doing another test. Let's test Opera with
adblock and Edge without, but only use websites that do not have ads. Sounds
good?

Problem with Opera's comparison is that it is not fair because they don't
display the same content - they (probably) need to render less. Of course they
will be better if they render less (unless they are really bad at what they
do). So to make the comparison meaningful, make them display exacly the same
thing - then we can talk about who is more power-efficient.

------
dep_b
Opera being better than Edge is a claim I might believe but I don't think Edge
is almost as bad as Chrome.

I stopped using Chrome when I discovered it took hours off my battery life on
a MacBook. Edge can't be almost as bad. Intuitively I would believe the
ranking in terms of battery life would be Safari > Edge or Opera > Firefox >
Chrome.

~~~
LoSboccacc
opera likes to boldly claim it was the first browser to do power management on
pages but truth is Microsoft has done this for many years prior and has the
operating system integration as an advantage.

[http://www.sevenforums.com/tutorials/159269-internet-
explore...](http://www.sevenforums.com/tutorials/159269-internet-explorer-
javascript-timer-frequency-power-plan-settings.html)

------
darkmuck
Now that Opera Software is owned by a Chinese company I've avoided their
browser anyways, regardless of potential battery/performance benefits.

~~~
mwcampbell
Isn't that just xenophobia?

~~~
darkmuck
No, I don't think so. Maybe I'm a bit paranoid but I tend to avoid software
from Chinese companies because of all recent stories of a apps
stealing/sending data that they don't necessarily need to function. But I
suppose software from any country could do the same thing though...

~~~
grouchoboy
And what is your position about hardware?

~~~
curiousDog
Most hardware we use here, though manufactured in China, is still wholly owned
by American companies though.

------
Raphael
> Reducing frame rate to 30 frames per second

Many users may not like this.

~~~
barsonme
probably depends what you're doing, right? e.g., if i'm just sitting here
reading hacker news i can't tell the difference between 60 fps or 30 fps
repainting, right?

------
post_break
What really surprised me were the run times of that laptop. Man that is crappy
battery life for both instances. My thinkpad T220 with extended battery and
13" Macbook pro both decimate that battery life.

~~~
ygra
The Lenovo Yoga (about 550 €) is hardly on the same price level as the
ThinkPad or Macbook Pro (1500 €) ... And you even mentioned the extended
battery.

------
callesgg
I have found that safari gives best battery time on OSX (like 9 hours of use
vs 6-7 when on firefox).

But i generally dont use safari as battery time is not the most important
factor in my common use cases.

~~~
Jerry2
>But i generally dont use safari as battery time is not the most important
factor in my common use cases.

I use Safari for almost everything. It's by far the best browser for browsing
and just reading & enjoying the web. Reader interface and the speed of it is
still unbeatable.

I use Chrome for web development, however.

~~~
eddieh
I agree Safari is the best. Every other browser is a clunker IMO. The Safari
web inspector keeps getting better. You might want to take another look at it.

------
z3t4
I wonder how much battery would be saved if not every site you visited was
reported to bing api or opera sitecheck. <g> But most power is is probably
used to light the screen, so an extension to make every page use a black theme
would probably beat most optimizations.

~~~
andromeduck
I don't think it would take much of a difference on an LCD display because
your backlight is going to be on regardless.

------
Theodores
Opera need to market themselves as the browser that extends your battery life
so you can surf without the range anxiety. Testimonials are better than tests
plus word of mouth.

~~~
outworlder
This is the first time I've heard the term "range anxiety" applied to laptops,
as opposed to electric vehicles.

------
nachtigall
Where's firefox in the perf?

~~~
Sylos
In the battery-benchmarks that I've seen so far, Firefox was always the
leading browser, so granted those benchmarks did not test Opera's battery
saving mode, Firefox should still be at least in second place.

And Firefox does also have a native ad-blocker, so if you enable that, too, I
could imagine that it maybe even still outperforms Opera's battery saving
mode, but yeah, a proper benchmark would be good...

~~~
nachtigall
> Firefox does also have a native ad-blocker

I don't think it has one natively. Which one do you mean? Or are you referring
to tracking protection (private mode only, but available in Fx 49 also in
"normal" mode iirc)

~~~
Sylos
Yeah, I'm referring to Tracking Protection. It's not supposed to be an ad-
blocker, but with almost all ads using tracking code, you rarely encounter ads
when it's enabled. It's probably even better for battery life than just an ad-
blocker, as a lot of webpages also execute tracking code which isn't bound to
ads.

And you could always enable it in normal browsing by setting
"privacy.trackingprotection.enabled" in about:config to true...

------
n-gauge
On a side note: I've been testing edge on a Dell 1012 netbook (atom, 2gb) and
it beats Chrome for rendering YouTube videos (480p max!) which I found
surprising (video stutters on Chrome). However on my main laptop when running
scratch (ok flash based), cloud based games are better sync'd when using
Chrome (it's a slither.io clone). Overall battery life is exceptable for both
browsers.

------
petre
After I read this I've installed Opera mobile and it's now my default browser.
The ad blocker runs pretty well, along with a customized hosts file mapping
advertisers' domains to localhost. Too bad DuckDuckGo is only available as a
search engine in Opera Mini beta. Looking forward to get rid of Google search.

------
yitchelle
Coming from the embedded world, I have always concern about CPU loading which
translates to how much power you are sucking.

I can't not understand why minimal CPU load is not on top of design
requirement of many software products, totally ignored if on a desktop system.

~~~
XCSme
Because PC Master Race. We only want 300FPS+ and 4k graphics, we don't care
about $5 extra to our electricity bill, so why would software developers even
bother with this and spend extra time and money for improving power drain?

~~~
yitchelle
Thanks for the laugh!

Seriously, imagine the world collective savings on energy if everyone reduces
their power consumption by small modest 5%.

------
buremba
Ad-blockers are taking over the internet. It seems that most people here
believe that Ad-blockers must be enabled by default in browsers which is quite
strange.

------
luckystarr
They should also release control videos which show both machines with battery
saver on or both off.

