* We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if you cut off ads, you're effectively skipping half the assignment. This is a valid way to improve the user experience, but not a valid way to measure browser efficiency, especially because it will disproportionately impact some sites (like the news sites that Opera's test focused on) and have no impact on other sites (like Netflix, where Edge demolishes the other browsers). It's basically skipping a lap of the race and then bragging that you finished faster.
* It's also worth noting that the ad blockers are often detected by many news sites and will actually prevent the main page content from loading at all. Not sure if Opera's test accounted for this.
* Our test was designed to mimic real-world behavior: Watching YouTube (foreground and background), shopping on Amazon, browsing the Facebook news feed, searching on Google, opening email in Gmail, and reading Wikipedia. To reduce variability, we used WebDriver to instrument the tests (supported by all four browsers tested) and made sure each task was timed rather than just a loop of consecutive tasks (which could disadvantage or advantage browsers depending on other factors like pageload performance or network conditions, in a way that doesn't reflect user experience, which is likely to linger on a page once it's loaded). We then used the Maxim 34407 power instrumentation built into the Surface Book (which is why we chose the Surface Book for this test) to measure actual instantaneous power consumption at the hardware, sampled once per second and then averaged across the duration of the test. We feel strongly that this is a highly scientific and defensible test setup which mimics typical user behavior and, significantly, measures the same markup and the same duration, on the same hardware, in every browser.
Why is a different question and it's not that relevant, I often don't care why something I USE works better I just care that it does.
If it something I BUILD then I would care much more but again this is a whole different issue.
The case for more normalized tests is to find out which browser is factually better designed/written.
I don't see anything that would somehow create a bias in favor of a specific browser as far as ad networks goes, if anything the stigma/stereotyping of IE/Edge users would probably mean that ad networks are more incentivized of sending the baity apps towards those browsers.
As for the network part well again that's an important metric if certain browsers perform better at adverse network conditions it's an important factor to know, you do not want to give them the best case scenario every time.
Giving a page a fixed amount of seconds to load is also completely the wrong approach you want to see how browsers behave when they can't load a page properly or when it takes more time than usual, maybe some browsers expend more resources by resubmitting the entire request, maybe some browsers do not parse the DOM tree from scratch when some of the requests stall, maybe some browsers have less resource intensive placeholders for DOM elements, maybe some browsers are better at adjusting the DOM preprocessor for network congestion than others.
So no I can't really see how would your approach would be any better, the approach that MSFT took was quite good, netflix, wikipedia, youtube, facebook etc. with what seems to be realistic user behaviour.
What you want to do is to put in test that would produce fair results for fairness sakes that's not how you evaluate anything because it would not yield you any real world data.
What you need here is a big enough sample.
The post was about that microsoft is not transparent about the methodology/setup/scripts/target websites used. A third party should be able to support Microsoft's claims.
Is it possible for the setup to be published to your github.com/microsoft so that it may be executed automatically? Heck, go the extra mile put it in a CI and publish the data on regular basis :)
I believe Firefox partly resolved the problem on their end later on, although I can't tell you the exact status of things. Nevertheless, there's definitely some precedence for claiming that ad blocking is an expensive operation. Intuitively, you'd think that network based blocking would be enough, but it won't work against same domain ad sources (a typical example being facebook ads), while css selectors are able to capture a bit more depth. Nevertheless, I don't know exactly what has happened since, although I do recall Firefox making some adjustments on their end that improved the situation.
The only reason I can think of for someone not using uBlock Origin is because they've never heard of it.
I've only ever had to whitelist two things in uBlock in all my time of using it and that's pretty good considering that I use basically all of the lists that aren't the language specific ones, the anti-anti-adblock one (which requires a user script) or the merged (ultimate) lists.
But yes, my browsing habits are clearly different from yours and what works for me may not for you. I understand that but what uBlock does is not any different than any of the other adblocks that are out there. It just happens to do it more efficiently and with a better UI than the rest of them. The only difference in behavior that you might encounter is likely to be related to uBlock Origin's strict blocking. In this case exception filters are very easy to create since you literally just have to press the disable permanently and it will forever be disabled for that site.
I've said this before the the best way to use uBlock Origin if you've never used something like NoScript/uMatrix before (or couldn't be bothered with the whitelisting approach) is to try and use what it calls "medium mode". Using its dynamic filtering in this way should net you with the largest gain with the least amount of effort. If you're looking for something with more control than you may want to look into uMatrix since I think the interface will is nicer for that sort of control.
The power button only disables it for one site at a time (or even just a single page, if you Ctrl-click it).
Of course, this is Lotus vs Office all over again. You're using the OS advantage (Windows 10 and its DRM platform) to show that you can do things more efficiently.
You chose the technology platform (I'm guessing Play Ready), you forced GPU vendors to comply, and now you give competitors a bad reputation for daring to use their own (or a competitor's) technology.
Apple has a different approach (own the entire stack) but they're going for the same thing and it has made them a lot of cash.
It's like Apple stressing how well their browser works on -- oh wait, Apple doesn't produce a proprietary plugin. Well, let's say the iTunes page, if it were specifically tuned to work well in Safari.
I'm not 100% sure on this, so any clarification welcome, but I thought that Netflix content contracts DRM requirements allowed GPU decoding via Playready, but not on some others like Widevine. Would the Netflix test still show such an advantage for Edge if there weren't any DRM?
It's not necessarily DRM in HTML5 which is the issue here although I suppose the fractured nature of it may be partially to blame. It's the fact that it's seemingly arbitrary that Chrome gets capped to 720p and no one knows why. Did Microsoft pay a big chunk of change to Netflix for exclusivity or something along those lines? Do the content creators prefer Microsoft's DRM implementation over Widevine's?
If the focus of this test is meant to be on the user experience, as the tone of the Microsoft blog post seems to suggest, then said native ad blocker is a killer feature that really might mean Opera users can "stay productive longer".
If not, then sure, Edge's underlying implementation is probably faster. It would be interesting to see a comparable benchmark of Edge with the best available ad blocking turned on.
* not based on observed user behavior
* not reproducible by you (recorded/replayed network where possible)
* not reproducible by peers
* in stark contrast to reproducible tests provided by others
You describe it as "highly scientific" and jump straight to a marketing campaign. Honestly, who's skipping a lap of the race?
It's pretty clear the data generated is useful from an engineering point of view. It can help identify problem areas. It's also absolutely clear that the conclusions being marketed are blatant misrepresentations of the work.
"In addition to testing battery life internally, Microsoft also looks at tens of millions of PCs that send telemetry data to the firm, Weber said. And what it sees there is simple: A small advantage over Firefox, but massive gains of about 50 percent additional battery life when compared to Chrome."
Whether that means Edge is more power efficient, or devices where Edge is used more often are more power efficient, or users who select for Edge tend to visit sites that allow Edge to be more power efficient isn't clear.
To be fair to both sides, this is a really hard thing to measure.
Is it though? It's more the fact that Big Data is totally useless because it contains thousands of uncontrolled variables that confound the results.
If you really want to know which browser is more power efficient, get a sample of the commonly visited sites and then visit the same sites on the same common devices in different browsers and see which one uses less power.
You can still get the wrong outcome by using a biased sample on purpose, but then the outcome is wrong because you fudged it, not because it's hard to get the real answer.
It's quite possible that both results are valid. It sounds like there's big difference in the sites that Microsoft visited compared to Opera.
It seems an even better methodology would be to record actual user browsing sessions, and then replay them through different browsers and measure power consumption.
Opera's simulated browsing session does not sound particularly sophisticated IMO.
The workloads Microsoft simulated were held constant for every platform and instrumented via a common command-and-control testing library. It is very much apples-to-apples. The additional telemetry they released is even more confirmation.
Opera replied by enabling a pair of special default-off features, both of which significantly change the workload and the user experience for their browser, put it on a random laptop pair which might not even be from the same lot or subtly different models, and then made a timelapse.
I'm not doubting this part at all.
> The additional telemetry they released is even more confirmation.
The additional telemetry doesn't really tell much of a story either way, given that it's presented as a single chart and little context is provided. Unfortunately there's no link to the data analysis to understand what factors were considered when looking at the data sets involved.
Personally I've found that it's hard to come to these high-level conclusions about performance from large data sets, because users and their devices are so incredibly diverse. It's really hard to draw definitive conclusions given all the data.
It's dangerous to take your data, aggregate it at a high level and look at the results from that point of view - you lose the context under which it was gathered and can come to the wrong answer (for example, browser X is popular in a country with great internet, so page loads are faster, and browser Y is popular in a country with terrible internet, so page loads are slower).
It's such a pity I don't use Microsoft Windows 10, because I'd be so thrilled to know that through the magic of telemetry, everything I do on a Microsoft Windows 10 enabled computer could help Microsoft win some crucial PR battles.
For clarity, they captured the extremely detailed telemetry from their test devices. Most of the telemetry that Microsoft ships back to the mothership is the exact kind of telemetry that apps from the Apple Store, iOS Store, Android Store and Chrome Web Store submit: app specific telemetry. Many of these include signals that help understand battery life impact, so every major application platform (including the web) is doing this.
That stuff–stuff you agree to in the EULA as it stands– is absolutely essential to improving app experience to the standards of the market. The reason the mobile ecosystem has developed so fast (and that the web ecosystem can develop so fast) is a whole lot of telemetry on user usage patterns, habits, device failures, device selection, etc.
I'm not sure why it's so controversial. Even the Ubuntu store apps can administer telemetry.
If you'd like to complain about the pervasive surveillance state in your national scale, there are more fruitful avenues.
This is not true at all. And there is a difference between your smartphone and your laptop, even if you can't see it.
They have and they do, on every device. And that's ignoring app specific telemetry, which is explicitly allowed on all app stores.
It is one thing to not want to share that data. It's another entirely to invoke a 7+ year old corporate culture as evidence Microsoft is the sole "bad" actor in the space when every vendor openly discusses their telemetry collection.
How much did we hear about Chrome telemetry at I/O and PWAS? A lot. More to come. And it's necessary to make software products affordable o produce, unfortunately.
There are some programs out there that can help block/disable it.
People who use the Edge browser may do less on their computers while web browsing than people who use chrome?
Even people who do use extensions probably don't really use more than an adblocker.
The more i read this the more pissed off i feel about using windows 10. Unfortunately, they made it very hard to install another OS on new laptops.
* Edge (no adblocker)
* Opera (default adblocker)
* Opera (adblocker disabled)
* Chrome (no adblocker)
* Chrome (uBlock Origin)
Otherwise you're just lying through omission to make your product seem better.
Like it would be fine if they say "in next version we will retake the top" or whatever but don't try to dispute their results by providing something they had no way of measuring.
Essentially they did some power saving improvements and want to create a bit of noise about it to increase their popularity.
It's kind of same as how Apple, Google etc announce new features/products before you even can use them.
I just loaded up MSN.com and scrolled down the page. I count 3 ads total: one gigantic banner at the very top of the page before all of the content, and two smaller panners that pop into place as you scroll. These are served by AdChoices. When I turn my ad blocker back on, these all disappear, and the site loads much faster.
I think if Microsoft were to release a default ad-blocker, they would need to re-code their own website to use native advertising rather than a third-party network. I believe this to be an excellent practice, but I just don't see Microsoft going this route. I'd love to be pleasantly surprised though.
It even works without a network connection.
I'd think the user agent string is more likely to cause a huge discrepancy, as many frameworks will be utilizing polyfills and employing different technologies depending on what each browser supports. (Or, in the case of the user agent string, what the framework thinks the browser supports, regardless of how correct its assumptions are.)
If I understand both tests correctly, Opera ran with adblock and battery saver enabled (those are apparently not default settings), versus Edge without adblock, on sites which supposedly are ad-heavy - this will bias the results towards browser with adblock enabled.
The point to me is that the situation is more nuanced than Microsoft apparently chose to present it.
But there's no reason anyone should be running a browser without an ad-blocker in the first place. Edge should have a lightweight ad-blocker built-in by default.
EDIT: Downvotes? Really this is factually accurate:
But you're right, Microsoft will have to decide which one is more important: driving ad eyeballs to Microsoft properties at the expense of Google properties -- or -- decreasing the share of users using Chrome and by proxy, the attractiveness of Google's integrated cloud email/drive/docs offerings.
That it's a "mode" which defaults to off says a lot - the things it does aren't acceptable all of the time apparently, so you choose between features/performance, and battery life.
In the end the user does not care which OS runs in the background, just which platform offers the best experience - in this case the longest battery time.
Safari, of course, has lots of support for ad blocking short of Apple actually doing the ad blocking itself. I suspect Safari + ad blockers would be the best option across the board (it absolutely rocks on iOS devices).
Opera battery saver is great but they should enable it automatically whenever you're on battery, as it has no performance impact that I've noticed.
Which is one of the reasons benchmarks are so difficult.
PS: A more insidious problem is when you specifically optimize for something because it's a benchmark.
With all this said, "proper-er" benchmark would be to show each of the browser's power consumption with or without adblock, with or without battery saver in Opera's case, all this done on a set of websites which are ad-heavy and ad-light. :)
More importantly even if you grab the content it's rendering that takes power downloading files is very low energy. Even just disabling auto play sound / video is going to save power.
PS: Assumptions about how browsers interact with websites is basically a failure to understand what's going on IMO. Telnet to port 80, send fetch request on a web-server and you will get HTML unless they require HTTPS.
By telnetting I will get a response, but this is only a first step. HTML will have references to huge amount of other content that also needs to be fetched (unless everything is embedded in the HTML - possible, but unlikely for any bigger website). Open developer tools in any browser and go to network tab and see for yourself how the amount of fetched data (and number of requests / number of fetched files) differs when you use adblock and not. This is the difference in the input for the browser. Opera mentions mlive.com - with uBlock, FF fetches 2.5MB of data (caching disabled), without uBlock I get 5MB (caching disabled). We can even take the example to the extreme - let's go to the website that has only ads. Of course Opera will be better there - it will have to render nothing, everything will be blocked by adblock.
As to downloading you can try it you self your talking fractions of a watt savings from not downloading a few Meg's.
In terms of other downloads from a page, most of that is things like tiny images which you can mostly skip. The Amazon homepage sends lots of junk but all you actually care about is the search bar and links to other areas.
Browser advertising fits into the same argument as adds on cable television. Subscribe to Hulu and no the adds don't go away.
Suggesting you need to add third party GPU acceleration to make things fair is ridiculous. But, you seem to think Adblock is somehow different.
Problem with Opera's comparison is that it is not fair because they don't display the same content - they (probably) need to render less. Of course they will be better if they render less (unless they are really bad at what they do). So to make the comparison meaningful, make them display exacly the same thing - then we can talk about who is more power-efficient.
Given that even with the ad-blocker it barely beat Edge I'm almost certain they tried it initially and found the result was true so enabled this feature so they could write this fluff piece.
I stopped using Chrome when I discovered it took hours off my battery life on a MacBook. Edge can't be almost as bad. Intuitively I would believe the ranking in terms of battery life would be Safari > Edge or Opera > Firefox > Chrome.
There are two sides to any story and countries have different versions of reality that they believe in and promote. Nonetheless it's foolhardy to not notice which countries take an overly keen interest in people's communications and the "correctness" of their views... and jail those saying things they don't like.
If you're completely unfamiliar with the history of recent Chinese-western software collaborations, try googling "TOM-Skype"
Software written by persons and companies subject to the will of the Chinese government -- huge problem.
It's really no different than European countries not wanting to do business with US companies because of the NSA.
On the jurisdiction in which I, or the company employing me, operates? That can well be legitimate. You're discriminating against the laws (or unwritten government policies) in that jurisdiction, not against people.
That said, I think a general lack of trust toward the PRC is understandable.
Many users may not like this.
But i generally dont use safari as battery time is not the most important factor in my common use cases.
I use Safari for almost everything. It's by far the best browser for browsing and just reading & enjoying the web. Reader interface and the speed of it is still unbeatable.
I use Chrome for web development, however.
And Firefox does also have a native ad-blocker, so if you enable that, too, I could imagine that it maybe even still outperforms Opera's battery saving mode, but yeah, a proper benchmark would be good...
I don't think it has one natively. Which one do you mean? Or are you referring to tracking protection (private mode only, but available in Fx 49 also in "normal" mode iirc)
And you could always enable it in normal browsing by setting "privacy.trackingprotection.enabled" in about:config to true...
I can't not understand why minimal CPU load is not on top of design requirement of many software products, totally ignored if on a desktop system.
Seriously, imagine the world collective savings on energy if everyone reduces their power consumption by small modest 5%.
If you're genuinely worried about whether HN is being manipulated in this way, you could read https://news.ycombinator.com/item?id=11844253, and if you want to know how serious we are about cracking down on abuse when we see it, you could read https://hn.algolia.com/?sort=byDate&prefix=true&page=0&dateR.... Considering how much work we put into it, I think "buy moderation points in bulk from ycombinator" was in rather bad taste.
We detached this subthread from https://news.ycombinator.com/item?id=11964225 and marked it off-topic.
People post things on HN all the time that criticize Microsoft without getting downvoted — because they do it without antagonizing other HN users.
Anyways, to perform my own little test, I re-wrote my post in an overwhelmingly nice way, and still got downvoted. So, no, it does not seem that the comment was downvoted for being rude.
I didn't read that at all. The situation, as it stands, involves claims and counterclaims and only one side of the argument appears as third-party verifiable. It's only natural to question why an apparent rebuttal from Microsoft doesn't come with any transparency (again!)
IMO, down voting the GP for pointing this out is bullshit.
I voted up your original comment, actually. Microsoft has a long history of flat-out lying; they have been getting better in recent years, but they haven't earned unquestioning trust.
I downvoted your other comments because you're being whiny and petulant and paranoid for no good reason.
your life must be a constant wonder
Apple, Google, Microsoft, and Facebook to varying degrees all have the same phenomenon on HN. It's not the PR team, it's the eng team in their spare time.