
Firefox Takes the Performance Crown From Chrome - bpierre
http://news.slashdot.org/story/13/07/02/146251/firefox-takes-the-performance-crown-from-chrome?utm_source=slashdot&utm_medium=twitter
======
evmar
(Disclaimer: I used to work on Chrome. I'm more than a year out at this point
but I still talk to Chrome developers.)

Chrome wins some tests but loses the most on the start-up benchmarks, which is
a real bummer. There was a time where we'd spend literally days figuring out
why startup regressed by a handful of milliseconds, and this test shows an
end-to-end test taking nearly six seconds.

One idea I saw mentioned is that there may be a bug related to how Chrome
reads the system proxy settings (something about an API change between Windows
7 and Windows 8). I think most of the Chrome developers are still on Windows
7; the bots that are tracking startup performance are definitely still Windows
7 as well. So maybe that number is more of a single bug than a systematic
thing.

All of the above is not intended to diminish Firefox's impressive results --
just thought I'd provide some background! What these sorts of tests show most
is that competition pushes browsers ever faster. If the proxy theory above is
right, then maybe this test will inspire Google to invest more into Windows 8
testing, which ultimately benefits users.

(Edit: Firefox won other tests too, I also didn't mean to say this was the
only factor.)

~~~
_pmf_
> Chrome wins some tests but loses the most on the start-up benchmarks

Well, congratulations to FF for being the perfect browser for the 0.01 percent
of people who don't keep their browser open all day.

~~~
mariusmg
Startup time matters (a lot). Even if i open it once a day i still want it to
be fast. Nobody likes slow apps.

~~~
coldtea
> _Startup time matters (a lot). Even if i open it once a day i still want it
> to be fast. Nobody likes slow apps._

Slow is the 99.9999% of time you use it. Not the few seconds it takes to open.

Also, get an SSD and be done with it.

Also, with sleep, you don't even startup once a day.

~~~
shocks
I have an SSD, Firefox still takes a few seconds to open. Chrome is instant.

~~~
derleth
But for the majority without an SSD, Chrome is a lot slower.

------
simonsarris
Performance is nice to be sure, but I think other things are much more
important in 2013, like consistent HTML Canvas implementations.

For one small example the in-canvas kerning on both Firefox and Chrome are
_awful_ compared to IE10.

From a while ago:
[http://i.imgur.com/62WBzVZ.png](http://i.imgur.com/62WBzVZ.png)

You can see for yourself here:
[http://jsfiddle.net/vVC4s/](http://jsfiddle.net/vVC4s/)

Notice also the difference between doubling the font and doubling the scale,
esp where each line gets cut off:
[http://jsfiddle.net/jGcrL/](http://jsfiddle.net/jGcrL/)

This makes animations rickety and un-smooth (text animation):
[http://jsfiddle.net/simonsarris/HZFcR/](http://jsfiddle.net/simonsarris/HZFcR/)

Surprisingly IE10 handles all of these perfectly. Of course, IE10 has totally
broken canvas clipping (non-rectangular clipping regions are impossible. They
worked fine in IE9.)

Firefox does render large text much better than Chrome, which is why I used it
when taking screenshots for my book. But scaling the text (as opposed to
setting a larger font) is a disaster.

This is just an example. I could go on for days about canvas bugs. I wish
there was a bigger push to fix those instead of eking out a performance
advantage.

To Firefox's huge credit, I've submitted a lot of bugs to Chromium and the FF
team, and the FF team _consistently_ gets back to me within a week and usually
fixes the bug within a month. The bug reporting experience with Chrome on the
other hand is rather disenchanting.

For a cross-platform bug example, the context's miterLimit is just plain
broken by default in Chrome. I reported this (with examples) back in April and
have yet to receive _any_ kind of reply. Thank god it's an easy workaround.

[https://code.google.com/p/chromium/issues/detail?id=225512](https://code.google.com/p/chromium/issues/detail?id=225512)

If you want people to use your browser and develop with it in mind, working
elements are more important than being slightly faster than the others. (Good
developer tools come in close second I think, at least for this crowd).

~~~
mati
My experience with reporting bugs in FF/Chrome is a bit different. I reported
a bug in canvas text implementation, which occurs in both of those browsers,
on Feb 20. I provided all the necessary info and even a minimal test case. The
bug has been confirmed in FF quite quickly (within days) and then... nothing.
It's quite serious, basically you can't do smooth animation (movement) of
text. Similar to the text scaling bug.

[https://bugzilla.mozilla.org/show_bug.cgi?id=843310](https://bugzilla.mozilla.org/show_bug.cgi?id=843310)

Reporting to Chrome has been even worse. No word, no nothing. Not even a
confirmation. (Although the submitting process itself has been simpler AFAIR).

That's really uncool, after a few of those you lose interest in submitting any
more since the feedback feels like "we don't care". So you end up using the
time to find workarounds instead.

Now obviously someone will point out everyone can go ahead and submit a patch
for the bug itself since it's open source. That's all cool and dandy, I love
open source as much as the next guy, but please, let's get serious. Who has
the time to dive into a massive codebase like that and fix the bug? Not
everyone has plans to become a browser developer.

~~~
lxt
I note a Mozilla developer has commented on this bug today:
[https://bugzilla.mozilla.org/show_bug.cgi?id=843310#c3](https://bugzilla.mozilla.org/show_bug.cgi?id=843310#c3)

~~~
mati
Indeed, I started getting some emails after posting my comment here. The power
of HN.

------
bd
Ehm, at least one of the tests they used is wrong. They used this one as WebGL
performance test:

[http://luic.github.io/WebGL-Performance-
Benchmark/](http://luic.github.io/WebGL-Performance-Benchmark/)

Despite the name this program in fact doesn't use WebGL, it runs using
three.js 2D canvas renderer (just check the source).

Additionally even the results they got there are suspicious, I got completely
opposite results on my system (Windows 7), with Chrome being faster than
Firefox by a large margin:

[http://www.tomshardware.com/reviews/chrome-27-firefox-21-ope...](http://www.tomshardware.com/reviews/chrome-27-firefox-21-opera-
next,3534-8.html)

They got 777 on Firefox vs 437 on Chrome, I got 290 on Firefox vs 441 on
Chrome (this is fully in line with my everyday experience, doing browser
graphics).

~~~
glomph
Toms hardware seem to consistently have methodology problems in their tests. I
don't understand why they are given so much credence.

------
forgotAgain
I just switched from Chrome to FF. The performance is close enough, and good
enough for both, that it's a secondary consideration at this point.

I switched because I got tired of hearing Chrome constantly accessing my hard
drive. I wound up going through the list of Chrome switches here
([http://src.chromium.org/svn/trunk/src/chrome/common/chrome_s...](http://src.chromium.org/svn/trunk/src/chrome/common/chrome_switches.cc))
to try to alleviate it. Some things helped but not to an acceptable level. I
use W7. Using procmon I could see Chrome constantly re-reading keys from the
registry and writing to temp files even though caching and pre-fetching were
disabled.

I was also was concerned for Chrome accessing my laptop SSD. Even though I
couldn't hear it I could see the lifetime allotment of reads and writes being
flushed.

~~~
rjh29
On a similar note, my PC's hard drive starts making crazy noises as soon as
Windows 7 goes into screensaver mode. I've turned off every disk-related
option I can find and it's still doing... something. Maddeningly it stops as
soon as the screensaver is interrupted.

Linux is a lot more predictable in that regard...

~~~
ericabiz
Probably the Windows indexing service. It's configured to run when you're not
using your computer. You can turn it off and see if that stops the hard drive
noises: [http://answers.microsoft.com/en-
us/windows/forum/windows_7-f...](http://answers.microsoft.com/en-
us/windows/forum/windows_7-files/disable-windows-search-
indexer/914286d5-9485-49a9-a3df-f9516c9970cd)

------
leeoniya
i'm developing some js compute-heavy apps and Chrome still pulls ahead quite
significantly (by as much as 5x) in many cases. i've filed js-perf bugs with
mozilla that have been accepted but seem to be quite low on the priority list.
artificial benchmarks don't always tell the full story :(

if any mozillian js dev is lacking stuff to work on ;)

[https://bugzilla.mozilla.org/show_bug.cgi?id=879393](https://bugzilla.mozilla.org/show_bug.cgi?id=879393)

[https://bugzilla.mozilla.org/show_bug.cgi?id=858986](https://bugzilla.mozilla.org/show_bug.cgi?id=858986)

------
driverdan
Actual benchmark story from yesterday:
[https://news.ycombinator.com/item?id=5970429](https://news.ycombinator.com/item?id=5970429)

------
speedyrev
Go straight to article referenced in Slashdot
[http://www.tomshardware.com/reviews/chrome-27-firefox-21-ope...](http://www.tomshardware.com/reviews/chrome-27-firefox-21-opera-
next,3534.html)

------
zimbatm
The issue with Firefox is that some times the whole browser is freezing.
That's what keeps pushing me back to Chrome, not performance issues.

~~~
matthiasv
That is _your_ issue with Firefox or your setup. I never had this for at least
one or two years now, even with misbehaving scripts.

~~~
ksec
Nope, It is a common issues with Firefox although it has been greatly reduced
in the last year or so.

The project to fix all this is called e10s. Moving Content, and UI etc to
seperate process as well as OMTx ( Off Main Thread Everything ) where Firefox
Dev trys to do everything in async style and move them off Main Thread.

Progress has been slow. But at least they are working on it

------
Pxtl
I gave up on Firefox a year or two back because of performance concerns - not
raw speed but ram consumption (leading to thrashing when tabbing between ram-
intensive processes) and its poor single-threaded freezing problems. Nice to
hear that FF is getting better and I'll be able to go back to it - its
extension community is far better than Chrome's, and Firebug is without peer.

~~~
UberMouse
Frankly I think Firefox performs a lot better than Chrome for memory usage
now. After a few days of having Chrome open it's usually using between 3 and
6GB of RAM (Usually Facebook is using a gigabyte so killing that tab frees up
a lot) and this is with <50 tabs open. Friends who use Firefox end up with
~1GB of RAM usage in a similar timeframe.

------
magicalist
I'm really surprised that no one in these comments and the thread on the
actual story[1] (instead of this...whatever the equivalent of blogspam is but
for slashdot) hasn't mentioned the horrible averaging methodology of this
benchmark suite.

The individual tests aren't even a problem (though I would maybe pick some
more and/or different ones, especially their somewhat odd benchmark choices in
performance and graphics), but the averaging makes no sense at all.

Averaging time-based benchmarks is problematic enough (it feels more right,
but it still isn't a good idea), but how on earth do you convert a "number of
times we had to refresh a page" result into a number that you can then average
with a "standards compliance" count and a measure of memory efficiency?? Even
if you normalize (which it doesn't seem like they did from the output numbers,
but maybe they did), the numbers still aren't comparable because you've given
no account for the relative magnitude of their effect.

e.g. if you have a test of "does the browser have a konami code easter egg?",
it doesn't matter if the geometric mean is less sensitive to outliers, because
it still doesn't make any sense to take an average of that with "the playback
framerate of an HD Trololo video" and then pretend that the average provides
any insight. And it's even worse if you then compare that average to other
crazy averages!

At best you can look at relative ranking, which they actually mention but then
proceed to give exact numbers for their relative ranking. There's no
information about "betterness" in there, though, except if you then divide up
the numbers again to say "these points came from the win in test A, and these
points from test B"...at which point you just have the original tests again.
Better to count win/no-wins and use that as your final result. Then at least
it's obvious that if some tests are much more non-trivial than others that
you'll have to give arbitrary weights for the final result, as opposed to
having the arbitrary weights being implicit in the tests themselves.

Sorry for the rant :) This is a good recognition of Mozilla's hard work,
though notably they've been winning many of these tests for a while now
(especially the memory ones), but it would be nice if tomshardware could drop
the basically meaningless overall scores (or we could just collectively ignore
them).

[1]
[https://news.ycombinator.com/item?id=5970429](https://news.ycombinator.com/item?id=5970429)

~~~
bzbarsky
I agree the need to produce a single overall number is nuts. We need a better
setup for comparing things, somehow...

That said, if you _do_ need to produce one in a situation like this, then
geometric mean is the least bad way to do it because it does not depend on the
normalization of the individual tests at all: if you have 'k' tests and change
the workload for one of them by a factor of N, then the average will change by
a factor of N^{1/k} no matter what the actual numbers are. This is why it's
used so commonly in benchmark that want to produce a single number.

Of course this very property is what makes geometric-mean-based benchmarks so
gameable: a 20% speedup of an already-fast operation (say eliminating a single
instruction) is worth as much for purposes of the benchmark as a 20% speedup
in something that's really slow. The result is that browsers keep micro-
optimizing already-fast things to win the benchmark game....

------
deepblueq
So much of any comparison like this is about performance, but I really don't
think that's as important as it used to be, and that's coming from someone who
generally uses slow computers and optimizes software until they run fast.

I'm typing this on a bit of an exceptional example, a 2.6 GHz Northwood
Pentium 4 with 1 GB of single-channel DDR-400 RAM. The one saving grace is
that it has an SSD, but I put the swap file on the spinning drive (which is
modern). It's running Linux Mint 14, Xfce edition, with a handful of minor OS-
level optimizations. Firefox 21, with a fairly standard configuration,
flawlessly handles a dozen or more tabs on a daily basis. It's even pretty
snappy, more limited by my internet connection (1.5 Mb DSL) than by the
hardware it's running on.

If this sorry excuse for a computer does that well, are the relatively minor
differences in performance between browsers going to be a big deal on modern
hardware? There will be edge cases, such as the people who have hundreds of
tabs open at a time, but for the average user I'm having trouble envisioning
that.

The things that make a difference anymore are very tough to quantify in tests
like Tom's Hardware did. I will always have Firefox around because I think
Mozilla actually cares about privacy. I use Presto on my phone because it's
the only one I've found that renders things how I want. Many people are tied
to a browser because of extensions. Standards support doesn't matter until you
find a page where a browser doesn't work, and those pages will be different
for different people. Browsers can be rock solid on one computer and
worthlessly crashy on another.

I don't think a round of benchmarks has meant anything to me in browser
selection for a long time, and when it did I did them myself so as to account
for the computer they were running on. I choose by trying to use a variety of
them for a while, and a winner always emerges quite quickly.

------
Myrth
I seriously can't understand how people can use it with its constant single
threaded UI freezing. I've just tried nightly and still the same issue!

------
ajays
I wish they (FF) would also spend some time on the small annoyances:

 _Lack of a restart button. Now when I upgrade FF, I have to "kill -9" it from
my terminal to get it to restore windows upon restart_

 _Memory leaks. Leave yourself logged in to Facebook for a few days, and watch
it take up 2GB of memory._

 _No way to easily filter sites with cookies like you can in the next tab
over, where you manage sites with saved passwords. Why is this? Does FF
secretly want you to not muck with cookies?_

Now, I'm sure there are addons and plugins for the above. But I should _not_
need addons for basic UX. Save the addons for fancier stuff.

~~~
padenot
* For the restart after upgrade, either it is a bug, or you have not seen the little "restore tabs from last time" button in the lower right part of the browser.

* I have not experienced that, I don't use Facebook.

* For the cookie stuff, you'll be pleased to learn about the about:permissions feature (type it in your address bar).

~~~
epmatsw
Whoa, about:permissions is awesome. Thanks for that!

------
ekm2
I just learned something my math teachers never told me from a tech blog:

 _Geometric mean is useful for comparing when the expected range or units of
values is different. For example, startup time is measured in seconds, but
BrowsingBench numbers are things like the unitless 6646. The arithmetic mean
would fail to "normalize" these values and give disproportionate weight to
some over others; the geometric mean is one way of trying to account for
this._

------
Roboprog
I suspect this was "by a nose". But that's good: I hope to see both browsers
trading blows in this little war, leap frogging back and forth in the lead.

~~~
zaph0d
Actually not; Firefox beat Chrome comprehensively in the overall test. Chrome
actually came 3rd in the list -
[http://www.tomshardware.com/reviews/chrome-27-firefox-21-ope...](http://www.tomshardware.com/reviews/chrome-27-firefox-21-opera-
next,3534-12.html)

~~~
magicalist
To be fair, this slashdot story covering the original story (presumably
submitted because the original story was submitted and talked about yesterday)
mentions the "performance crown", which was just "by a nose". You're right
about the overall test, though.

------
3327
I did not find this objective, the test scope was limited and the sample set
of browsers tested were not the latest version. Not that I use IE but why use
IE10 when IE11 is available for testing? If you are doing a performance
benchmark browser vs browser you will not be taken seriously if your test is
not objective. In this case I find the sample set skewed.

~~~
pix64
If I understand correctly, IE11 is only a preview version not the final
release.

~~~
3327
Correct it is a preview.

If that is the case then I am wrong in my assessment and withdraw my
statement.

------
ebbv
These types of headlines are so meaningless. On what benchmark on what
platform?

On OS X Safari outperforms both for many tasks (again, YMMV depending on what
you're doing.) But that's meaningless for the billions of Windows and Linux
users out there.

I wish our industry would move past these kinds of silly headlines. When it
first started when Chrome came out there was a _massive_ difference between
Chrome and the other browsers, now all of them are very capable and
competitive for pretty much anything you want to do.

~~~
mtgx
TomHardware's browser benchmarks are pretty comprehensive. However, what
annoys me about this test and this headline, is that Chrome 28, with its new
faster Blink core, is literally a week away from being released, which means
Firefox will only have its 15 seconds of fame (or rather a week).

Why did they make the test immediately after Firefox was out? Or do they
repeat the test immediately after each browser version comes out? In that case
I'm looking forward to the test with Chrome 28 included.

I'm not saying Chrome 28 will necessarily win in the next one. I just find it
a little strange that they did it without waiting a bit more for the next
Chrome version, too, before writing a headline like that. It just reminds of
me those polls who turn out a certain way depending on how you ask the
question.

~~~
mariusmg
Most likely because it takes time to do those benchmarks. After Chrome 28, FF
23 should be out so why not wait for that either ?

~~~
mtgx
The difference between FF release and next Chrome release is 2 weeks at most.
The different between the Chrome release and the next Firefox is 4 weeks. But
whatever, I'm not too hung up on this. Good for Firefox, if they can maintain
that lead.

------
jumpbug
In my daily use, I find Chrome to be overall faster and easier on the computer
in general. I have recently run into a case during my development where Chrome
actually has trouble compared to Firefox ANNNNNNND IE. It involves a large
unordered list with several different divs, buttons, and links in it. There is
some serious jumpy scrolling compared to both of the others, which
surprisingly render smoothly.

This is the only instance I've ever run into where it didn't quite measure up.

------
tommorris
The reason I use Firefox is because it has a set of extensions I find useful,
and because I like Firefox Sync (end-to-end encryption of bookmarks/history
etc.).

Whether the browsers shave a few microseconds off JavaScript performance is
neither here nor there for me: but perhaps I'm a weirdo.

Performance is nice, but not enough of a reason for me to switch browser.

------
Shivetya
I just want to know one thing. When I have only two add ons in Firefox why
does it take up nearly 200m of memory with no pages displayed? IE is taking
38m with this page displayed, FF has moved to 208m. Firefox is so damn quick
to eat memory it causes my laptop to start caching which decreases performance
and eats battery.

------
jscheel
Still has big issues with mouse events between iframes and parent documents.
Been an issue since at least 2009. Sad.

Edit: issue was reported in 2001. I can't even comprehend that.

------
mrbill
I stopped using Firefox and switched to Chrome when, even on an x86 Mac, I
could see the browser's UI redraw itself. Not going back at this point.

------
youngtaff
Doing comparative tests of browsers is OK up to a point but in reality the
performance problems on the web are due to the way we build and deploy sites.

------
visarga
And that's the day I switched back. I abandoned after FF 3.x

------
brownbat
What, no Lynx?

~~~
Argorak
WebGL is not implemented, though 3d renderers to tty exist since the 90s, e.g.
ttyquake! A scandal!

[http://web.archive.org/web/20021113144923/http://webpages.on...](http://web.archive.org/web/20021113144923/http://webpages.onvoy.com/bobz/ttyquake/)
(original site gone)

------
lesslaw
I switched to Chrome to see what the speed was like and I came to the
conclusion it was a slicker browser for daily use.

But then I reminded myself that some things are more important than raw speed
and responsiveness.

Google's interest is tracking and targeting. Chrome actively worked against me
in this regard. Firefox, despite its financial ties to Google, put control and
freedom onto my desktop.

A bit of occasional sluggishness is a worthy price to pay.

The beer is free at all the browser bars and while I was drinking it, I
remembered I liked free as in speech.

~~~
Karunamon
>Google's interest is tracking and targeting.

Last I checked, every bit of this can be turned off in the browser settings if
they really bother you that much. (Or heck, use Chromium.)

In my experience, they offer quite a bit. Typo correction, instant and
stupidly fast searches from the address bar, that kind of thing.

I'll never understand the pathological hate-on that people have for
advertising, I suppose.

~~~
lesslaw
Advertising is annoying and rude. Flashing away trying to steal my attention
from the activity I am trying to pursue.

"don't do that, come and do this instead"

"skip this ad in 5 seconds" \- no thanks

The interesting thing about the New Albion was that it was so completely
modern in spirit. There was hardly a soul in the firm who was not perfectly
well aware that publicity--advertising--is the dirtiest ramp that capitalism
has yet produced. In the red lead firm there had still lingered certain
notions of commercial honour and usefulness. But such things would have been
laughed at in the New Albion. Most of the employees were the hard-boiled,
Americanized, go-getting type to whom nothing in the world is sacred, except
money. They had their cynical code worked out. The public are swine;
advertising is the rattling of a stick inside a swill-bucket. And yet beneath
their cynicism there was the final naivete, the blind worship of the money-
god. Gordon studied them unobtrusively. As before, he did his work passably
well and his fellow-employees looked down on him. Nothing had changed in his
inner mind. He still despised and repudiated the money-code. Somehow, sooner
or later, he was going to escape from it; even now, after his first fiasco, he
still plotted to escape. He was IN the money world, but not OF it. As for the
types about him, the little bowler-hatted worms who never turned, and the go-
getters, the American business-college gutter-crawlers, they rather amused him
than not. He liked studying their slavish keep-your-job mentality. He was the
chiel amang them takin' notes.

------
humanspecies
Thanks Google, for investing on the best web browser out there!

------
mdhgriffiths
Websites like "Tom's Hardware" really piss me off.

