

The Surprising Worst Browser - jmathes
http://saucelabs.com/blog/index.php/2011/08/the-surprising-worst-browser/

======
hammock
"Here's the raw data. Firefox beats Chrome and I don't like that result, so
let's fudge with the data a little bit."

"Look, now Chrome is beating Firefox. But IE still isn't losing yet, so let's
make up some reasoning and fudge some more until we're happy!"

"Ah, that's better, now IE looks like it sucks and my worldview is no longer
shattered. Hooray for fudging data to make it say what we want it to!"

~~~
jmathes
Firefox never beat chrome, and I like both of them the same amount.

When I first did the data dive, I thought Safari was worse than IE, which I
thought made a much better blog post. That would have been way more
surprising. It was after _due diligence_ that I was forced to discard that
result.

~~~
archangel_one
Firefox 4 and 5 in the first graph are significantly lower than Chrome.
Obviously it's a pretty dodgy comparison to make since Chrome isn't split by
versions, but I agree with hammock that it seems statistically unsound to
arbitrarily drop some versions of some browsers, especially when they can't do
the same for the Chrome results.

------
archangel_one
"All jobs run recently have lower error rates because our service continues to
become more reliable as we fix bugs our users discover. "

This says to me that none of the results can really be trusted because the
error count is not just errors in the browser - it's apparently significantly
affected by errors in their system, and we don't really know by how much.

Plus it seems kind of anachronistic to come to some conclusion about how the
browsers compare having removed FF4 and FF5. It's only of historical interest
how FF3.6 compares since anyone that downloads it now is going to get FF6 now
anyway.

------
rwolf
It may be true that SauceLabs's code for each browser is the same, but
Selenium surely has significantly divergent code for each browser. I wonder
how to rule out crashes caused by Selenium's browser-specific code...

------
mvzink
_At first we thought the high error rate could be the result of the fact that
we always run Safari on Windows, while it’s made by Apple. That’s easy to
dismiss, because earlier versions of Safari were fine._

I'm sorry, what? That's a pretty weak dismissal.

~~~
rwolf
The earlier data points on the graph show low error rates for Safari browsers
on Windows.

The article may have benefited from including a larger caveat about only
testing one OS, but this seems like an intellectually honest elimination of a
confounding variable.

~~~
mvzink
Perhaps intellectually honest, but a weak conclusion nonetheless: in my
understanding, getting Safari 5 to run on Windows requires porting parts of
Apple's huge, complicated Cocoa framework to Windows. That framework is always
changing, and so is the browser-- and they both primarily target OS X. With OS
X, Windows, Cocoa, and Safari all moving quite a bit between Safari versions,
I would be surprised if it didn't become harder to support Windows as the
product grows.

------
untog
"all FF3.5 jobs are from a long time ago, before we’d had much time to
streamline and errorproof our system."

"This is the same graph as above, with the unfairly advantaged or
disadvantaged browsers removed"

What? Why not take out _all_ results from the period when your system was
unstable, rather than removing the results that looked 'wrong' to you? Good
grief.

~~~
jmathes
The system was never very unstable. The error rates are all low. The point is
that it's now much more stable.

The reason I didn't sort error rates by month was to preserve statistical
significance. If we try to only look at error rates from specific periods of
time, we have to decide how short those periods of time are. Too long and we
don't have data from some of the browsers; too short and we don't have enough
data to form a significant opinion. It may have been possible to find a middle
ground, but why bother when we can still get unbiased results for 80% of the
browsers?

~~~
untog
I get what you're saying, but you still removed data that _you thought was
wrong_ in order to give the results you were expecting. Who's to say that the
results for Chrome in that period were not also wrong?

------
socratic
What is the point of this article? It appears to be that Safari proxy mode is
unreliable, which I guess is surprising but not that interesting.

Given that the actual article does not have anything to do with the headline
("error rates" is never really satisfactorily defined, the real story is proxy
mode Safari, and IE6 came out 9 years ago), what makes this headline so
compelling?

Do headlines with the word "surprising" always work? Is it that everyone deals
with browsers, so a browser related story is likely to have a wider mind
share? Is it that web developers all just want a chance to make fun of IE6,
due to the collective pain that it has inflicted on all of us?

------
sdfjkl
Oh look, free, anonymous Windows VMs for 20 minutes (it's trivial to open
CMD.EXE from any browser in the saucelabs setup). Any statistics on what
people run in those VMs, other than browsers?

~~~
hammock
How do you do that? When I type "C:\windows\system32\cmd.exe" in the browser
bar it says "Access to that resource has been disallowed."

~~~
walrus

      1. Go to File > Open, then click Browse...
      2. In 'File name', type 'c:\windows\system32' and press enter
      3. Change the 'Files of type' field to 'All Files'
      4. Drag any of the files into the 'cmd' icon
    

Bonus: type `explorer C:\\` if you want to snoop around. There's a copy of
Safari at C:\Program Files\Safari\Safari.exe

------
gabyar
Regardless of possible testing or analysis concerns, it's pretty clear that
Safari 5 has a higher error rate than Safari 4 or 3. Safari is the only
browser to go down in quality over time. This appears to be further evidence
that Apple is putting less resources towards Mac software in favor of
supporting ios devices. Fairly concerning for all of us Mac users.

------
sapphirecat
So you claim to have "significant data" but the first thing you do is throw
out huge swaths of it because it's not actually significant?

Also regarding graph 2's commentary, 0.51 is not "almost" double 0.34. It
turns out to be only 1.5x, so calling it "almost double" is about as
meaningful as calling it "almost the same".

------
kenjackson
Does this diffentiate between bugs in the browser and plugins? I've debugged
several IE issues in the past and it's almost always a plugin.

------
msredmond
Spoiler: it's IE 6 -- and the copy even says at the end that the surprise is
that it's not a surprise. Points for the Ash reference though.

~~~
corin_
IE6 is a bad browser, how shocking.

The actual interesting story here is Safari 5.

~~~
seanalltogether
Safari on windows is terrible. In my own tests last year it kept erroring out
or rendering incorrectly. <http://www.craftymind.com/guimark2/>

~~~
rjd
Safari 5 on OSX Lion is terrible.

I'm usually a safari fan, but since upgrading last week nothing but trouble.
Opened Chrome and Firefox a few times instead but I'm to accustomed to Safari
now... and admittedly the gestures are AWESOME.

I was hoping the 10.7.1 update would have fixed it but nope :/ Maybe 10.7.2, I
feel my days numbered ATM though

------
twodayslate
IE6 is not a surprise...

